1 Comment

The Benchmarking Trap

Further working hard on ruining Adobe‘s public face in the ongoing Flash vs. HTML5 debate, John Nack unwisely cited some – umm, I wouldn’t even call it that, it’s really more a random experiment – benchmark and drew the wrong conclusions. Now apparently he’s a Photoshop Product Manager, so we probably can forgive him for not considering the larger issues, but still, it once again feels like a desparate cry from Adobe to prove Flash is better and justify its ongoing development and use.

Anyway, let’s not get distracted too much by this company nonsense, and let’s focus on the bigger issue behind it: What are benchmarks good for, anyway? You can find them all over the place, yet they pretty much tell you nothing. Be it Sunspider, Viewperf, Cinebench or even this totally biased After Effects benchmark, they only tell you what you already know – that on faster hardware your software will operate faster, that a newer version of a program on the same hardware is usually also faster and that sloppily programmed software generally sucks. *boo* And here comes the real deal: While all these nice numbers those tests produce may have you gloat over your buddy’s seemingly weaker machine for 5 seconds, their practical relevance goes toward zero. There’s always a performance killer somewhere, be it just some bad plug-in in an After Effects project or those blurred reflections in a 3D render.

So to return to the starting point of this post: This is no different with programming for the next generation of the web, mobile or otherwise. Poor code such as used in this example will always lose out on an infrastructure like Flash that can precompile optimized bytecode and has a ton of runtime optimizations, even more so since native rendering of JavaScript and HTML5‘s <canvas> element is solely a matter of the browser’s virtual machine performance and rendering speed. In current browsers there’s certainly a lot of room to improve upon that and eventually, they will get there…

Advertisements

One comment on “The Benchmarking Trap

  1. Unfortunately I don’t know John Nack, nor do I work for Adobe so I can’t comment on the re-posting my results. I agree completely that performance is based on the browsers ability to render the content and that this will improve over time. What’s interesting is that the iPod Touch is brand new and not multi-tasking anything. The Nexus One has similar specs but can multi-task while outperforming using the canvas. Even more interesting is the gap between performance and display of CSS3 content where iOS clearly wins. Everyone is touting HTML5 for mobile, I’m merely trying to suggest that HMTL5 isn’t quite ready for mobile. Flash definitely appears to be ready with 10.1. Would be interested in hearing more feedback.

    Here is a post relating to CSS3 and Flash on iOS and Android:
    http://www.blackcj.com/blog/2010/09/19/css3-excels-on-ios-but-lacks-on-android/

    Cheers,

    Chris

Comments are closed.

%d bloggers like this: