http://asawicki.info/ Programming, graphics, games, media, C++, Windows, Internet and more...
What do we have from benchmarks?
There was this case some time ago about some graphics vendors cheating in Futuremark benchmark (see this). They basically detected this particular application and raised frequency to increase performance and gain higher score. So some devices have been delisted from the Best Mobile Devices list for cheating and they published this document: Benchmark Rules and Guidelines.
My first thought was: Good, they just want everyone to play fair. But then I read the rules again, especially this one: "The platform may not replace or remove any portion of the requested work even if the change would result in the same output." and I said: Wait, what? Isn't it a generic definition of every optimization? If a developer writes 2+2 in GLSL and the platform just uses 4, is it cheating because it removed requested work (addition in this case) even if result is the same?
And then I started thinking: What do we have from benchmarks after all? Is their importance a good thing for gamers and other customers of graphics technology? In theory, benchmarks should mimick some aspect of real applications to measure and compare how different hardware performs in this type of applications (e.g. games). But it may be that decision makers want to just see good scores in benchmarks (bosses generally like numbers and bars and graphs :) so engineers implement optimizations or even some cheats just for these benchmarks. And then media notice that, devices get delisted, benchmark creators write such rules... and gamers just want to play games.
If performance was measured just in real games, and platform vendors optimized or even cheated for a particular title, then at least we would have a better performing game. Just my personal opinion :)