|
Benchmarking Breeds Controversy
Editor's Forum
By its very nature, graphics performance benchmarking is controversial;
especially when you are doing it across different hardware architectures
and APIs. How couldn't there be controversy? You are taking different
systems with different performance assets and trying to apply a single
criteria to measure performance. And not just any kind of performance,
but graphics performance, arguably one of the most difficult characteristics
of a computer system to measure.
The situation becomes even more heated when you try to take a series
of performance numbers and boil them down to a single number that is
supposed to be the final word on performance measurement. In actuality,
everyone knows what the composite number is -- it's the performance
equivalent of a sound bite (I'll leave it to the reader to supply the
bite/byte pun).
The reason more than a dozen prominent vendors and researchers on the
GPC Group take on this seemingly thankless task is simple: standardized
performance measurement is desperately needed in the industry, by both
vendors and users. But, just because it's a good cause doesn't mean
everyone is going to agree about how to go about it. Good intentions
don't necessarily lead to equanimity when it gets down to the dirty
details.
Certainly the GPC Group has had its share of controversy. Compromises
are reached, but the controversy never really goes away; it's endemic
to performance benchmarking. As evidenced by the performance benchmarking
results found in this publication, plenty of progress has been made.
And, no doubt there will be plenty of progress to report over the upcoming
months. But, as always, it's likely to be ushered in by controversy.
|
