|
In Search of Credible Composites
Editor's Forum
Composite performance numbers are somewhat like film treatments of
novels : a lot of decisions need to be made in the name of condensation
and mass appeal and rarely are those decisions universally applauded.
In performance benchmarking, there are greater risks than even those
associated with subjective judgments about what's important and what's
not so important in the grand scheme of things. There's a chance that
if a composite is done a certain way, it could give an advantage to
one vendor's systems over another's. But, just as consumers like to
see blockbuster novels come to the big screen, so do computer users
like to see the one number that sums everything up.
Of course, it's not that simple. The increasingly complex world of
computer graphics performance can't be summed up by a "one-size-fits-all"
philosophy. Benchmarking organizations such as the GPC Group try to
balance the need for accurate performance measurement with the public
demand for "quotable" numbers. So, while it provides composite numbers,
the committee encourages users to look at the individual reports when
comparing systems. Ideally, the committee urges users to benchmark their
own applications.
It took the Picture-Level Benchmark (PLB) project group more than three
years to come up with its composite numbers. The XPC project group took
more than a year to develop Xmark93. After more than a year of
deliberation, the OPC project group decided on a single composite number
for each Viewperf viewset.
In all three situations, the arrival of a composite number was met
with gnashing of teeth and more than a few sighs that indicate
"there's got to be a better way." Members of groups no doubt feel
the same kind of regrets as film editors who have just reduced
a hundred hours of raw film into the two-hour movie that is supposed
to capture the novel perfectly.
|
