|
The FAQs (Frequently Asked Questions) About
the SPECglperf® Performance Evaluation Toolkit
Q. What is SPECglperf®?
A. SPECglperf is an executable toolkit that measures the
performance of OpenGL 2D and 3D graphics operations. These operations
are low-level primitives (points, lines, triangles, pixels, etc.)
rather than entire models such as those used in the SPECviewperf
benchmark. Because of the time commitment needed to conduct SPECglperf
testing and the complexity of presenting the results, the SPECopc
project group decided in summer 2000 to discontinue reporting
for this benchmark on the web site. SPECglperf will continue to
be offered, however, as a free downloadable toolkit that is valuable
for internal performance evaluation for vendors and users of graphics
systems.
Q. What is the purpose of SPECglperf?
A. The purpose of SPECglperf is two-fold: to standardize
the method by which vendors measure OpenGL primitive performance
on their systems, and to use interactive graphs and other data
analysis tools to present a much fuller performance picture than
that provided by isolated primitive numbers.
Q. Who has developed SPECglperf and who will maintain it?
A. SPECglperf was developed by the OpenGL Performance
Characterization (SPECopcSM) project
group, a graphics benchmarking organization that operates under
the auspices of the Graphics Performance Characterization (SPEC/GPC)
Group.
Q. What is a SPECglperf script?
A. A SPECglperf script describes the graphics primitives
that will be included in performance tests. One script tests performance,
for example, based on a 50-pixel triangle strip. A script allows
many disparate primitives to be tested during one SPECglperf run
and allows easy changes and comparisons between different tests.
Q. What scripts are used within the SPECglperf toolkit?
A. There are 13 SPECglperf scripts, split into 10 RGB
scripts and three color index scripts. The scripts are further
divided by functionality: the OPClist scripts (RGB and color index)
contain a number of tests for a variety of graphics primitives
and other operations (such as window-clears). These tests are
probably the closest parallel to primitive-level results available
from most vendors today. Other scripts feature specific graphics
operations, such as CopyPixl.rgb, DrawPixl.rgb,
ReadPixl.rgb, TexImage. rgb measure glCopyPixels,
glDrawPixels, glReadPixels and glTexImage2D
RGB operations. DrawPixl.ndx and ReadPixl.ndx are
the color index analogs to DrawPixl.rgb and ReadPixl.rgb.
Remaining scripts address underlying graphics concepts that affect
OpenGL performance -- BgnEnd.rgb measures performance as
it varies with the number of primitives batched together (in a
glBegin/glEnd pair), FillRate.rgb measures how fast
rasterization operations are performed (how many pixels are drawn
per second), Light.rgb measures the effect of the number
of enabled light sources on drawing a particular primitive, and
LineFill.rgb and TriFill.rgb measure the effect
of increasing primitive size on the drawing rates of lines and
triangles, respectively.
Q. Why are there so many numbers, instead of just one?
A. One number is certainly preferable for marketing purposes.
It does not serve justice, however, to boil down meaningful results
into one meaningless number. Take the analogy of buying a car:
one must compare many aspects of each car to arrive at the best
buying decision. Most car shoppers do not take totally unrelated
metrics and composite them for comparison purposes. They don't,
for example, take an average of the acceleration, top speed, trunk
space, fuel efficiency, leg room, gas tank capacity and inverted
braking distance. So why would one insist on doing so for something
as complex as 3D graphics performance? Rather, one should choose
the metrics most related to one's needs and compare using those
figures. A CAD user, for example, might not care particularly
how fast a system can transfer an image from memory to the screen,
but this is vitally important to others, such as those in the
film and video industry.
Q. What is the difference between SPECviewperf™ and
SPECglperf?
A. Both SPECviewperf and SPECglperf measure the graphics
performance of a system through the OpenGL API. They were designed,
however, with different goals in mind. While SPECviewperf draws
an entire model with differing sizes of primitives (as you would
see in an actual application), SPECglperf artificially assigns
a specific size to every primitive drawn within a test. While
SPECviewperf attempts to emulate what an application would do
graphically and measure it, SPECglperf makes no such attempt.
Instead, SPECglperf provides a more controlled environment within
which to extract and measure the highest performance or "upper
bound" of a particular system.
Another difference is that SPECviewperf reports results in frames
drawn per second, whereas SPECglperf measures its results in primitives
drawn per second, whether the primitive is pixels, points, lines,
triangles or some other object. To give an analogy to the automotive
world, SPECglperf would be the equivalent of a speedometer measuring
top speed, while SPECviewperf would be a stopwatch measuring the
average speed through a slalom course.
Q. How do I look at all these numbers and make sense of
them?
A. A substantial amount of work went into making the visualization
and analysis of SPECglperf data as simple as possible. This goal
is realized in the form of Java applets that read the SPECglperf
performance results from the server's database, and allow tabular
and graphical comparisons on the client's web browser. At the
top level, the SPECglperf visualization tools are broken into
two branches: the OPClist and the other scripts. Going down the
OPClist branch, one can construct a performance comparison by
selecting the desired primitives from one list, and the desired
systems on another list. The other branch allows users to choose
from the different SPECglperf scripts to display the data that
most interests them. Each choice enables a table to be displayed
with graphs on one axis and system configurations on the other.
By selecting different cells, different sets of data will be displayed.
Also, the axes can be linear or a log scale, allowing the most
flexible plotting of data.
Q. What systems are capable of running SPECglperf?
A. Those systems supporting OpenGL drivers on OS/2, Windows
95/98, Windows NT and UNIX derivatives.
This document was prepared by John Spitzer,
former chair of the SPECopc project group.
|
