Skip navigation

Standard Performance Evaluation Corporation

Facebook logo LinkedIn logo Twitter logo
 
 

SPEC Describes SPEC95 Products And Benchmarks

Jeff Reilly
Intel Corporation
Santa Clara, Calif.

Published September, 1995; see disclaimer.


With the SPEC95 suites announced by SPEC this summer, it is important to introduce the product and its contents. Below is an introduction to the SPEC95 product. Upcoming SPEC Newsletter articles will discuss the benchmarks, analysis and results in more detail.

SPEC95 Product

SPEC95 refers to the total SPEC95 product provided by SPEC. SPEC95 is composed of two suites of benchmarks:

SPEC CINT95:
a set of eight compute-intensive integer/non-floating point benchmarks
SPEC CFP95:
a set of 10 compute-intensive floating point benchmarks

These are intended to provide a measure of compute-intensive performance of the processor, memory hierarchy and compiler components (the 'C' in CINT95 and CFP95) of a computer system for comparison purposes. They are not intended to stress the graphics, network or I/O. When you receive the SPEC95 media, it will contain the following:

The SPEC95 benchmarks:
these are applications (referred to as SPEC benchmarks hereafter) provided as source code. These are divided into two separate suites.
CINT95:
eight compute-intensive integer benchmarks.
CFP95:
10 compute-intensive floating point benchmarks.
The SPEC95 tools:
these are executable scripts that are used to compile, run and validate the results of each of the individual SPEC benchmarks. These will be provided in binary format for a variety of operating systems. The source code for these tools will also be provided with directions and scripts for compiling them in the case you are using an environment not covered by the provided binaries.
Run Rules:
Rules for running the benchmarks (needed to ensure fairness and comparability).
Reporting Rules:
Guideline for reporting results.

The procedure for running the benchmarks include:

  • Read/understand the SPEC documentation, including the SPEC Run and Reporting Rules.
  • Compile and create appropriate tool environment.
  • Edit configuration files to include compiler parameters for the compiler to be used (in accordance with the SPEC Run and Reporting Rules.
  • Compile the benchmarks with the SPEC95 tools.
  • Run and validate the benchmarks with the SPEC95 tools.
  • Generate the appropriate individual and combined metrics with the SPEC95 tools.

Benchmark Descriptions

The table below contains a brief description of the benchmarks and their SPEC reference times (used for calculating SPEC metrics). More detailed analysis and descriptions will be provided in future issues of the SPEC Newsletter.

CINT95 Benchmarks

Benchmark Reference Time (Sec) Application Area Specific Task
099.go 4600 Game playing; artificial intelligence Plays the game Go against itself.
124.m88ksim 1900 Simulation Simulates the Motorola 88100 processor running Dhrystone and a memory test program.
126.gcc 1700 Programming & compilation Compiles pre-processed source into optimized SPARC assembly code.
129.compress 1800 Compression Compresses large text files (about 16MB) using adaptive Limpel-Ziv coding.
130.li 1900 Language interpreter Lisp interpreter.
132.ijpeg 2400 Imaging Performs jpeg image compression with various parameters.
134.perl 1900 Shell interpreter Performs text and numeric manipulations (anagrams/prime number factoring).
147.vortex 2700 Database Builds and manipulates three interrelated databases.

CFP95 Benchmarks

Benchmark Reference Time (Sec) Application Area Specific Task
101.tomcatv 3700 Fluid Dynamics / Geometric Translation Generation of a two-dimensional boundary-fitted coordinate system around general geometric domains.
102.swim 8600 Weather Prediction Solves shallow water equations using finite difference approximations. (The only single precision benchmark in CFP95.)
103.su2cor 1400 Quantum Physics Masses of elementary particles are computed in the Quark-Gluon theory.
104.hydro2d 2400 Astrophysics Hydrodynamical Navier Stokes equations are used to compute galactic jets.
107.mgrid 2500 Electromagnetism Calculation of a 3D potential field.
110.applu 2200 Fluid Dynamics/Math Solves matrix system with pivoting.
125.turb3d 4100 Simulation Simulates turbulence in a cubic area.
141.apsi 2100 Weather Predication Calculates statistics on temperature and pollutants in a grid.
145.fpppp 9600 Chemistry Performs multi-electron derivatives.
146.wave 3000 Electromagnetics Solve's Maxwell's equations on a cartesian mesh.

Metrics

So how are these benchmarks used to measure and compare performance? Within the realm of compute-intensive performance, several quantities can be considered. The following categories are part of SPEC95:

  • integer versus floating point.
  • conservative versus aggressive compilation.
  • speed versus throughput.

Based on these choices, the SPEC95 tools allow you to generate the following composite metrics:

    SPEED   THROUGHPUT
Aggressive
    SPECint95 SPECint_rate95
    SPECfp95  SPECfp_rate95
Conservative
    SPECint_base95  SPECint_rate_base95
    SPECfp_base95 SPECfp_rate_base95

For SPEC purposes, the rules for aggressive and conservative/base optimizations are listed extensively in the SPEC Run and Reporting Rules supplied with SPEC95.

Note these are the composite metrics; an individual score is calculated for each benchmark in CINT95 or CFP95 and used to calculate this composite.

For the speed measures, each benchmark has its own SPECratio. A SPECratio is the runtime of the benchmark on the measured system divided into the SPEC reference time.

Mathematically: SPECratio for xxx.benchmark = xxx.benchmark reference time /xxx.benchmark run time.

The composite metrics are then calculated as:

  • SPECint95: The geometric mean of eight SPECratios (one for each integer benchmark) when compiled with aggressive optimizations for each benchmark.
  • SPECint_base95: The geometric mean of eight SPECratios (one for each integer benchmark) when compiled with the conservative optimizations for each benchmark.
  • SPECfp95: The geometric mean of 10 normalized ratios (one for each floating point benchmark) when compiled with aggressive optimization for each benchmark.
  • SPECfp_base95: The geometric mean of 10 normalized ratios (one for each floating point benchmark) when compiled with the conservative optimizations for each benchmark.

For example, the SPECint95 of a system would be: SPECint95 = (099.go SPECratio * 124.m88ksim SPECratio ... * 147.vortex SPEratio) ^ (1/8)

For the throughput measures, each benchmark has its own SPECrate. A SPECrate is a function of the number of copies run, the time it took to complete all of the those copies and a reference factor.

Mathematically: SPECrate for xxx.benchmark = # of copies run * (reference time of xxx.benchmark * the number of seconds in a day/longest SPEC95 Reference Time)/runtime of all xxx.benchmark executions.

The composite metrics are then calculated as: Throughput metrics:

  • SPECint_rate95: The geometric mean of eight normalized SPECrates (one for each integer benchmark) when compiled with aggressive optimization for each benchmark.
  • SPECint_rate_base95: The geometric mean of eight SPECrates (one for each integer benchmark) when compiled with conservative optimizations for each benchmark.
  • SPECfp_rate95: The geometric mean of 10 SPECrates (one for each floating point benchmark) when compiled with aggressive optimization for each benchmark.
  • SPECfp_rate_base95: The geometric mean of 10 SPECrates (one for each floating point benchmark) when compiled with conservative optimizations for each benchmark.

For example, the SPECint_rate95 of a system would be: SPECint95 = (099.go SPECrate * 124.m88ksim SPECrate ... * 147.vortexSPECrate) ^ (1/8)

Jeff Reilly is the Release Manager for SPEC95 and is a Project Lead at Intel Corporation in Santa Clara, Calif.