SPEC subject: SPEC Reporting Rules For date: January 24, 1992 CINT92 from: SPEC Steering Committee ABSTRACT This memorandum describes the Rules for reporting the results of SPEC CINT92 suite. This is a companion document to SPEC Run Rules, which describes the process of running the benchmarks contained in the SPEC Suite. Memorandum to Performance Analysts SPEC subject: SPEC Reporting Rules For date: January 24, 1992 CINT92 from: SPEC Steering Committee 1. SPEC_CINT92_Introduction The SPEC CINT92 benchmark suite consists of CPU intensive benchmarks that are intended to be meaningful samples of applications which perform non-floating point logic and computations in a technical computing environment. SPEC CINT92 does not assess the ability of a system under test to handle disk, graphics, communication, or multi-user activity. Many of the SPEC CINT92 benchmarks have been derived from publicly-available application programs, and they are intended to be portable to as many current and future hardware platforms as possible. Consequently, the version of the programs in this distribution contains no hardware dependencies. Where possible, the programs have been structured to ensure that they do not unfairly favor one hardware architecture over another. For this reason, the application programs in this distribution should not be used to assess the probable performance of commercially available, tuned versions of the same application. Different degrees of tuning and availability of specialized hardware features may make large (and varying) differences between the performance of the programs in this distribution and that of commercial applications for production use. The individual benchmarks in this suite may be similar, but not necessarily identical to benchmarks with the same name which are available from sources other than SPEC. Therefore it may not be valid to compare the individual benchmark results of SPEC CINT92 to any benchmarks not obtained from SPEC. Reporting of results must follow the guidelines established in the "Reporting Guidelines" section of the distribution documentation. It is not intended that the SPEC CINT92 benchmark suite be used as a replacement for the benchmarking of actual customer applications to determine vendor or product selection. - 2 - 2. REPORTING_GUIDELINES This section describes the standard reporting format that must be used when reporting results using the SPEC CINT92 benchmarks to obtain a SPECint92 rating for a system. The SPECint92 rating is used to describe the speed of a system. EXAMPLES OF REPORTING RULES FOR SPEC RESULTS CAN BE SEEN IN THE QUARTERLY SPEC NEWSLETTER. 1 The intent of the SPECint92 Reporting Guidelines is to describe a consistent method of reporting results to aid customers in interpreting and comparing results of different systems' performance. All measured and calculated values are reported to the nearest tenth. Unless noted to the contrary, the report must contain the following items of information: 2.1 Table_of_Run_Times A table of 7 rows and 4 columns structured as follows: a. One row for each individual benchmark in SPEC CINT92 b. Four columns i. Name of benchmark (preferred order (numerical) shown in example below) ii. SPEC Reference Time for each benchmark in the suite. A. 008.espresso: 2270 seconds B. 022.li: 6210 seconds C. 023.eqntott: 1100 seconds D. 026.compress: 2770 seconds E. 072.sc: 4530 seconds __________ 1. Articles describing the suite and metrics can be found in the December 1991 SPEC Newsletter. - 3 - F. 085.gcc: 5460 seconds iii. For SPECint92, Elapsed time (rounded to the nearest 0.1 second) for the System-Under-Test (SUT) taken to run each benchmark of SPEC CINT92 suite. The elapsed times reported are for each benchmark run one at a time. For the SPECint92, the elapsed time value measures the execution of a single copy of each benchmark. The reported times are the best times that can be consistently reproduced. If any changes to an individual benchmark are made for portability purposes (e.g. to obtain a clean compile), the elapsed time should be followed by an "*", and a description of the change should be described in PART 7 of the report. The asterisk is dropped after the portability change has been approved by the SPEC steering committee as a portability change and not a performance change and when this change has been implemented in the tape. 2 iv. For SPECint92, Column 4 has Ratio of SPEC Reference Time to SUT time (divide number in column 2 by number in column 3). The ratio is without dimension, and will be referred to as SPECratio. c. A row with the Geometric mean for the numbers listed in column 4. This row is labeled as "Geom Mean: SPECint92". The Geometric Mean is calculated as the 6th root of the product of the individual values in the SPECratio column. For SPECint92, the Geometric Mean of the SPECratios in column 4 is defined as the SPECint92 of the SUT. Please note: if an "*" sign appears in any of the __________ 2. REFER to portability rules in SPEC run rules document. - 4 - rows to indicate a portability change in a benchmark, the "*" must be displayed after the Geometric Mean calculation for elapsed time and SPECratio. 2.2 Identification_Information This information includes the date (month/year) that the benchmarks were run, the name and location of the company that ran them and the license number of the SPEC tape used. It is typically listed under the table generated in Section 2.1. 2.3 Hardware_Configuration A description of the system (cpu/clock, fpu/clock) number of processors, relevant peripherals, etc are included in this space. It is typically listed next to the table generated in Section 2.1. The amount of information supplied should be sufficient to allow duplication of results by another party. The following checklist is provided to show examples of hardware features which may affect the performance of SPEC CINT92 benchmarks. The checklist is not intended to be all-inclusive, nor is each feature in the list required to be described. The rule of thumb is: "if it affects performance or the feature is required to duplicate the results, describe it". HARDWARE PLATFORM FEATURES o Manufacturer's model number o CPU component ID and clock speed o Floating Point Unit (FPU) ID and clock speed o Number of CPUs and FPUs o Cache Size (per CPU), description and organization o Memory (Amount and description) o Disk subsystem configuration - Number of active connected disks - Disk controllers: ID, number and type - Disk: ID, number and type - 5 - o Network Interface 2.4 Software_Configuration The description of the software configuration used should be detailed enough so that another party could duplicate the results reported. This is typically reported after the hardware decription. The following is a list of possible items that might be documented: o Operating System Type and Release level (with revision/update level if appropriate) o Compiler release levels used. If multiple compilers are available, specify which ones were used. o Optional software required to reproduce results o File system type used o Firmware level 2.5 Systems_Environment This section should document the overall systems environment used to run the benchmark. It is typically listed after the software description. The following is a list of possible items that might be documented: SYSTEMS ENVIRONMENT CHECKLIST o Single or multi-user state o System tuning parameters (e.g. MAKEFS and TUNEFS parameters) o Process tuning parameters (e.g. stack size, time slice) o Background load, if any 2.6 General_Availability_of_the_SUT This item indicates the availability (or projected availability) date for the system or components being benchmarked. If both hardware and software are not available at the same time, list them separately. SPEC - 6 - requires that the availability of hardware and software should be within 6 months of the date of publication of the data. It should be listed in Month/Year format. 2.7 Individual_Benchmark_Notes This section documents ANY changes made to the individual benchmark source code or execution parameters for portability reasons. For example, if difficulty arises in compiling or executing an individual benchmark because of portability problems, the difficulty should be described, and the code changes made to resolve the portability problem documented. This should include module name, line number and actual source code changes. This section must also include any use of alternate source files, alternate input files, or alternate result files, as noted in "Runrules" section of the distribution documentation. 2.8 Graphical_Representation If desired, a graphical summary of the table from Section 2.1 may be included as an additional item of information. If included for SPECint92, it should be shown as follows: a. The 6 benchmarks comprising of SPEC CINT92 suite should appear as individual points on the x-axis ordered by benchmark numbers. b. The SPECratio of each benchmark (obtained from column 4 in the table described above), should be plotted as a vertical bar graph. c. A horizontal line should be drawn across the graph intersecting the y-axis at the SPECint92 geometric mean value. This permits differentiation between each benchmark's SPECratio and the overall SPECint92 geometric mean. 3. SAMPLE_REPORTS - 7 - 3.1 SPECint92_-_Summary_of_SPEC_CINT92_Results __________________________________________________________ | SPEC Benchmark| SPEC | System Under| SPECratio| | CINT92 | Reference | Test | | | | Time Elapsed| (Sample) | | | | Time | Elapsed Time| | | | (seconds) | (seconds) | | |_______________|______________|______________|___________| |_______________|______________|______________|___________| | 008.espresso | 2270 | 1245.0 | 1.8 | |_______________|______________|______________|___________| | 022.li | 6210 | 2275.0 | 2.7 | |_______________|______________|______________|___________| | 023.eqntott | 1100 | 684.0 | 1.6 | |_______________|______________|______________|___________| | 026.compress | 2770 | 1390.0 | 2.0 | |_______________|______________|______________|___________| | 072.sc | 4530 | 783.0 | 5.8 | |_______________|______________|______________|___________| | 085.gcc | 5460 | 1521.0 | 3.6* | |_______________|______________|______________|___________| | Geometric Mean| | | 2.6* | |_______________|______________|______________|___________| *Changes made to benchmark for portability reasons, see PART 7 for details The geometric mean is the nth root of the product of n elements. Therefore, in this case it is the 6th root of the product of 6 SPECratios in the 4th column. 3.2 Identification_Information Tested on mm/yy, tested by xxxxx of (location), SPEC License No. yyyyy. 3.3 Hardware_Configuration Hardware Configuration Detail for "Sample" system o Vendor machine xxxxxxx, model number nnnnnnn o RISC processor xxxxxx; nn MHz clock speed o FPU xxxxxxxx, nn MHz clock speed o One processor/FPU - 8 - o n MB main memory o n disk drives, each nnn megabytes capacity 3.4 Software_Configuration Software Configuration Detail for "Sample" system o Operating System name, Version nnn, update nnn applied o FORTRAN Version nnn o C Version nnn o Other software: (if applicable) o Firmware level: (if applicable) 3.5 General_Availability The harware wil be available mm/yy and the software will be available in nn/yy. Note: When a beta version is used, the date of the final release must be included which must be within 6 months of result publication. 3.6 Systems_Environment Systems Environment Detail for "Sample" system o Tuning Parameters: o Background load: o System state: (Mult-user or single user) 3.7 Notes_On_Individual_Benchmark_Runs - 085.gcc: was compiled using gnu compiler * 085.gcc: changed variable name xxxx to yyyy to avoid library conflict.