Version 1.11
Last updated: 17 Sep 2008 02:00PM cgp
Variations Between MPI2007 Releases
Steps of usage, including troubleshooting
Setting up the run directories
(Click on an item above, to go to the detailed contents about that item.)
Document.01: What is this document for?
Document.02: How is this document organized?
Document.03: Who wrote this document?
Document.04: Where is the latest version of this document?
Document.05: What other documentation is there?
Document.06: Where is the list of known problems with SPEC MPI2007?
Document.07: Now that I have read this document, what should I do next?
SPEC.01: What is SPEC?
SPEC.02: Where do I find more information on SPEC?
SPEC.03: How do I join SPEC?
SPEC.04: How do I contact SPEC for more information or for technical support?
MPI2007.Benchmarking.01: What is a benchmark?
MPI2007.Benchmarking.02: Should I benchmark my own application?
MPI2007.Benchmarking.03: If not my own application, then what?
MPI2007.Benchmarking.04: Some of the benchmark names sound familiar; are these comparable to other programs?
MPI2007.Details.01: What does SPEC MPI2007 measure?
MPI2007.Details.02: Why use SPEC MPI2007?
MPI2007.Details.03: What are the limitations of SPEC MPI2007?
MPI2007.Details.04: What are the limits on the medium data set for SPECmpi2007?
MPI2007.Details.05: What kinds of parallelism can I use with MPI2007?
MPI2007.Details.06: Does SPEC MPI2007 have "speed" runs and "rate" runs?
MPI2007.Details.07: What criteria were used to select the benchmarks?
MPI2007.Details.08: What source code is provided? What exactly makes up these suites?
MPI2007.Details.09: How do benchmarks combine C and Fortran sources?
MPI2007.Details.10: Do I need both C and Fortran compilers for all the combined-language benchmarks?
MPI2007.Details.11: What am I allowed to do with the source codes?
Variations Between MPI2007 Releases
MPI2007.Versions.1: How does Version 1.1 of MPI2007 differ from Version 1.0?
MPI2007.Versions.2: Now that Version 1.1 is released, can I still make submissions with the Version 1.0?
MPI2007.Relatives.01: SPEC CPU2006 and OMP2001 and already available. Why create SPEC MPI2007? Will it show anything different?
MPI2007.Relatives.02: Aren't some of the SPEC MPI2007 benchmarks already in SPEC CPU2006 and HPC2002? How are they different?
MPI2007.Relatives.03: Why were most of the benchmarks not carried over from CPU2006, HPC2002 or OMP2001?
MPI2007.Relatives.04: What happens to SPEC HPC2002 now that SPEC MPI2007 has been released?
MPI2007.Relatives.05: Is there a way to translate measurements of other suites to SPEC MPI2007 results or vice versa?
Steps of usage, including troubleshooting
Usage.GettingStarted.01: What does the user of the SPEC MPI2007 suite have to provide?
Usage.GettingStarted.02: What are the basic steps in running the benchmarks?
Usage.Installation.01: How/where do I get the MPI2007 package?
Usage.Installation.02: How am I notified of updates?
Usage.Installation.03: What is included in the SPEC MPI2007 package?
Usage.Installation.04: How do I install the package?
Usage.Installation.05: Why am I getting a message such as "./install.sh: /bin/sh: bad interpreter: Permission denied"?
Usage.Installation.06: The DVD drive is on system A, but I want to install on system B. What do I do?
Usage.Installation.07: Why did the installation fail with "Error occurred while processing: C:\Documents"?
Usage.Installation.08: How do I uninstall? This script uninstall.sh doesn't seem to be it.
Usage.Installation.09: What if the tools cannot be run or built on a system? Can the benchmarks be run manually?
Usage.Invocation.01: What parameters do I pass to runspec?
Usage.Invocation.02: Can I (or should I) use an old config file?
Usage.Invocation.03: I'm using a config file that came with the MPI2007 image. Why isn't it working?
Usage.Invocation.04: How do I write a config file?
Usage.Invocation.05: Is there a config file for Visual C++?
Usage.Invocation.06: When I say runspec, why does it say "Can't locate strict.pm"?
Usage.Invocation.07: Why am I getting messages about "specperl: bad interpreter: No such file or directory"?
Usage.Building.01: How long does it take to compile the benchmarks?
Usage.Building.02: How much memory do I need to compile the benchmarks?
Usage.Building.03: Can I build the benchmarks as either 32- or 64- bit binaries?
Usage.Building.04: I'm using Windows and I can't build any benchmarks. What does "CreateProcess((null), ifort ...) failed" mean?
Usage.Building.05: The file make.clean.err is mentioned, but it does not exist. Why not?
Usage.Building.06: Why is it rebuilding the benchmarks?
Usage.Building.07: Building one of the benchmark fails. What should I do?
Usage.Building.08: How can I build different .o files using different flags?
Setting up the run directories
Usage.Setup.01: How much file system space do I need?
Usage.Setup.02: Do I need a shared file system?
Usage.Setup.03: What does "hash doesn't match after copy" mean?
Usage.Setup.04: Why does it say "ERROR: Copying executable failed?"
Usage.Running.01: How long does it take to run the SPEC MPI2007 benchmark suites on my platform?
Usage.Running.02: How much memory does it take to run the benchmarks?
Usage.Running.03: Why does this benchmark take so long to run?
Usage.Running.04: The running of one of the benchmark fails. What should I do?
Usage.Running.05: Why did 126.lammps fail?
Usage.Running.06a: Why did I get the error message "Benchmark does not support size 'ref'?
Usage.Running.06b: Why did I get the error message "read_reftime: "/spec/mpi2007/benchspec/MPI2007/*/data/ref/reftime" does not exist"?
Usage.Running.07: How can I run a profiler, performance monitor, etc. in conjunction with a benchmark?
Usage.Validation.01: What is the difference between a valid and a reportable run?
Usage.Validation.02: I got a message about a miscompare...
Usage.Validation.03: How were the result tolerances chosen?
Usage.Validation.04: The benchmark ran, but it took less than 1 second and there was a miscompare. Help!
Usage.Validation.05: The .mis file says short
Usage.Validation.06: My compiler is generating bad code! Help!
Usage.Validation.07: The code is bad even with low optimization! Help!
Usage.Validation.08: I looked in the .mis file and it was just full of a bunch of numbers.
Measurements.Metrics.01: What metrics can be measured?
Measurements.Metrics.02: What is the difference between a base metric and a peak metric?
Measurements.Metrics.03: What are the different data-set sizes?
Measurements.Metrics.04: Which SPEC MPI2007 metric should be used to compare performance?
Measurements.Timing.01: Why does SPEC use a reference machine? What machine is used for SPEC MPI2007?
Measurements.Timing.02: How long does it take to run the SPEC MPI2007 benchmark suites on the reference platform?
Measurements.Timing.03: The reports don't list the reference times. Where can I find them?
Measurements.Timing.04: Why aren't the scores for the reference platform all 1's?
Measurements.Timing.05: I bought a machine from company XYZ. It doesn't score as high as what was published...
Measurements.Timing.06: The machines we're shipping do not yield the same measurements as what we published...
Measurements.Timing.07: How was the 5% tolerance decided?
Measurements.Reporting.01: Where are SPEC MPI2007 results available?
Measurements.Reporting.02: How do I submit my results?
Measurements.Reporting.03: How do I edit a submitted report?
Measurements.Reporting.04: What do I do if a flag or library changes in the GA?
Measurements.Reporting.05: What's all this about "Submission Check -> FAILED" littering my log file and my screen?
Measurements.Reporting.06: What is a "flags file"? What does "Unknown Flags" mean in a report?
Measurements.Reporting.07: Can SPEC MPI2007 results be published outside of the SPEC web site? Do the rules still apply?
Measurements.Reporting.08: Can I use "derived" metrics, such as cost/performance or flops?
Measurements.Reporting.09: It's hard to cut/paste into my spreadsheet...
This document answers Frequently Asked Questions and provides background information about the SPEC MPI2007 benchmark suite, as well as providing usage and troubleshooting details. SPEC hopes that this material will help you understand what the benchmark suite can, and cannot, provide; that it will serve as a shortcut reference for following the process and solving problems; and that it will help you make efficient use of the product.
This document is organized as a series of questions and answers. It follows the natural process flow from the principles of SPEC and the MPI2007 benchmarks, to the steps of benchmarking a real system (including dealing with problems you may run into), and finally what to do with the benchmark measurements. Hopefully this arrangement will allow you to find your question in the list by following the categories.
This document is the merger of earlier "faq" and "readme1st" documents developed for SPEC/CPU2006, which in turn had evolved through earlier releases of SPEC/cpu. Contributors include
The latest version of this document may be found at http://www.spec.org/mpi2007/Docs/faq.html.
The website http://www.spec.org/mpi2007/Docs/ contains links to all the MPI2007 documents, describing the benchmark programs, how to install and run them, and technical support and other documents.
On an installed machine, the Docs directory contains copies of these documents that are current at the time the MPI2007 image was packaged.
Please see the link http://www.spec.org/mpi2007/Docs/errata.html for an updated list of known problems with the SPEC MPI2007 suite.
If you haven't bought MPI2007, it is hoped that you will consider doing so. If you are ready to get started using the suite, then you should pick a system that meets the requirements as described in
and install the suite, following the instructions in
or
Next, read runrules.html on how to perform a reportable run. The runspec.html shows how to execute the benchmarks and config.html shows how to construct a configuration file that controls the compilation and execution of the benchmarks.
SPEC is the Standard Performance Evaluation Corporation. SPEC is a non-profit organization whose members include computer hardware vendors, software companies, universities, research organizations, systems integrators, publishers and consultants. SPEC's goal is to establish, maintain and endorse a standardized set of relevant benchmarks for computer systems. Although no one set of tests can fully characterize overall system performance, SPEC believes that the user community benefits from objective tests which can serve as a common reference point.
The website http://www.spec.org/ contains information on membership, the benchmark suites, and the posted benchmark results.
The website http://www.spec.org/spec/membership.html describes the SPEC organization and how to join each of the component committees.
SPEC can be contacted in several ways. For general information, including other means of contacting SPEC, please see SPEC's Web Site at:
General questions can be emailed to:
info@spec.org
MPI2007 Technical Support Questions can be sent to:
mpi2007support@spec.org
A benchmark is "a standard of measurement or evaluation" (Webster’s II Dictionary). A computer benchmark is typically a computer program that performs a strictly defined set of operations - a workload - and returns some form of result - a metric - describing how the tested computer performed. Computer benchmark metrics usually measure speed: how fast was the workload completed; or throughput: how many workload units per unit time were completed. Running the same computer benchmark on multiple computers allows a comparison to be made.
Ideally, the best comparison test for systems would be your own application with your own workload. Unfortunately, it is often impractical to get a wide base of reliable, repeatable and comparable measurements for different systems using your own application with your own workload. Problems might include generation of a good test case, confidentiality concerns, difficulty ensuring comparable conditions, time, money, or other constraints.
You may wish to consider using standardized benchmarks as a reference point. Ideally, a standardized benchmark will be portable, and may already have been run on the platforms that you are interested in. However, before you consider the results you need to be sure that you understand the correlation between your application/computing needs and what the benchmark is measuring. Are the benchmarks similar to the kinds of applications you run? Do the workloads have similar characteristics? Based on your answers to these questions, you can begin to see how the benchmark may approximate your reality.
Note: A standardized benchmark can serve as reference point. Nevertheless, when you are doing vendor or product selection, SPEC does not claim that any standardized benchmark can replace benchmarking your own actual application.
Many of the SPEC benchmarks are derived from publicly available application programs. The individual benchmarks in this suite may be similar, but are NOT identical to benchmarks or programs with similar names which may be available from sources other than SPEC. In particular, SPEC has invested significant effort to improve portability and to minimize hardware dependencies, to avoid unfairly favoring one hardware platform over another. For this reason, the application programs in this distribution may perform differently from commercially available versions of the same application.
Therefore, it is not valid to compare SPEC MPI2007 benchmark results with anything other than other SPEC MPI2007 benchmark results.
SPEC MPI2007 focuses on performance of compute intensive applications using the Message-Passing Interface (MPI), which means these benchmarks emphasize the performance of:
It is important to remember the contribution of all these components. SPEC MPI performance intentionally depends on more than just the processor.
SPEC MPI2007 is not intended to stress other computer components such as the operating system, graphics, or the I/O system. They may have an effect in some cases, if the component is exceptionally slow or if there is an unusually small resource bound, e.g. paging delays due to a slow disk or too little memory. Note that there are many other SPEC benchmarks, including benchmarks that specifically focus on graphics, distributed Java computing, webservers, and network file systems.
SPEC MPI2007 provides a comparative measure of MPI-parallel, floating point, compute intensive performance, across the widest practical range of cluster and SMP hardware. If this matches with the type of workloads you are interested in, SPEC MPI2007 provides a good reference point.
Other advantages to using SPEC MPI2007 include:
As described above, the ideal benchmark for vendor or product selection would be your own workload on your own application. Please bear in mind that no standardized benchmark can provide a perfect model of the realities of your particular system and user community.
The data set was intended to work with a range of rank sizes from 4 to 128 ranks. Valid runs up to 512 ranks have been reported but some workloads, notably GemsFDTD, do not scale positively beyond 256 ranks. The large data set is intended to work with larger rank sizes, starting from 64 ranks and going beyond 512 ranks. At this time, that data set is still in development.
The benchmarks utilize process-level parallelism as managed by the MPI standard. Thread-level parallelism is not allowed:
Note that this is a deviation from CPU2006, OMP2001, and HPC2002. Low-level hardware parallelism is allowed to the extent that it does not require parallel software threads.
A SPEC/CPU benchmark is intended to run on a single processor core. On multiprocessor systems, a "rate" run consists of multiple copies of the benchmark running on the system concurrently. This measures the total system computing capacity, as opposed to a "speed" run which only runs a single copy on an otherwise empty system.
SPEC/HPG benchmarks are parallel programs, so only one copy of the benchmark runs on the system at a time. The program itself uses multiple processor cores. A SPEC OMP2001 benchmark decomposes into multiple threads under the OpenMP standard, while a SPEC MPI2007 benchmark decomposes into multiple processes under the MPI standard. These have characteristics common with both "speed" and "rate" runs but does not precisely correspond to either one.
Note that an OMP2001 or MPI2007 benchmark usually runs faster on larger systems, since the problem is divided into smaller pieces which execute more quickly, while with a SPEC CPU "rate" run, a benchmark tends to take longer to run on larger systems since more copies of the same program tend to cause contention for system resources.
In the process of selecting applications to use as benchmarks, SPEC considered the following criteria:
Note that not every benchmark satisfies every criterion. 122.tachyon, for example, is "embarassingly parallel". Also, the total memory requirement seems to have a lower bound of 16GB even when there are fewer than 16 ranks being run.
MPI2007 is composed of MPI-parallel compute-intensive applications provided as source code. 2 are written in C, 4 are written in Fortran, 6 combine C and Fortran, and 1 is written in C++. The benchmarks are:
| 104.milc | C | Physics: Quantum Chromodynamics (QCD) |
| 107.leslie3d | Fortran | Computational Fluid Dynamics (CFD) |
| 113.GemsFDTD | Fortran | Computational Electromagnetics (CEM) |
| 115.fds4 | C/Fortran | Computational Fluid Dynamics (CFD) |
| 121.pop2 | C/Fortran | Climate Modeling |
| 122.tachyon | C | Graphics: Parallel Ray Tracing |
| 126.lammps | C++ | Molecular Dynamics Simulation |
| 127.wrf2 | C/Fortran | Weather Prediction |
| 128.GAPgeofem | C/Fortran | Heat Transfer using Finite Element Methods (FEM) |
| 129.tera_tf | Fortran | 3D Eulerian Hydrodynamics |
| 130.socorro | C/Fortran | Molecular Dynamics using Density-Functional Theory (DFT) |
| 132.zeusmp2 | C/Fortran | Physics: Computational Fluid Dynamics |
| 137.lu | Fortran | Matrix Decomposition |
Descriptions of the benchmarks, with reference to papers, web sites, and so forth, can be found in the individual benchmark descriptions (click the links above). Some of the benchmarks also provide additional details, such as documentation from the original program, in the nnn.benchmark/Docs directories in the SPEC benchmark tree.
The numbers used as part of the benchmark names provide an identifier to help distinguish programs from one another. For example, some programs in SPEC CPU2006 derive from the same sources and need to be distinguished from the MPI2007 versions. Note: even if a program has the same name as in another suite - for example, 127.wrf2 vs. 361.wrf_m from the HPC2002 suite - the updated workload and updated source code mean that it is not valid to compare the SPEC MPI2007 result to the result with the older SPEC HPC2002 benchmark.
In the combined-language benchmarks, most of the C files are used to implement I/O and memory-allocation utilities, or specific mathematical calculations like FFT, Bessel functions, or random number generation. But fundamental pieces of 115.fds4, 127.wrf2, 128.GAPgeofem, and 130.socorro are also written in C. In 121.pop2, the C code is limited to the netcdf library, which is also used in 127.wrf2.
You will only be able to study a few of the benchmarks if you do not have compilers for both C and Fortran. Of the combined-language benchmarks, you may be able to compile the Fortran files in 121.pop2 and link them with a precompiled netcdf library on your system. The C components cannot not replaced in the other combined-language benchmarks. It is important to note that for a reportable MPI2007 result, the SPEC MPI2007 Run and Reporting Rules (a) do not allow the netcdf library substitution, and (b) require that all of the benchmarks in the suite be compiled and run.
The SPEC license allows you to compile and run the benchmarks. Source code changes are not allowed, beyond the use of src.alt patches that are specially approved and packaged by SPEC. If you are interested in improving the codes or re-engineering them for your own purposes, contact the individual benchmark authors as identified in the HTML description file under benchspec/MPI2007/*/Docs for each benchmark.
See the Changes document for the list of differences between Version 1.0 and 1.1. To summarize:
As noted in the question above, MPI2007 Version 1.1 contains a number of fixes to the benchmarks, tools and documentation. Version 1.0 is obsolete and is no longer meant to be used. The SPEC HPG recognizes that it takes some time to distribute the new version and each user requires some time to migrate their run procedures to the new tools and conventions; a 3 month window is provided (10/01/2008 - 12/31/2008) after which no submissions using the Version 1.0 distribution will be accepted.
If you are in a position where you could make valid runs with Version 1.0 but not Version 1.1, the SPEC HPG needs to be notified.
Technology is always changing. As the technology progresses, the benchmarks have to adapt to this. SPEC needed to address the following issues:
Application type:
Many native MPI-parallel applications have been developed for cluster systems and are widely used in industry and
academia. SPEC feels that standard benchmarks need to be available for comparing cluster systems for the same reason that
SPEC CPU was developed for measuring serial CPU performance and SPEC OMP was developed for measuring SMP performance.
Moving target:
OMP2001 has been available for SMP systems for six years. In the meantime, clusters have become increasingly
popular as a low-cost and flexible alternative to configure parallel systems. The HPC2002 suite allows MPI parallelism but is
not an adequate bridge because it only contains 3 benchmarks, has too short of a runtime, and is not as strictly
standardized as MPI2007 or OMP2001.
Application size:
As applications grow in complexity and size, older suites become less representative of what runs on current systems.
For MPI2007, SPEC included some programs with both larger resource requirements and more complex source code than previous
suites.
Run-time:
As of spring, 2007, many of the OMP2001 benchmarks are finishing in less than 5 minutes on leading-edge
processors/systems. Small changes or fluctuations in system state or measurement conditions can therefore have
significant impacts on the percentage of observed run time. SPEC chose to make run times for the CPU2006 and MPI2007 benchmarks longer
to take into account future performance and prevent this from being an issue for the lifetime of the suites.
Some of the benchmarks in CPU2006 and MPI2007 derive from the same source codes. CPU2006 benchmarks are serialized versions of the applications while the corresponding MPI2007 benchmarks preserve their original MPI-parallel nature. They all have been given different workloads to better excercise a large parallel machine, and in some cases have additional source-code modifications for parallel scalability and portability. Therefore, for example, results with the CPU2006 benchmark 433.milc may be strikingly different from results with the MPI2007 benchmark 104.milc.
One benchmark, 127.wrf2, is derived from a more current version of the WRF source than is 361.wrf_m from the earlier HPC2002 suite. Its larger workload is better for excercising the performance of modern parallel systems. Further, the HPC2002 rules allow the use of OMP parallelism, while this capability has been removed from the MPI2007 source code. So, again, results with the HPC2002 benchmark 361.wrf_m may be strikingly different from results with the MPI2007 benchmark 127.wrf2.
Many applications in the CPU2006 suite were not designed to run with MPI parallelism, so would not realistically measure MPI performance. The benchmarks in the HPC2002 and OMP2001 suites either
The HPC2002 suite had been retired in 2008. No further results will be accepted for publication. The MPI2007 suite does a better job of measuring and standardizing MPI performance.
There is no formula for converting CPU2006, OMP2001 or any other measurements to MPI2007 results and vice versa; they are different products. We expect some correlation between any two given suites, i.e. machines with higher results with one suite tend to have higher results with another suite, but there is no universal formula for all systems.
SPEC encourages SPEC licensees to publish MPI2007 numbers on older platforms to provide a historical perspective on performance.
Briefly, you need an SMP or cluster system, running Unix, Linux or Microsoft Windows with compilers; a minimum of 1GB of free memory per rank, although more may be required, with an overriding minimum of 16GB for the whole system; and 5.2GB of free disc space. For cluster configurations, the file system will need to be shared across the nodes. See the system-requirements.html document for the detailed requirements as well as a listing of systems that have successfully run the benchmarks.
Note: links to SPEC MPI2007 documents on this web page assume that you are reading the page from a directory that also contains the other SPEC MPI2007 documents. If by some chance you are reading this web page from a location where the links do not work, try accessing the referenced documents at one of the following locations:
Installation and use are covered in detail in the SPEC MPI2007 User Documentation. The basic steps are:
If you wish to generate results suitable for quoting in public, you will need to carefully study and adhere to the run rules.
If you're reading this from the Docs directory on your system, then the MPI2007 package is likely to be installed already. If the package is not installed, and likely you're reading this from the SPEC public website, you will need to (a) contact your company HPG representative for the DVD or tarfile, (b) download the tarfile from the SPEC members' website, or (c) order the DVD from SPEC.
If your company is not already a member of SPEC/HPG, you will need to join or specially purchase a copy of the DVD, as described on the SPEC public website.
You company SPEC/HPG representative will be following the hpgmail mailing-list and will receive notices of any new editions of the MPI2007 benchmark kit. You will need to make arrangements with her/him to make sure you have access to the latest kit and any possible patches.
Alternately, you can check the SPEC members' website to see if any new version or patches have been posted. This may not tell you if any new kit or patch will soon come available, however, and you may require it for technical reasons, so it is best to keep informed of such activity from within the HPG committee.
SPEC provides the following on the SPEC MPI2007 media (DVD):
If you are installing from the DVD sent to you by SPEC, use the following steps:
su - spec # Become whatever ID will be used to make the runs. mkdir -p /spec/mpi2007 # Create the directory to put the tree. cd /spec/mpi2007 # Go there. cp -R /mount/dvd/* . # Copy the directory tree from wherever the DVD is mounted. ./install.sh # Run the installer script. Use path /spec/mpi2007? yes # Tell it where to put the directory tree.
If you are installing from a tar-file downloaded from the SPEC website, use the following steps: bunzip2/tar ./install.sh . ./shrc
su - spec # Become whatever ID will be used to make the runs. mkdir -p /spec/mpi2007 # Create the directory to put the tree. cp $SPEC_KITS/mpi2007.tar.bz2 . # Bring over the file. bunzip2 mpi2007.tar.bz2 # Uncompress it. tar -xvf mpi2007.tar # Unroll the directory tree. ./install.sh # Run the installer script. Use path /spec/mpi2007? yes # Tell it where to the directory tree is.
If you are installing from the DVD, check to be sure that your operating system allows programs to be executed from the DVD. For example, some Linux man pages for mount suggest setting the properties for the CD or DVD drive in /etc/fstab to "/dev/cdrom /cd iso9660 ro,user,noauto,unhide", which is notably missing the property "exec". Add exec to that list in /etc/fstab, or add it to your mount command. Notice that the sample Linux mount command in install-guide-unix.html does include exec.
Perhaps install.sh lacks permission to run because you tried to copy all the files from the DVD, in order to move them to another system. If so, please don't do that. There's an easier way. See the next question.
The installation guides have an appendix just for you, which describe installing from the network or installing from a tarfile. See the appendix to install-guide-unix.html or install-guide-windows.html.
C:\Documents and Settings\John\mpi2007> install The environment variable SPEC should point to the source of the SPEC distribution as an absolute path. I will now try to set the variable for you... SPEC is set to C:\Documents and Settings\John\mpi2007 If this is NOT what you want, press control-C Press any key to continue . . . Installing from "C:\Documents and Settings\John\mpi2007" Checking the integrity of your source tree... Depending on the amount of memory in your system, and the speed of your destination disk, this may take more than 10 minutes. Please be patient. The system cannot find the file specified. Error occurred while processing: C:\Documents. The system cannot find the file specified. Error occurred while processing: and. The system cannot find the path specified. C:\Documents and Settings\John\mpi2007\tools\bin\windows-i386\specmd5sum: MANIFEST.tmp: no properly formatted MD5 checksum lines found Package integrity check failed! Installation NOT completed!
The problem is that the SPEC tools do NOT support spaces in path names. Sorry. This is a limitation of the SPEC toolset and there are currently no plans to change this requirement. Please use a path that does not contain spaces.
You are correct that uninstall.sh does not remove the whole product; it only removes the SPEC tool set, and does not affect the benchmarks (which consume the bulk of the disk space). At this time, SPEC does not provide an uninstall utility for the suite as a whole. But it's easy to do: on Unix systems, use rm -Rf on the directory where you installed the suite: e.g.
rm -Rf /home/cs3000/saturos/spec/mpi2007
On Windows systems, select the top directory in Windows Explorer and delete it.
If you have been using the output_root feature, you will have to track those down separately.
Note: instead of deleting the entire directory tree, some users find it useful to keep the config and result subdirectories, while deleting everything else.
To generate rule-compliant results, an approved toolset must be used. If several attempts at using the SPEC-provided tools are not successful, you should contact SPEC for technical support. SPEC may be able to help you, but this is not always possible -- for example, if you are attempting to build the tools on a platform that is not available to SPEC.
If you just want to work with the benchmarks and do not care to generate publishable results, SPEC provides information about how to do so.
The runspec document explains all the parameters available to runspec. Some important examples are the following:
Many of these parameters can also be hard-coded into the config file that is passed into runspec. Note that some of them, such as running peak without base, will result in a non-reportable run.
This is a risky issue. The SPEC/HPG committee will require a submitted rawfile to accurately and completely describe the system under test. A config file used to measure a different system is unlikely to contain a correct description of the system you are measuring. So expect to have to correct the details by (a) editing the config-file prior to the run, (b) editing the resulting rawfile prior to submitting it, or (c) editing the pending submission file before it is accepted for publication by the committee.
The real risk is that the config-file contains compilation flags or environment settings which are disallowed for a base run, in which case the whole measurement is invalid and cannot be corrected by editing the result file. In this regard, you can generally trust a config-file that has passed review in a prior submission. It is possible though, that in the time since that prior measurement, your compiler or OS has changed such that some portability flag or environment setting is no longer necessary, and a new measurement from that config-file will not pass review.
The config subdirectory contains config files that were valid at the time the MPI2007 directory tree was packaged for delivery. There is one config file for each platform that MPI2007 was tested on. These are meant to serve as examples of use on each platform and are used for testing builds of the MPI2007.
Even if you are using the same series of compiler and/or OS as the config file in the directory, these components may have evolved since the config file was written, and portability settings in the config file may no longer be necessary so the config file cannot be used to make a valid submission. Even if all the portability settings are correct for your system, it is likely that the choice of optimization settings will not be ideal for your system.
Since these example config files are risky to use, runspec will print a warning that you may be using a stale config-file, and pause briefly to give you a chance to stop it from proceeding:
============================================================================= Warning: You appear to be using one of the config files that is supplied with the SPEC MPI2007 distribution. This can be a fine way to get started. Each config file was developed for a specific combination of compiler / OS / hardware. If your platform uses different versions of the software or hardware listed, or operates in a different mode (e.g. 32- vs. 64-bit mode), there is the possibility that this configuration file may not work as-is. If problems arise please see the technical support file at http://www.spec.org/mpi2007/Docs/techsupport.html A more recent config file for your platform may be among result submissions at http://www.spec.org/mpi2007/ Generally, issues with compilation should be directed to the compiler vendor. You can find hints about how to debug problems by looking at the section on "Troubleshooting" in http://www.spec.org/mpi2007/Docs/config.html This warning will go away if you rename your config file to something other than one of the names of the presupplied config files. ==================== The run will continue in 30 seconds ====================
If one is available, start with a config file used with the same compiler and OS in a successful submission. Read the config-file document (reference?) to understand line-by-line how it works. Then read the compiler, library and OS documentation to see what each setting does, and compare this with the run rules document (reference?) to make sure that the arrangement is allowed.
Once you inderstand these details, you will be able to modify the config file so that it (a) works and (b) is compliant with the SPEC/MPI2007 submission rules.
Users of Microsoft Visual C++ should please note that there are 2 sample config files that can be used as a starting point for your use. One for Intel and VisualC++, the other for PGI and VisualC++. These starting points can be found in config\hpmpi-win-vs-intel.cfg and config\hpmpi-win-vs-pgi.cfg.
D:\mpi2007>runspec --help Can't locate strict.pm in @INC (@INC contains: .) at D:\mpi2007\bin\runspec line 62. BEGIN failed--compilation aborted at D:\mpi2007\bin\runspec line 62.
You can't use runspec if its path is not set correctly. On Unix, Linux, or Mac OS X, you should source shrc or cshrc, as described in runspec.html section 2.4. For Windows, please edit shrc.bat and make the adjustments described in the comments. Then, execute that file, as described in runspec.html section 2.5.
bash: /mpi2007newdir/bin/runspec: /mpi2007/bin/specperl: bad interpreter: No such file or directory
Did you move the directory where runspec was installed? If so, you can probably put everything to rights, just by going to the new top of the directory tree and typing "bin/relocate".
For example, the following unwise sequence of events is repaired after completion of the final line.
Top of SPEC benchmark tree is '/mpi2007' Everything looks okay. cd to /mpi2007, source the shrc file and have at it! $ cd /mpi2007 $ . ./shrc $ cd .. $ mv mpi2007 mpi2007newdir $ runspec -h | head bash: runspec: command not found $ cd mpi2007newdir/ $ . ./shrc $ runspec --help | head bash: /mpi2007newdir/bin/runspec: /mpi2007/bin/specperl: bad interpreter: No such file or directory $ bin/relocate
Compile times can vary markedly between compilers offering different degrees of optimization. Likewise, the choice of compiler flags will affect compile times. The reference cluster takes about 1 hour to build the base versions of the MPI2007 benchmark executables.
Likewise, the space requirement can markedly between compilers and the degree of optimization that is applied.
For the Medium input set, we expect the individual ranks to run within 2GB of memory per rank, given at least 4 ranks. On some systems, this is sufficient to fit in a 32-bit address space so the binaries can be compiled as 32-bit. Note that on some of the benchmarks, as you increase the number of ranks, the memory requirement decreases in each of the ranks, so you may be able to run a benchmark as a 32-bit binary by adding more nodes to the system.
You can choose to compile your peak binaries as 32- or 64- bit individually for each benchmark. For base you will need to compile all benchmarks of the same language the same way, unless you can demonstrate that this is not possible. In this case 64-bits must be chosen as the default, and benchmarks which cannot be run 64-bit can be compiled 32-bit.
Error with make 'specmake build': check file 'D:\mpi2007\benchspec\MPI2007\410.bwaves\run\build_base_mpi2007.win32.fast.exe.0000\make.err' Error with make! *** Error building 410.bwaves ----- make.err: process_begin: CreateProcess((null), ifort -c -Foblock_solver.obj -Ox block_solver.f, ...) failed. make (e=2): The system cannot find the file specified. specmake: *** [block_solver.obj] Error 2
This CreateProcess failure occurs on Windows when specmake cannot find your compiler. (Recall from system-requirements.html that the benchmarks are supplied in source code form, and that they must be compiled.)
The reason that it cannot find your compiler is, most likely, because:
To fix your problem, investigate and address both items.
In shrc.bat, you need to either:
Reference a vendor-supplied file that sets the path.
The supplied shrc.bat mentions sample .bat files, such as c:\program files\microsoft visual studio .NET 2003\Vc7\Bin\vcvars32.bat (Visual C++) and c:\program files\Intel\Compiler\C++\9.1\IA32\Bin\iclvars.bat (Digital Visual Fortran), but other compilers also sometimes provide similar .bat files. If you want to use this option, but xxvars.bat isn't where it is shown in the example in shrc.bat, you might try searching your hard drive for *vars*bat.
As of August, 2006 the following was observed working for Visual C++ (Express edition), but your
paths may be different depending on version and installation options chosen:
call "c:\program files\microsoft visual studio 8\Common7\Tools\vsvars32.bat"
Notice above that the reference in this case is to
vsvars32.bat, not
vcvars32.bat.
Or, edit the path yourself.
Notice the PGI example in the install guide. If you want to use this option, but you can't figure out what path to set, try looking in the documentation for your compiler under topics such as "setting the path", or "command line", or "environment variables". The documentation should mention whether any other environment variables (such as LIB or INCLUDE) are required, in addition to PATH.
You must also use a config file that is appropriate for your compiler.
Microsoft C++ users: please note that the config file windows-ia32-icl.cfg isn't really appropriate for you, because the Intel compiler has additional switches that the Microsoft compiler does not recognize, and it spells the compiler name differently ("icl" vs. "cl"). If you use the Intel C++ config file as a starting point, read the comments in the config file, and be prepared to make adjustments.
Building 104.milc mref base mpi2007.win32.fast.exe default: (build_base_mpi2007.win32.fast.exe.0000) Error with make.clean 'specmake clean': check file 'D:\mpi2007\benchspec\MPI2007\104.milc\run\build_base_mpi2007fast.exe.0000/make.clean.err' *** Error building 104.milc If you wish to ignore this error, please use '-I' or ignore errors.But on investigation, the file make.clean.err does not exist. Why not?
A missing .err file has been reported on Microsoft Windows if a path from shrc.bat uses quoted strings that include semicolons within the quotes, such as:
set PATH="%PATH%;d:\My Compiler Path\bin" <--- wrong set PATH="%PATH%;d:\mycompiler\bin" <--- wrong
Paths such as the above may appear to work if you enter them at the command line. But when the tools call CreateProcess, they cause mysterious-seeming failures. The correct way to do it is:
set PATH=%PATH%;"d:\My Compiler Path\bin"
or
set PATH=%PATH%;d:\mycompiler\bin
Notice that in both cases, there are no semicolons inside of quotes.
You changed something, and the tools thought that it might affect the generated binaries. See the discussion of unknown options in config.html.
The building of a benchmark is done using a version of the make tool adapted by SPEC. This tool is called specmake. If the building of a benchmark fails, you can list the actual commands performed by specmake. This can help you in finding the reason of the build failure. How to do this is described in the utility.html document, in the section describing specmake.
The SPEC tools will apply the same set of compilation flags to each source file of a given language, consistent with the MPI2007 Run Rules. For the purpose of debugging, however, you may need to circumvent this and compile specific .o files with flags different from the ones inherited from the config-file. Note that this cannot be done in a reportable run, so be sure not to use the "dirty" executables beyond your debugging effort.
%$(OBJ): %.cpp $(CXXC) $(CXXOBJOPT) $(FINAL_CXXFLAGS) $< ifdef NEEDATFILE $(ECHO) $@ >> objectnames endif angle.o: angle.cpp $(CXXC) $(CXXOBJOPT) $(OPTIMIZE) -DSPEC_MPI -DNDEBUG -DFFT_NONE -O5 -qipa=noobject -qipa=threads -q64 $(PORTABILITY) $(CXXPORTABILITY) $< ifdef NEEDATFILE $(ECHO) $@ >> objectnames endifThe first form of the rule applies generically to all files with the .cpp extension; the second form applies specifically to the angle.cpp source file and its corresponding .o file. The compilation rule is an "unrolling" of the FINAL_CXXFLAGS and CXXOPTIMIZE variables, which are composed from the optimization and portability flags from the config file and the portability flags from the benchspec/MPI2007/< benchmark>/Spec/object.pm file. Note that different benchmarks may use the same file names, e.g. main.f, so you should only debug one benchmark at a time with this method.
5.2GB of free disc space should be enough to contain the SPEC tree, your binaries, and all the intermediate files from the compile and run. If you build more than one set of binaries, you will need additional space for them and any runs you make with them. See the system-requirements.html document for the detailed requirements.
The MPI2007 run rules require that the benchmarks be run inside a single file system (ref???). Running within a single OS partition on an SMP, nearly any file system will do. When running across a cluster, real or virtual, this single filesystem will need to be shared across all the nodes or partitions.
hash doesn't match after copy ... in copy_file (1 try total)! Sleeping 2 seconds...followed by several more tries and sleeps. Why?
During benchmark setup, certain files are checked. If they don't match what they are expected to, you might see this message. Check:
If the condition persists, try turning up the verbosity level. Look at the files with other tools; do they exist? Can you see differences? Try a different disk and controller. And, check for the specific instance of this message described in the next question.
ERROR: Copying executable to run directory FAILEDor
ERROR: Copying executable from build dir to exe dir FAILED!along with the bit about hashes not matching from the previous question. Why?
Perhaps you have attempted to build the same benchmark twice in two simultaneous jobs.
On most operating systems, the SPEC tools don't mind concurrent jobs. They use your operating system's locking facilities to write the correct outputs to the correct files, even if you fire off many runspec commands at the same time.
But there's one case of simultaneous building that is difficult for the tools to defend against: please don't try to build the very same executable from two different jobs at the same time. Notice that if you say something like this:
$ tail myconfig.cfg 445.gobmk=peak: basepeak=yes $ runspec --config myconfig --size mtest --tune base 445.gobmk & $ runspec --config myconfig --size mtest --tune peak 445.gobmk &
then you are trying to build the same benchmark twice in two different jobs, because of the presence of basepeak=yes. Please don't try to do that.
This depends on the data set size, the compiler, and the machine that is running the benchmarks. The reference cluster was sold circa 2003 and is correspondingly slower than contemporary machines, so expect a 2 iteration base run of the medium workload to take less than 24 hours on a 16-core system.
Expect larger data set sizes to take longer to process than the medium sized data set.
For the Medium data set, the intention is that each benchmark in the suite be runnable with a minimum of (1) 16GB of memory for the whole system, or (2) a 1GB of memory per rank, whichever is larger. If you run one rank per processor core, which is the usual case, the second requirement implies 1GB of memory per enabled core on the system.
You may find that 1GB per rank is not sufficient on your system because of other demands on memory from the OS or libraries, or that the arrangement of the process images is fragmented in memory. We expect that 2GB of memory per rank, in addition to the 16GB system minimum, should be sufficient in all cases.
See the system-requirements.html document for a detailed list of individual resource requirements.
Please understand that the suite has been designed to be useful for at least 5 years. Benchmark that seem slow today probably will not seem slow at the end of life of the suite. You can see a bit more on this topic regarding the relationship between MPI2007 and CPU2006 & OMP2001.
On the flipside -- if the benchmark is taking much longer than you think it should, check that the rank counts are being set correctly. If the rank is not set, you may be running the benchmark as a single process, which won't work for some of them and will make the others take an extraordinary amount of time to finish.
The running of a benchmark is done using a SPEC tool called specinvoke. When something goes wrong, it is often useful to try the commands by hand. Ask specinvoke what it did, by using the -n switch. This is described in the utility.html document, in the section describing specinvoke.
The SPEC/HPG committee is aware of a bug with 126.lammps, likely within the MPI middleware or C run-time library, or the application itself. The symptom of this behavior is an entry like
ERROR on proc 52: Failed to reallocate 720000 bytes for array atom:x
in one of the output files
friction.out.big chain.out.big chute.out.big eam.out.big lj.out.big largeeam.out.big
This error message is followed by an MPI specific error message which varies with different MPI implementations.
The allocation failure happens in spite of the machine having sufficient free memory, indicating that this is a software bug. The SPEC High Performance Group has tested the application extensively with a memory allocation consistency checker, without detecting any such erroneous application behavior, suggesting that the inconsistency is contained within the MPI middleware or C run-time library rather than the application code. The failure might also depend on interactions between the compiler and the libraries.
The failure has only manifested itself using MPI implementations or C run-time libraries which include ptmalloc, a dynamic memory allocation library, and appears to go away with the current version ptmalloc3. The earlier ptmalloc2 library - possibly in combination with certain parameter settings from the run-time environment - contains, with a high probability, the source of this allocation failure.
Since we have a high degree of confidence that the failure is contained outside the application sources and toolset distributed by SPEC, we advise any user experiencing this failure to report it to the MPI vendor in question or the internet community dealing with the MPI implementation, whichever is appropriate.
Version 1.0 of MPI2007 contained a data set with three elements: test, train, and ref. The ref element was to be used in reportable runs and the test and train elements were to be used to test the binaries and the tools. Version 1.1 of MPI2007 renamed these elements to mtest, mtrain, and mref since they compose the medium sized data set. Version 2.0 of MPI2007 will contain a large sized data set with the corresponding ltest, ltrain, and lref elements.
With the current version of MPI2007, if you use the ref element as a parameter to runspec, i.e.
runspec -c ... -i ref .... 104.milc
it will issue the error message
Benchmark '104.milc' does not support size 'ref'
indicating that there is no such input defined. You should be using mref or lref as the input designator. Note that you may not see an error if you use test instead of mtest or ltest, or train instead of mtrain or ltrain, due to the way the tools are designed.
As with the question above, this is a result of your using the ref input instead of mref or lref, and the runspec tool is unable to find it.
The config files (reference?) allow declarations for invoking profilers and monitors etc. with the benchmarks. One method is with the submit directive:
submit = mpich $command
which can be rewritten to invoke a wrapper program:
submit = mpich run_profiler $command
The wrapper program starts the profiler or monitor, runs the $command. Note that the stdout from the $command must be captured and passed back as the output of the run_profiler wrapper.
A second form of declaration is
monitor_pre = run_profiler submit = mpich $command monitor_post = end_profiler
which invokes the profiler before each benchmark is executed, then stops it after the benchmark completes. (reference?)
The SPEC tools consider a run to be invalid of it does not satisfy all the following criteria:
The output report will state invalid run if any of these criteria fail to hold. A report that doesn't state this, though, may still contain violations of the SPEC MPI2007 Run and Reporting Rules. and isn't guaranteed to pass the SPEC/HPG committee review. The SPEC tools cannot determine that a portability flag is really required, for example.
A report that passes SPEC/HPG review must also give a complete and accurate description of the system under test. Reports are typically submitted by the company that developed the system under test, so the submitter is typically in the best position to complete the details of the report. The committee, on the other hand, has developed the conventions for how this information is to be stated, and may require edits to the report before it will be published.
Running Benchmarks
Running 121.pop2 mref (ref) base oct09a default
Error with '/spec/mpi2007/bin/specinvoke -E -d /spec/cpu2006/benchspec/MPI2007/121.pop2/run/run_base_mref_oct09a.0000
-c 1 -e compare.err -o compare.stdout -f compare.cmd'
: check file '/spec/mpi2007/benchspec/MPI2007/121.pop2/run/run_base_mref_oct09a.0000/.err'
*** Miscompare of pop2.out, see /spec/mpi2007/benchspec/MPI2007/121.pop2/run/run_base_ref_oct09a.0000/pop2.out.mis
Error: 1x121.pop2
Producing Reports
mach: default
ext: oct09a
size: mref (ref)
set: medium
|
Why did it say that? What's the problem?
We don't know. Many things can cause a benchmark to miscompare, so we really can't tell you exactly what's wrong based only on the fact that a miscompare occurred.
But don't panic.
Please notice, if you read the message carefully, that there's a suggestion of a very specific file to look in. It may be a little hard to read if you have a narrow terminal window, as in the example above, but if you look carefully you'll see that it says:
*** Miscompare of pop2.out, see /spec/mpi2007/benchspec/MPI2007/121.pop2/run/run_base_mref_oct09a.0000/pop2.out.mis
Now's the time to look inside that file. Simply doing so may provide a clue as to the nature of your problem.
On Unix systems, change your current directory to the run directory using the path mentioned in the message, for example:
cd /spec/mpi2007/benchspec/MPI2007/121.pop2/run/run_base_mref_oct09a.0000/
On Microsoft Windows systems, remember to turn the slashes backwards in your cd command.
Then, have a look at the file that was mentioned, using your favorite text editor. If the file does not exist, then check your paths, and check to see whether you have run out of disk space.
You can ask for more detail of the miscompare to be listed in the file. How to do this is described in the utility.html document, in the section on specdiff.
In a validating run, the SPEC tools compare the program results against a reference output, and the results are valid if they differ by no more than a specified margin of tolerance. The tolerances themselves are the smallest margin that permited a "pass" on all the test platforms.
Because of communication race-conditions between processes in the MPI2007 applications, they behave less deterministically than the single process applications of CPU2006 or the threaded applications of OMP2001. Consequently the margin of tolerance for the MPI2007 applications is correspondingly wider than it is for CPU2006 or OMP2001.
If the benchmark took less than 1 second to execute, it didn't execute properly. There should be one or more .err files in the run directory which will contain some clues about why the benchmark failed to run. Common causes include libraries that were used for compilation but not available during the run, executables that crash with access violations or other exceptions, and permissions problems. See also the suggestions in the next question.
'rand.234923.out' short
What does short mean?
If a line like the above is the only line in the .mis file, it means that the benchmark failed to produce any output. In this case, the corresponding error file (look for files with .err extensions in the run directory) may have a clue. In this case, it was Segmentation Fault - core dumped. For problems like this, the first things to examine are the MPI parameters, followed by the portability flags used to build the benchmark.
Have a look at the sample config files in $SPEC/config or, on Windows, %SPEC%\config. If you constructed your own config file based on one of those, maybe you picked a starting point that was not really appropriate (e.g. you picked a 32-bit config file but are using 64-bit compilation options). Have a look at other samples in that directory. Check at www.spec.org/mpi2007 to see if there have been any result submissions using the platform that you are trying to test. If so, compare your portability flags to the ones in the the config files for those results.
If the portability flags are okay, your compiler may be generating bad code.
Try reducing the optimization that the compiler is doing. Instructions for doing this will vary from compiler to compiler, so it's best to ask your compiler vendor for advice if you can't figure out how to do it for yourself.
If you're using a beta compiler, try dropping down to the last released version, or get a newer copy of the beta. If you're using a version of GCC that shipped with your OS, you may want to try getting a "vanilla" (no patches) version and building it yourself.
In this case, the benchmark is probably running, but it's not generating answers that are within the tolerances set. See the suggestions for how to deal with compilers that generate bad code in the previous two questions. In particular, you might see if there is a way to encourage your compiler to be careful about optimization of floating point expressions.
After the benchmarks are run on the system under test (SUT), a ratio for each of them is calculated using the run time on the SUT and a SPEC-determined reference time. From these ratios, the following metrics are calculated:
Larger data sets will be added later on, with the metrics
In all cases, a higher score means better performance on the given workload.
In order to provide comparisons across different computer hardware, SPEC provides the benchmarks as source code. Thus, in order to run the benchmarks, they must be compiled. There is agreement that the benchmarks should be compiled the way users compile programs. But how do users compile programs?
Some users might experiment with many different compilers and compiler flags to achieve the best performance, and use complex combinations of specific optimization flags.
Other users might prefer the relative simplicity of using a single, short, set of switches and a single-step make process.
In addition to the above, a wide range of other types of usage models could also be imagined, ranging in a continuum from -Odebug at the low end, to inserting directives and/or re-writing the source code at the high end. Which points on this continuum should SPEC MPI2007 allow?
SPEC recognizes that any point chosen from that continuum might seem arbitrary to those whose interests lie at a different point. Nevertheless, choices must be made.
For MPI2007, SPEC has chosen to allow two types of compilation:
The base metrics (e.g. SPECmpiM_base2007) are required for all reported results and have stricter guidelines for compilation. For example, the same flags must be used in the same order for all benchmarks of a given language. This is the point closer to those who might prefer a relatively simple build process.
The peak metrics (e.g. SPECmpiM_peak2007) are optional and have less strict requirements. Different compiler options may be used on each benchmark. This point is closer to those who may be willing to invest more time and effort in development of build procedures.
Note that options allowed under the base metric rules are a subset of those allowed under the peak metric rules. A legal base result is also legal under the peak rules but a legal peak result is NOT necessarily legal under the base rules.
A full description of the distinctions and required guidelines can be found in the SPEC MPI2007 Run and Reporting Rules.
The MPI2007 suites include mtest and mtrain data sets for the purpose of validating the correctness of the compile and the run environment. Note that the term mtrain is a holdover from previous suites that allowed feedback-directed optimization using measurements from training runs. For the purpose of MPI2007 it refers to a larger testing data set.
The mref data-set is the largest and is used in reportable runs. Since it runs longer it gives a clearer picture of system performance. In the future it will be supplemented with even larger lref and xref data sets.
A base measurement is required in every reportable run. The corresponding peak measurement is optional but is useful to show how high performance your system can achieve.
When the larger lref and xref data sets become available, you will have to decide which to use. The larger data sets will scale to larger machines and so give a more accurate picture of the performance on large problems. On the other hand, more results will have been reported with the mref data set size, so it will provide a more common reference point of comparison.
SPEC uses a reference machine to normalize the performance metrics used in the MPI2007 suites. Each benchmark is run and measured on this machine to establish a reference time for that benchmark. These times are then used in the SPEC calculations.
As a reference machine, SPEC/MPI2007 uses an 8-node cluster of Celestica A2210 (AMD Serenade) systems connected by a TCP (GigE) Interconnect. Each A2210 system contains two 940 sockets, each holding one single-core AMD Opteron 848 processor chip running at 2200 MHz with 1 MB I+D L2 cache, plus 4 GB of DDR 333 memory per socket. The MPI implementation is MPICH2 version 1.0.3 running on SLES 9 SP3 Linux OS with Pathscale 2.5 compilers. Measurements of the reference platform are posted on the SPEC Website.
Note that this machine differs dramatically from the ones used for SPEC/OMP and SPEC/CPU, since the MPI2007 suites represent a fundamentally different class of applications.
Note also that when comparing any two systems measured with the MPI2007, their performance relative to each other would remain the same even if a different reference machine was used. This is a consequence of the mathematics involved in calculating the individual and overall (geometric mean) metrics.
The reference cluster takes about 1 hour to build the base versions of the MPI2007 benchmark executables and 24 hours to finish a rule-conforming (2 iteration) run of the base metrics for the medium sized workload.
Right here:
104.milc 1565 107.leslie3d 5220 113.GemsFDTD 6308 115.fds4 1951 121.pop2 4128 122.tachyon 2797 126.lammps 2915 127.wrf2 7796 128.GAPgeofem 2065 129.tera_tf 2768 130.socorro 3817 132.zeusmp2 3103 137.lu 3676
You can also extract these using the command grep '[0-9]' benchspec/MPI2007/*/data/mref/reftime from the top of the MPI2007 directory-tree. The MPI2007 reports include the ranks run, the run time, and the SPEC score for each observation. All this is useful, but the reference times, which were listed in the CPU2000, OMP2001, and HPC2002 reports, are omitted to limit the size of the table.
To derive the MPI2007 score, the medians of the measured times are divided by the reference times, and the Geometric Mean of these quotients is the actual score. If you look at the scores for the reference platform you will see that they vary from 0.96 to 1.03, while you would expect them to all be 1.00 since the reference times are being divided by themselves. There are two reasons that this is not the case:
SPEC member companies follow due diligence in preparing and reviewing the reports for publish, and these kinds of problems have not been an issue thus far. Note that SPEC/HPG allows a 5% tolerance in measured performance, to account for run-to-run variability. Measurements within this range are considered to be consistent. Measurements which are consistently below this range would indicate that something is different in the hardware or software configuration.
First, make sure that your system is configured the same as the one in the published report. Then use the same config-file as what was used in the report. If the results don't match, you will need to notify company XYZ that the performance doesn't match, either through their Marketing department or through their SPEC/HPG representative. They may be able to correct the problem. If not, contact the SPEC/HPG committee at mpi2007support@spec.org.
In many cases it is not possible to exactly reproduce the configuration that was used to produce the published report: the size or type of the memory cards may be different, or the OS or compilers may be at different levels than were used in the publish. These kinds of differences could account for the gap in performance. SPEC/HPG would require the vendor to add further details to the published report so it will be valid for the configuration that was used.
SPEC/HPG recognizes that changes from pre-production to GA hardware and software, or changes between levels of hardware or software releases, can cause degradations in performance as measured by the benchmarks. A general tolerance of 5% is allowed, to account for run-to-run variations, and a degradation within this 5% is acceptible even if it is due to specific causes.
Measurement discrepancies have never been an issue so far. SPEC member companies have been careful with their measurement and review procedures, and in at least one case have withdrawn and replaced results on their own initiative, that had been internally determined to be irreproducible.
Assuming you don't correct the results, you take risks that disappointed customers will return your systems or that competitors' marketing literature will exploit these inconsistencies and damage your credibility. SPEC itself could publish comments on your claims or mark your published results as invalid.
Benchmark timings cannot be exactly reproduced even between consecutive runs of the same binary with the same system settings. The reproducibility tolerance is expected to account for allowable run-to-run variability, and any degradation beyond this is likely to indicate deficiencies in the hardware or software.
Since MPI-parallel applications are composed of communicating asynchronous processes, they show higher run-to-run variability than do serial benchmarks. SPEC/HPG studied the benchmark runtimes on a variety of systems used in the MPI2007 internal acceptance tests and chose 5% as a tolerance that such systems can be expected to meet. As MPI2007 comes into wider use and is measured across a larger set of machines, the SPEC/HPG committee can increase this range of tolerance if it is decided to be necessary.
If you are overly concerned with the stability of your measured results, you can improve it by running more iterations of the benchmark suite. Statistically, both the median measurements of the individual benchmarks and the geometric mean composite of the whole suite will show much lower variability than the individual benchmark measurements themselves. At least two iterations of benchmark runs are required in a reportable run, and increasing this iteration count will reduce the variability of the composite even further.
Results for measurements submitted to and approved by SPEC are posted at http://www.spec.org/mpi2007.
The runspec utility leaves a file with a path result/MPIM2007.*.rsf. If the run was reportable, you can mail this file (or attach it to a message) to submpi2007@spec.org. If your company is not a member of SPEC/HPG, you will need to make arrangements with the committee to have the results reviewed.
The SPEC members' website contains lists of reports that have been submitted or are under review. When your company has joined SPEC, you will have access to these web-pages. To edit a submitted report, you will download the .sub file for the submission, edit it, and mail it to resubmpi2007@spec.org. Note that the file will not be accepted if you edit anything below the line
# =============== do not edit below this point ===================
If your company is not a member of SPEC/HPG, you will need to make arrangements with the editor at editor@spec.org to receive a copy of the editable .sub file.
If you report a measurement using pre-GA software, some component may change by the time it GA's. If it is the case that only the version number changes, you can edit the published reports to state the GA version number.
If it is the case that a compilation flag or library name changes, such that the form you had used is invalid, you can append a note to the published reports to state the correct form to use. Note that if a third party downloads the config-file from your submission, and runs it on a GA system, it will fail due to the invalid flag or library name. The information you provide should be enough for them to recognize the problem and apply the change.
If it is that case that the semantics of compilation flag or function of a library changes in a way that the benchmarks still work but do not deliver the claimed level of performance, and some other choice of settings will achieve the stated level of performance, you need to repeat your measurement under these GA conditions and resubmit it to SPEC.
format: Submission Check -> FAILED. Found the following errors: - The "hw_memory" field is invalid. It must contain leading digits, followed by a space, and a standard unit abbreviation. Acceptable abbreviations are KB, MB, GB, and TB. The current value is "20480 Megabytes".
A complete, reportable result has various information filled in for readers. These fields are listed in the table of contents for config.html. If you wish to submit a result to SPEC for publication at www.spec.org/mpi2007, these fields not only have to be filled in; they also have to follow certain formats. Although you are not required to submit your result to SPEC, for convenience the tools try to tell you which reporting details will need to be corrected. In the above example, the tools would stop complaining if the field hw_memory said something like "20 GB" instead of "20480 Megabytes".
Notice that you can repair minor formatting problems such as these without doing a re-run of your tests. You are allowed to edit the rawfile, as described in utility.html.
SPEC MPI2007 provides benchmarks in source code form, which are compiled under control of SPEC's toolset. Compilation flags (such as -O5 or -unroll) are detected and reported by the tools with the help of flag description files. Therefore, to do a complete run, you need to (1) point to an existing flags file (easy) or (2) modify an existing flags file (slightly harder) or (3) write one from scratch (definitely harder).
Notice that you do not need to re-run your tests if the only problem was Unknown flags. You can just use runspec --rawformat --flagsurl
Yes, SPEC MPI2007 results can be freely published if all the run and reporting rules have been followed. The MPI2007 license agreement binds every purchaser of the suite to the run and reporting rules if results are quoted in public. A full disclosure of the details of a performance measurement must be provided on request.
SPEC strongly encourages that results be submitted for publication on SPEC's web site, since it ensures a peer review process and uniform presentation of all results.
The run and reporting rules for research and and academic contexts recognize that it may not be practical to comply with the full set of rules in some contexts. It is always required, however, that non-compliant results must be clearly distinguished from rule-compliant results.
The MPI2007 suite defines and calculates the speed metrics SPECmpiM_base2007 and SPECmpiM_peak2007. These are the only MPI2007 metrics defined by SPEC. Any other metrics you derive from these are *not* endorsed by SPEC, and must not be referred to as SPEC metrics. For example, using the term "SPEC megaflops" is misleading, since SPEC does not define any such metric. This is a violation of the SPEC trademark.
Further, while the SPECmpiM_base2007 and SPECmpiM_peak2007 measurements may have been reviewed and approved by SPEC, any calculations based on these meas