|
SPEC/GWPG
Benchmarks
Download/Order
SPEC
Mirror Sites
Resources
|
The Workstation Performance Characterization Project Committee Rules
Version 3.0
Last Updated: 10/30/2018
- Overview
- General Philosophy
- All rules declared in "The
Graphics and Workstation Performance Group (SPEC/GWPG): Rules For
Project Groups" document (known hereafter as the GWPG
Project Groups Ruleset) apply, unless specifically overruled by
a rule in this document.
- General Philosophy
- Within the SPEC/GWPG (Graphics and Workstation Performance Group) committee,
there is a strong belief that it is important to benchmark system performance
running algorithms used in popular workstation applications, but without requiring
the full application and associated licensing to be installed on the system under
test. Thus, the SPECwpcSM (Workstation Performance
Characterization) project group was created within the SPEC/GWPG committee to
create a broad-ranging set of standardized benchmarks for workstation applications.
- The SPECwpc Project seeks to develop benchmarks for generating accurate
workstation performance measures in an open, accessible and well-publicized manner.
- The SPECwpc Project wishes to contribute to the coherence of the field
of application performance measurement and evaluation so that vendors will be better
able to present well-defined performance measures and customers will be better able
to compare and evaluate vendors' products and environments.
- The SPECwpc Project will provide formal beta benchmarks to members and final benchmark
releases to the public in a timely fashion.
- Hardware and software used to run the SPECwpc Project benchmarks must provide
a suitable environment for running typical (not just benchmark) workloads for the
applications in question.
- The SPECwpc Project reserves the right to adapt its benchmarks as it deems
necessary to preserve its goal of fair and useful benchmarking (e.g. remove benchmark,
modify benchmark code or data, etc). If a change is made to the suite, the SPECwpc Project
committee will notify the appropriate parties (i.e. SPECwpc Project members and paying users
of the benchmark) and the SPECwpc Project committee will re-designate the benchmark by
changing its name and/or version. In the case that a benchmark is removed in whole or in part,
the SPECwpc Project committee reserves
the right to republish in summary form "adapted" results for previously published
systems, converted to the new metric. In the case of other changes, such a republication
may necessitate re-testing and may require support from the original test sponsor.
- Overview of Optimizations
- The SPECwpc Project committee is aware of the importance of optimizations in producing the best
system performance. The SPECwpc Project committee is also aware that it is sometimes hard to draw an
exact line between legitimate optimizations that happen to benefit the SPECwpc project
group benchmarks and optimizations that specifically target those benchmarks. However,
with the list below, the SPECwpc Project committee wants to increase awareness of implementers
and end-users to issues of unwanted benchmark-specific optimizations that would be incompatible
with the SPECwpc Project committee's goal of fair benchmarking.
- To ensure that results are relevant to end-users, the SPECwpc Project committee expects that
the hardware and software implementations used for running its benchmarks adhere to a set of
general rules for optimizations.
- General Rules for Optimization
- Optimizations must improve performance for a class of workloads where the class of
workloads must be larger than a single SPECwpc project group benchmark or benchmark suite.
- For any given optimization a system should generate correct results with and without
said optimization. An optimization should not reduce system stability.
- In the case where it appears that the above guidelines have not been followed, the
SPECwpc Project committee may investigate such a claim and request that the optimization in
question (e.g. one using a benchmark-specific pattern matching) be removed and the results
resubmitted. Or, the SPECwpc Project committee may request that the vendor correct the deficiency (e.g.
make the optimization more general purpose or correct problems with image generation) before
submitting results based on the optimization.
- It is expected that system vendors would endorse the general use of these optimizations
by customers who seek to achieve good application performance.
- No pre-computed intermediate or final results may be substituted within a SPECwpc project
group benchmark on the basis of detecting that said benchmark is running (e.g. pattern matching
of command stream or recognition of benchmark's name).
- Benchmarks
- Benchmark Acceptance
- Benchmark components are defined as
- specific revision of an application,
- run rules, scripts and associated data sets.
- New or modified benchmark components require a 2/3-majority vote
to be accepted for publication. Selection of datecode versions of
a specific revision of an application is by majority vote.
- A minimum 3-week review period is required for new or significantly
modified benchmark components.
- At the end of the review period a vote will be called to approve
the proposed changes.
- An amendment to a benchmark component during the review period
must be unanimously accepted. If not, the review period shall be
restarted.
- Benchmark Code Versioning
- Benchmarks use the following version coding: M.m (e.g.
the SPECworkstationSM 3.0).
M is the major release number and m is the minor release number.
- The minor release number is incremented when performance-neutral
modifications are made to the benchmark.
- The major release number is incremented when changes to the benchmark
cause results to no longer be comparable with previous results.
- When there is a new major release of a benchmark, submissions using
the previous release will be accepted for at least one submission cycle.
When a superseded benchmark is retired, the associated run-rules will
be archived in an "archived benchmark run-rules document" posted
on the Previous SPECwpc Project Group Benchmarks web-page.
- Benchmark Rules
- The system under test must correctly perform all of the operations being requested by the application during the benchmark.
- No changes to any files associated with the benchmark are permitted excepted
as noted in the benchmark run rules (section 4 of this document).
- The entire display raster must be available for use by the application being benchmarked, and the display must be powered on.
- It is not permissible to override the intended behavior of the tests through any means including, but not limited to, registry settings or environment variables.
- No interaction is allowed with the system under test during the benchmark, unless required by the benchmark. It is not permissible to interact with the system under test to influence the progress of the benchmark during execution.
- The system under test cannot skip steps during the benchmark run.
- It is not permissible to change the system configuration during the running of a given benchmark. That is, one can't power off the system, make some changes, then power back on and run the rest of the benchmark.
- Results submitted must be obtained using the scripts, models, and application revisions which are specified for that submission cycle by the SPECwpc Project committee.
- Optimizations for viewsets used in this benchmark must adhere to the
same standards as outlined in the SPECgpc® (Graphics Performance Characterization) Project Group Rules, Version 2.13,
Section 3.
- Optimizations for the workloads used in this benchmark cannot modify
the executable or the dataset used in the test.
- The number of threads spawned for multi-threaded workloads may be modified
for a given benchmark run by changing the values in the advanced tab of the
GUI. The test harness will calculate the number of threads that are evenly divisible
by the number of logical cores in the system and adjust the actual number of
threads spawned by each process.
- Caching of files in a high-performance storage device is allowed as
long as the mechanism for populating the cache is generally applicable
to a wide range of applications run on the device under test.
- SPECworkstation is only supported on systems running Microsoft Windows 10
64bit.
- The color depth used must be at least 24 bits (true color), with at least
8 bits of red, 8 bits of green and 8 bits of blue.
- If a depth buffer is requested, it must have at least 24 bits of resolution.
- The display raster resolution must be at least 1920 pixels by 1080 pixels and must fully display all of the benchmark tests being submitted. Higher desktop resolutions may be used, and while the operating system may create the appearance of a higher raster resolution, the actual raster resolution remains at 1900 x 1060 and will be noted in the results file and log files. An optional feature to adjust the raster resolution to the native resolution of the display (less a ten-pixel border) may be selected, but not all viewsets will support this method of scaling. Therefore, native resolution runs shall not be accepted for publication or otherwise be considered official until such a point that all viewsets can be scaled to higher native display resolutions.
- Screen resolution must be large enough to run the individual tests at their requested window size, with no reduction or clipping of test window.
- Screen DPI scaling must be set to 100% on Microsoft Windows, and no UI or other window elements, including the task bar, may occlude the viewport rendering context.
- The benchmark must be run with Windows User Account Control (UAC)
and firewall disabled, or with administrator privileges and the appropriate
firewall settings to allow the benchmark to complete without interruption.
- The benchmark can be installed and run on any storage device connected to the system. 190GB of free space is required after the benchmark is installed to run the storage portion of the benchmark.
- The system under test must have at least 16GB of system RAM.
- The scoring utility must be used to generate valid results.
- Virtualized configurations, defined as any operating system configuration
running on a Hyper Visor or virtualization layer of any kind, must include the word
"virtualized" in the comment field of the config.txt file. This information
must be populated before the execution of the benchmark to ensure the results
reflect this attribute.
- Virtualized configurations as defined in rule 18 must also include a declaration
of the transport layer name and version used by the virtualization software in parenthesis
after the graphics accelerator listed in the Graphics Accelerator field of the Graphics
Hardware Configuration section of the result file.
- Virtualized configurations as defined above must manually modify the Model field on
the submission form to include a three line link on the summary page that includes:
- Physical server supplier and model number
- Hypervisor supplier and version
- VDI supplier and version
- Submission, Review and Publication
- General Submission Content Rules
- These rules are specific to SPECworkstation 3 and shall apply in addition to the Submission Content Rules in the GWPG Project Groups Subset
- A SPECworkstation 3 submission can be for one or more subsegments per configuration. Subsections are defined as Media and Entertainment, Product Development, Life Sciences, Financial Services, Energy, and General Operations.
- The SPECworkstation 3 submission must contain the SPECworkstation/Results_xxxx directory structure for the run. The results folder is created by the benchmark with the following naming convention: "results_"YYYYMMDD"T"HHMM"_r1"
- Submission Process Rules
- These rules are specific to SPECworkstation 3 and shall apply in addition to the Submission Process Rules in the GWPG Project Groups Ruleset.
- The submission file must be named company_wpc_vN.zip where company is the member company or organization name in lower case and vN is the file version (e.g. hp_wpc_v0.zip.) The initial submission is v0. Resubmitted files must have the version number incremented.
- Review Period Rules
- These rules are specific to SPECworkstation 3 and shall apply in addition to the Review Period Rules in the GWPG Project Groups Ruleset.
- Reviewers will decide if the image quality and results of the submission are sufficiently correct with respect to the intent of the ISV to satisfy the intended end-users' expectations.
- Reviews will ensure that all tests ran without reporting an error. Errors will be propagated to the summary page and displays as such.
|
