SPEC/GWPG Project Group Rules

The Graphics and Workstation Performance Group (SPEC/GWPG):
Rules For Project Groups

Version 1.10

Last Updated: 09/13/2007

Overview
1. Project Groups and Scope of Rules
  1. Two project groups exist under the umbrella of SPEC/GWPG:
    - The Graphics Performance Characterization Project Group (henceforth abbreviated as SPECgpc^SM)
    - The Application Performance Characterization Project Group (henceforth abbreviated as SPECapc^SM)
  2. The rules contained in this document shall apply to both project groups (SPECgpc and SPECapc).
  3. Each project group shall maintain its own project group rules document, which shall apply in conjunction with this document. - SPECgpc Project Group Rules - SPECapc Project Group Rules
  4. Where a project group's rule overrides a rule in this document, this will be explicitly indicated in that project group's rule document.
  5. Should a new project group be approved or an existing one dissolved, this document shall be updated accordingly.

Membership
1. Membership
  1. Membership in the SPEC/GWPG and its project groups is open to any organization that has a direct and/or material interest in graphics or workstation-related application performance benchmarking.
  2. Membership in one or more SPEC/GWPG project groups leads to membership of SPEC/GWPG.
  3. Members are expected but not required to be active participants developing and improving the respective project group's benchmarks.
  4. Members are entitled to secure access to development code.
  5. Members are entitled to unlimited publication rights.
  6. New members become eligible for voting on the 2nd consecutive qualified meeting. The first qualified meeting may have been attended prior to becoming a member. Qualified meetings are defined in Section II.4.b.
  7. A member maintains voting rights by attending 1 out of the last 3 qualified meetings. A member loses their voting rights upon missing 2 consecutive qualified meetings.
  8. A member regains voting rights on attending a second consecutive qualified meeting.
  9. For a qualified meeting for which attendance in person is expected, attending remotely (e.g. by telephone) does not count as qualified attendance.
  10. Voting status is lost if the organization fails to remit payment for membership fees or annual dues. Voting status is restored by payment of these fees or dues.
2. Associate Status
  1. Associate status is available to non-profit organizations.
  2. All rights and rules of the respective project group, GWPG and SPEC apply to Associates unless specifically stated otherwise.
  3. Associates are entitled to secure access to development code.
  4. Associates do not have voting rights.
3. Officers and Elections
  1. On an annual basis the project groups will elect from their eligible voting memberships the following officers:
    1. Chairperson
    2. Vice Chairperson
    3. Secretary
  2. The Chairperson's responsibilities are to
    1. conduct meetings,
    2. send out the agenda on time,
    3. conduct votes on time,
    4. deal with outside organizations such as the press,
    5. represent and respond on behalf of the group to external questions and queries,
    6. interact with the SPEC/GWPG committee, and
    7. police the submission, review and appeal process.
  3. The Vice-Chairperson's responsibility is to do the chairperson's job when the chairperson is not available, or if the chairperson is subject to a conflict of interest.
  4. The Secretary's responsibilities are to:
    1. record minutes,
    2. maintain the rules document,
    3. keep a history of email.
  5. If an officer is subject to a conflict of interest in pursuance of his or her duties and if any defined succession of responsibility would fail to resolve the conflict of interest, the committee may appoint any committee member to fulfill that officer's duties for the scope of the discussion in which the conflict of interest exists.
4. Meetings
  1. SPEC/GWPG project groups have three types of meetings (not including ad-hoc working-group meetings)
    1. Regular quarterly face-to-face meetings
    2. Special face-to-face meetings for the full membership
    3. Conference-call meetings
  2. Meetings which qualify for attendance are limited to:
    1. face-to-face meetings scheduled at least one month in advance and
    2. conference calls scheduled at least two weeks in advance and which are explicitly indicated as qualified at least two weeks in advance.
5. Voting
  1. Issues may be designated for resolution by ballot by voting members of the GWPG or Subcommittee. Ballot may be by standard mail, by electronic means, conference call voice ballot or a combination of any of the three. A ballot is deemed valid if a quorum of eligible voting organizations returns ballots. Voting is Approved, Approved with Comment, Disapproved with Comment, or Abstained with Comment. Disapproval and Abstained votes require comment on the nature of the vote.
  2. A valid vote requires a quorum. A quorum is met if at least 66% of eligible voting members respond.
6. Membership Dues and Billing
  1. Dues are assessed on the basis of membership of SPEC/GWPG's project groups.
  2. Dues for the SPEC/GWPG project groups will be set annually by the SPEC Board of Directors with input from the SPEC/GWPG. Once set, the dues amount will be recorded in the SPEC minutes and communicated to the SPEC/GWPG by the SPEC office.
  3. Dues payment, purchase order or letter of intent to pay for a given calendar year must be received at the SPEC office by March 1st of that year. Alternatively, a letter of intent to join the respective project group must be received by the SPEC office by March 1st of that year with a subsequent dues payment by May 1st of that year. Failure to meet these deadlines will result in loss of membership and voting rights. Membership will be reinstated when full payment is received at the SPEC office. Voting rights will be reinstated according to the attendance rules in section II.1.g and II.1.h.
7. Non-Member Publication
  1. The SPEC/GWPG project groups will accept submissions from non-members for review and publication on the SPEC public website.
  2. Non-member submissions must follow the same rules and procedures as member submissions.
  3. Non-members are not eligible to participate in reviewing results.
  4. Non-members will be charged for their submissions according to an approved fee structure. Any change in hardware or software constitutes a new configuration.
  5. On an annual basis the SPEC/GWPG will establish the pricing and periods for non-member publication. These will be recorded in the SPEC/GWPG minutes and published on the GWPG web-site.
  6. Following acceptance by the assigned reviewers, a non-member's submission will not be published until the SPEC office has received the submission fee in full.
  7. The SPEC office will not deposit funds provided by the non-member submitter until the submission has been accepted by the assigned reviewers.
  8. A configuration will be published on-line for six months, unless the submitter notifies the publisher that it should be removed.
  9. After six months, the configuration will be removed automatically, unless the submitter notifies the publisher that it should remain on-line.
  10. There are no additional non-member fees for extending on-line publication beyond six months.
  11. Each SPEC/GWPG project group may remove published results from its web pages due to benchmark revision. In this case, the submitter will be given notice by the project group and may, at no charge, resubmit the identical configuration for the revised benchmark.
Benchmarks
1. Each project group shall document all benchmark-related rules in its respective project group rules document.
Submission and Review Rules
1. Submission Preparation Rules
  1. The rules for the submission and review cycle to be used are those approved by the respective project group's committee prior to the submission deadline. The approved rules must be posted to the respective project group's web-site by the first publication date for the benchmark.
  2. Version compliance: The benchmark and (where applicable) application versions to be used are those approved by the respective project group's committee prior to the submission deadline. The approved benchmark (and application) versions must be posted to the respective project group's web-site by the first publication date for the benchmark.
  3. All benchmark sources for a submission must be the same as that approved by the respective project group's committee prior to the submission deadline. The approved benchmark sources must be posted to the respective project group's web-site by the first publication date for the benchmark.
2. Submission Content Rules
  1. The information supplied must reflect the system as tested.
  2. Configuration description: All fields in a submission's results file must be supplied, unless the field names are marked "opt.", indicating an optional field.
  3. Submitters must specify a date for 'General Availability' that is accurate for the entire system - hardware, software, O/S, drivers, etc.
  4. The "Comments" area of the results page must describe how the system may be acquired.
  5. Date fields must always contain a valid date. "Now" is not valid in a date field.
  6. Price includes system and monitor as tested.
  7. Price may be submitted with an alterate currency from the US dollar. The submission will be sorted separately on the summary pages for Price and Price/Performance.
  8. The submitter is required to declare sufficient information to reproduce the performance claimed. This includes but is not limited to:
    1. non-default environment variables,
    2. non-default registry variables,
    3. hints,
    4. compiler name and version,
    5. compiler command line,
    6. changes to the standard makefiles.
  9. Any information required to be reported such as non-default environment variables, registry variables or hints, that does not have a predefined field must be documented in the "Comments" area of the results page.
  10. Valid submissions must include screen captures if required by the benchmark.
  11. Results previously published for a system can be resubmitted. Resubmissions do not require the inclusion of screen capture images.
  12. Previously published results being re-submitted can only have price changes.
  13. Each member company must ensure that the upload file contains data for all the new configurations and existing published configurations they wish to continue publishing.
  14. Standardized CPU nomenclature is as follows:
    1. CPU / Processor: a physical package containing one or more cores.
    2. Socket – Receptacle or physical connection between processors and the system.
    3. Core: set of execution units which completely implement the instruction set of a processor architecture and are capable of running one or more threads.
    4. Thread: Processor-directed sequence of instructions
    5. All processors in the system, the number of their cores and the number of threads (if more than one) a core can execute must be disclosed in the system description whether or not they are directly enabled by system software or application software. If different from the number physically present, the number of processors, cores and threads enabled must also be disclosed.
  15. Standardized CPU cache nomenclature is as follows:
    1. (D+I) designates a unified instruction and data cache
    2. (D/I) designates separate instruction and data caches
    3. A number followed by KB or MB can be used to describe the size of the cache.
    4. Caches dedicated to a processor are listed as per processor cache size.
    5. Caches shared by multiple processors are listed by total size
  16. Each component of the submitted configuration (including the graphics driver) shall be:
    1. uniquely identified,
    2. available to members of the respective project group, upon demand, by the submission deadline and for the duration of the review process,
    3. verifiably available to the public by the publication date, with continued availability at least for the life of the submission, with sufficient information in the comment field to enable users to directly obtain this component.
  17. Price must be valid for two submission periods from date of publication. Quantity 1 pricing must be used.
  18. Subsequent to publication, any change to or replacement of elements for a submitted configuration must not result in more than a 5% performance degradation in the submitted benchmark results. Upon demonstration of such a degradation, the submitted results for this configuration will be removed from the SPEC public website.
  19. On or before the date of publication, the submitted configuration shall be available for purchase by the public, for the specified price or less, with a firm delivery date of 60 days or less. Submissions will be categorized as either “Single Supplier” or “Parts Built”, where “Single Supplier” is defined as a configuration where all hardware, OS, and drivers are sold and supported by the submitting supplier. “Supported” is defined as providing hardware, drivers and associated technical support, and that the drivers are available from the system supplier. “Parts Built” is defined as a configuration built and supported by multiple suppliers.
  20. “Parts built” system pricing must include enough detail to reproduce all aspects of the submission, including performance and price, and include all hardware and O/S costs necessary to run benchmark.
3. Submission Process Rules
  1. Each benchmark is considered a separate submission.
  2. Submissions of each benchmark's results (e.g. Maya6.5™, Solidworks 2007™, SPECviewperf, etc.) must be in separate tar/zip files.
  3. A submitter of benchmark results must upload his or her submission to the proper server location by the submission deadline date and time. The submitter must not create any new directories on the server when uploading the submission.
  4. The submitter must notify SPEC Office after a submission is uploaded to the server prior to the submission deadline with contact information for questions about the submission.
  5. The submitter must contact the SPEC office if they have attempted to upload their submission and were not successful.
  6. The SPEC office will not disclose who has submitted results until the submission deadline has passed.
  7. Submissions will not be accepted after the submission deadline.
  8. The upload directory will be set to write-only until the submission deadline has passed. Then it is set to read-write (not modify) after the submission deadline.
  9. If a submitter is notified that their submission format is incorrect, they must re-send their submission in proper format within 3 business days of notification.
4. Review Period Rules
  1. SPEC/GWPG project group members shall keep all submitted results confidential to the respective project group until those results appear on the public SPEC web site. The exception to this rule is that members are free to make their own submitted results public at any time.
  2. SPEC Office assigns reviewers to submitters.
  3. Members who wish not to review the submission of other specific members due to conflict of interest must submit that list to the SPEC office prior to the submission deadline. The SPEC office will hold the list in confidence from other members.
  4. The various SPECapc and SPECgpc pools of eligible reviewers will be independent of each other. The SPEC office will send the list of contact information for the submissions under review.
  5. All members will have access to all benchmark submissions once the review period begins.
  6. The review period shall be 5 calendar days.
  7. Submissions cannot be withdrawn during the review period without cause and without prior approval of the primary reviewer. A submitter who is granted permission to withdraw a submission must inform the committee by email of the reason for withdrawal.
  8. If a primary reviewer has a question with a submission they must pose the question to the submitter first. The primary reviewer may also pose questions to the respective project group's officers or SPEC/GWPG Chair for clarification of rules if needed.
  9. Any reviewer who has one or more questions relating to a submission must:
    1. Pose the question(s) to the submitter and cc the primary reviewer, OR
    2. Pose the question(s) to the primary reviewer. The primary reviewer must then pose the question(s) to the submitter, OR
    3. Pose the question(s) to an officer of the respective project group. The officer must then pose the question(s) to the submitter and cc the primary reviewer
  10. With permission of the primary reviewer, as communicated through the respective project group's email alias, the submitter can request that their submission be rejected on stated technical grounds.
  11. With permission of the primary reviewer, as communicated through the respective project group's email alias, a submitter may resubmit a submission to resolve issues found during the review process. The submitter must notify the respective project group's mailing list with the date and version of the resubmitted file(s).
  12. The submitter must provide the primary reviewer access to the system under test at the submitter's facilities if requested by the reviewer during the review period. The reviewer must state prior to the visit what part of the submission is going to be verified. Travel expenses are the responsibility of the reviewer.
  13. Previously published results being re-submitted can only be reviewed for consistency with the previous submission, and price changes.
  14. If the reviewer disputes the stated pricing, the submitter must provide documentation that the system can be purchased for the price quoted.
  15. By the end of the review period, the primary reviewer of a submission must designate the status of the submission one of: “accepted without comment”, "accepted with comment", “pending with comment”, or “rejected with comment”. The submitter may appeal a rejection as described in "Review Appeal Rules" below.
  16. Any comments for rejection of a submission received after the end of the review period will not delay publication of the submission.
  17. A submission designated “pending with comment” will not go public and will remain pending until the submitter addresses all comments. Once the comments are addressed the web master will post to the public site. Any member who feels comments are not satisfactorily addressed may challenge the submission according to Section IV.6 for challenging approved results.
  18. If a submitter repeatedly makes submissions that are non-compliant or which do not address concerns identified in the previously-assigned reviewers' rejection comments, the reviewer may engage the committee to solicit appropriate action, which may be up to and including an embargo on submissions from that submitter for a period of time.
5. Review Appeal Rules
  1. The appeal period shall have the same duration as one submission cycle, and shall immediately follow the review period.
  2. Any submitter of a rejected submission can make their case to the respective project group's email alias during the appeal period.
  3. At the end of the appeal period, if there is no resolution, the project group Chair shall call a vote to accept or reject the submission.
  4. The project group electorate votes on accepting or rejecting an appealed submission. A simple majority is required to accept or reject the appeal. In case of a tie the submission is rejected.
6. Challenging Accepted Results
  1. Any member may challenge accepted results at any time. This includes:
    1. archived results,
    2. currently published results, and
    3. resubmitted results not subject to the regular submission review process.
  2. The burden of proof that the result should be modified is on the member who is challenging the result.
  3. The challenge must be ratified by a majority vote of the project group's electorate.
  4. The project group Chair will call a special review cycle for a resubmission in the event that there is a ratified challenge to currently published results.
  5. A ratified challenge to archived results can only result in annotation, not removal or modification. The annotation will be determined by the majority of the electorate. It is the responsibility of the challenger to verify that the results have been annotated correctly on the public website within two working days from the ratification of the challenge.
Publication Rules
1. Official Publication
  1. Benchmark results for publication by the SPECgpc or the SPECapc must adhere to Articles concerning "Overview", "Benchmark Run Rules" and "Submission and Review Rules" as presented in this document AND the respective project group's rules document.
2. Unofficial Publication
  1. Benchmark results for publication elsewhere (e.g. industry journals, vendor web sites, analyst reports) must adhere to Articles concerning "Overview" and "Benchmark Run Rules" as presented in this document AND the respective project group's rules document.
  2. The respective project group or any member thereof reserve the right to request and receive evidence that the published results have been achieved in accordance with the rules and that published information is accurate.
  3. SPECgpc or SPECapc metrics may be estimated. Metrics shall not be estimated for configurations that are capable of running the benchmark. All estimated metrics must be clearly identified as estimated. Licensees are encouraged to publish actual SPECgpc or SPECapc metrics as soon as possible.

Adoption

V1.10 adopted on 09/13/2007 (reflects transition from GPC to GWPG)
V1.04 adopted on 10/20/2006
V1.03 adopted on 08/04/2006
V1.02 adopted on 04/27/2006
V1.01 updated on 02/09/2006 to align wording with SPEC policy
V1.00 adopted on 01/25/2006

Standard Performance Evaluation Corporation