SPEC Open Systems Group
Policies and Procedures Document
Version 2.09
Recent Revisions
| Version V2.09
25 June 2008 |
|
For previous revision history, see the bottom of the document.
- 1. General Policies and Principles
- 2. Open Systems Group
- Appendix A. OSG Benchmark Suite Development Overview
- Appendix B. Guidelines for Handling SPEC Information
- Appendix C. Guidelines for General Availability
- Appendix D. Template SPECbenchmarkYYYY Run and Reporting Rules
- Appendix Z. Miscellaneous Policies and Procedures
1. General Policies and Principles
The intent of this section is to provide an overview of the general policies and principles of the SPEC organization, as context for the policies of the Open Systems Group (OSG). The controlling documents for governance of SPEC are the Articles of Incorporation and Bylaws, filed with the State of California, and available from the SPEC Administrator.
In the event of disagreement between this document and SPEC's Bylaws, or disagreement between this document and decisions by the SPEC Board of Directors, said Bylaws or decisions shall take precedence over this document.
Note: much of the material in this section of this document resides here due to historical reasons. It is possible that SPEC's Board of Directors may decide to create a SPEC-wide document that will supercede this section at a later time, in which case this section will be removed and replaced with a pointer to the new material.
1.1 SPEC's Purpose
The primary purpose of the Standard Performance Evaluation Corporation (SPEC) is to develop suites of benchmark programs that are effective and fair in comparing the performance of high performance computing systems, and to assure that they are readily available to manufacturers and users of such systems.
1.2 SPEC's Formation
SPEC was formed from the instigation and sponsorship of Electronic Engineering Times (E.E. Times), and by the cooperative development work of Hewlett-Packard Co., Sun Microsystems Inc., Apollo Computer Inc., and MIPS Computer Systems Inc.
1.3 The Corporation
Upon approval by the representatives of the founding companies, their effort to develop a benchmark standardizing activity was converted into a non-profit Corporation of the state of California, on November 14, 1988.
The Corporation has a Board of Directors, President, and a staff to carry out the business of SPEC. The board has established three technical groups:
- The Open Systems Group (OSG)
- The Graphics and Workstation Performance Group (GWPG)
- The High Performance Group (HPG)
Each group has a steering committee to manage and supervise the development of applicable benchmarks. Each steering committee may sponsor a number of subcommittees or working groups to facilitate benchmark development. The Board may establish additional technical groups.
1.4 Meetings
The annual meeting of SPEC members will be held in January.
Election of the Board of Directors and members of the steering committee(s) and a financial report by the Board will be a necessary part of that meeting. The Board of Directors will meet at least quarterly.
1.5 The Board
To facilitate continuity and consistency on the Board, SPEC's policy is that the Board of Directors has staggered two-year terms, with half of the Board elected each year. Decisions of the board require agreement of the majority. Elections to the Board will be held at the annual meeting to be held at the beginning of the calendar year. Board positions are held by individuals, not by companies. Currently there are eight (8) elected directors on the board.
Each member of each SPEC technical group (OSG, GWPG, and HPG) may cast one vote per open board seat. E.g., if a company is a member of two groups it may cast two votes per open board seat.
1.5.1 Officers
The board elects a president, who generally serves for a term of one year, with eligibility for re-election, from among candidates proposed by the board members.
The board also elects a secretary and a treasurer of the Corporation, who are not required to be members of the board. The secretary and treasurer serve terms of one year and are eligible for re-election.
1.5.2 Administrative Duties
SPEC has an administrative organization that carries out the publicity, accounting, organizational and other necessary work of the Corporation. The administrative staff reports to the Board, through the president. The budget of the staff is developed by the president and the treasurer and approved by the board.
1.6 Participation levels
Terminology note about SPEC "Membership": SPEC is a California nonprofit mutual benefit corporation. Governing law has certain formal terminology regarding membership (see, for example, California Corporations Code sections 5056 and 7331). SPEC's Bylaws authorize only one class of membership in the corporation. It is understood that one sometimes sees the term "member" used in a looser sense, and indeed the law appears to recognize that the term "member" may be used in a less formal sense (7333). However, this section 1.6 follows the formal usage of the Bylaws, which clearly say that SPEC has only one class of "Membership".
1.6.1 Sustaining Memberships
Sustaining members are dues paying members and accrue all the rights, privileges and responsibilities that full membership entails. Sustaining memberships are available to computing-related commercial and non-commercial entities as specified in the Bylaws. Any eligible entity that wishes to join SPEC may join provided they pay the initiation fee and annual dues as determined by the Board of Directors. The fees and dues for sustaining membership must not discriminate among prospective members.
Universities and non-profit institutions are eligible to become sustaining members, but at their option may prefer to join as "Non-profit Associates", as described in section 1.6.2.
Sustaining members may become active participants in any working group or subcommittee of the OSG.
Sustaining members are eligible to stand for election to the steering committee seats, to nominate candidates for offices and the Board of Directors, and to champion proposed benchmarks.
Sustaining members have the opportunity to review and comment on all benchmarks developed by the OSG. This can be done through active participation in the subcommittee developing the suite or during the benchmark's final call for comments.
1.6.2 Associates
To encourage participation from universities and other non-profit institutions interested in its work, SPEC maintains a SPEC Associate category that enables non-profit organizations to share in the SPEC process, but not to participate in general member voting.
Associates pay an initiation fee and annual dues as determined by the Board of Directors.
Non-profit Associates have the opportunity to review and comment on all benchmarks developed by the OSG. This can be done through active participation in the subcommittee developing the suite or during the benchmark's final call for comments.
1.6.3 Supporting Contributors
The role of a Supporting Contributor is available by invitation to a commercial organization, to an academic institution, or to an individual. A subcommittee may issue an invitation to an interested party to apply for this role. The application must be approved by the Open Systems Steering Committee and the Board. Supporting contributors may participate in only one subcommittee and have no voting rights and pay no dues. Supporting contributors may help develop new benchmark suites, provide expertise to the subcommittee or help provide other support within the subcommittee.
1.6.4 Table of Membership Rights and Privileges
Sustaining Associate Supporting
Member Contributor
---------- --------- ----------
Dues Full Discount None
General Member Voting Yes No No
Participate in meetings, Yes Yes Yes (for 1 subcommittee)
concalls, benchmark
development
Eligible for Subcommittee Yes Yes No
Voting Rights
Number of Subcommittees Any/All Any/All 1
in Group in Group
Copies of Benchmarks All Releases All 1
in Group in Group
On SPEC email aliases Any/All Any/All 1
in Group in Group
Access to members' private Yes Yes Limited access con-
server, or data from it trolled by subcommittee
Chair, to subcommittee's
materials
Free Results publication Unlimited Unlimited Subcommittee may award 1
on www.spec.org for free - if in support of
Group's benchmarks new benchmark release.
Otherwise non-member
price.
Term Limit Until stop Until stop 1 year. Subocmmittee may
paying dues paying renew it annually.
dues Subcommittee may terminate
at any time.
Results Review Yes Yes Cannot vote on results.
May participate in review
discussion only if allowed
by subcommittee.
1.7 Copyrights and Trademarks
The standards and documents authorized by the Corporation may be copyrighted or trademarked, as appropriate.
The purpose of the copyrights and trademarks is to assure the computer industry that programs and documents developed by SPEC are genuine. The intent of protecting by trademarks and copyrights is to promote confidence in the metrics, and prevent mis-representation of them.
1.8 Openness
Information about the activities and developments of SPEC are intended to be readily available to all parties with legitimate interests in the performance of advanced computer systems. The board is responsible to carry out this policy.
SPEC meetings must be open to representatives of each member company. For guidelines regarding participation by non-members (including press representatives), see section B.6.
1.9 Fair Use of SPEC Benchmark Results
Consistency and fairness are guiding principles for SPEC. To assure that these principles are sustained, Fair Use guidelines apply when making public comparisons that use SPEC benchmark results. The guidelines for SPEC as a whole are located at http://www.spec.org/fairuse.html and for the Open Systems Group at http://www.spec.org/osg/fair_use-policy.html and are incorporated herein by reference.
2. Open Systems Group
The Open Systems Group (OSG) is the original SPEC committee. This group focuses on benchmarks for workstations and servers running open systems environments.
2.1 Open Systems Steering Committee
The Open Systems Steering Committee (OSSC) manages and supervises the development of SPEC's OSG benchmarks. The intent of this section is to provide an overview of the general policies and principles of the Open Systems Steering Committee.
Exceptions to policy: For due cause, the OSSC may vote to approve exceptions to this policy or may adopt resolutions that may conflict with policies expressed in this document; in such cases an amendment to this document will be brought forth within a reasonable time period afterwards.
Conflict with Bylaws or Board: In the event of disagreement between OSSC resolutions vs. SPEC's Articles of Incorporation and Bylaws or decisions by the SPEC Board of Directors, the OSSC shall yield.
Size of OSSC: The Board sets the number of OSSC members, currently seven. The OSSC may recommend to the Board an increase or decrease in size based on activities and participation.
Qualifications: The OSSC is elected from the OSG membership. OSSC members agree to the following conditions and obligations:
- Have the engineering and technical resources to run all benchmarks, and commit to running all proposed OSSC benchmarks.
- Devote the equivalent of one full-time engineer to SPEC activities.
- Commit to developing benchmark suites for high performance computer systems - the goal of the SPEC organization.
Election timing: To maintain continuity and consistency of representation on the OSSC, the OSSC has staggered two-year terms, with half of the OSSC elected each year. Elections to the OSSC are held at the annual meeting.
Vacancy: If an OSSC member institution resigns or leaves, the Board may appoint another OSG member to fulfill that OSSC member's remaining term, or may choose to leave the seat vacant until the next Annual Meeting, thereby temporarily decreasing the size of the OSSC.
Affiliation: Entering into a relationship of affiliation (as defined in the Bylaws, section 2.1) does not alone constitute the creation of a vacancy. (For example, Company B is not a member of the OSSC. It buys all the assets of, and takes on all the responsibilities of, Company A, including Company A's OSSC seat.) If, however, such affiliation would cause or appear to cause a member and its affiliate(s) to have two or more votes on the OSSC, then, as described in the Bylaws, a single voting representative shall be designated and one or more vacancies shall be declared.
Voting representative: Each OSSC member institution shall designate a voting representative. If the designated voting representative is absent from an OSSC meeting, another person employed by the member institution (or its affiliates) may temporarily represent the member. In the event of a lack of clarity as to who has the vote, the OSSC Chair may refuse to accept a vote until the designated voting representative appoints a substitute in writing.
Proxies: The OSSC does not allow use of proxy voting (that is, the authorization of an individual who is not an employee of a member institution to vote on behalf of that member), because OSSC members are expected to participate, to be actively involved in discussion, and to evaluate the evidence that may be presented. In rare cases, the Chair may acccomodate legitimate scheduling difficulties (e.g. a Special Meeting scheduled at a particularly user-hostile hour for a directly interested participant), by requiring use of the Voting Tool rather than taking a vote during the meeting.
Officer elections: The OSSC elects a chair. The chair serves for one year and is eligible for re-election. The OSSC chair schedules election of OSSC officers for a face-to-face meeting in the US, 3 to 6 months after the Annual Meeting of SPEC.
OSSC Quorum: In order to have quorum, the OSSC must have greater than 2/3 of its members present. (To calculate this, the table for the "SPEC Consensus Voting Rule" may be referenced in section 2.2.5.) Votes may be taken only if there is a quorum of the current voting members present.
Voting; passage of motions: To pass a proposal at the OSSC, the OSSC uses the "SPEC Consensus Voting Rule", which is defined in section 2.2.5.
All OSSC votes shall be open and public votes, except election of officers, which is by secret ballot. Exceptions to the secret ballot may be made if there is only one candidate per office.
It should be noted that the lack of passage of a motion by the OSSC does NOT imply approval of the CONVERSE of the motion.
Voting Tool: The SPEC private server offers a "SPEC OSSC Proposal/Voting Tool" which may be used to act on proposals separately from meetings. When this tool is used, it automatically enforces the SPEC Consensus Voting Rule.
Advisers: The OSSC can, after notification in writing to the Board of Directors, enlist the aid of advisers and consultants from the industry and academe.
Coordination with Board: Certain organizational activities require board participation and approval. For example, the board must approve press releases, setting prices for benchmark licenses, and any major resource reallocations of SPEC administrative staff and Public Relations to assist with the non-technical aspects of releasing a new benchmark suite. This work may be facilitated by a joint committee of the board and the OSSC (e.g. Planning Committee) or proposals may be brought directly to the board by the Subcommittee or Steering Committee Chair.
2.1.1 OSSC Meetings
Parliamentary procedure: OSSC meetings are conducted in accordance with general parliamentary procedures under the direction of the chair. Meeting participants may remind the chair of such principles, e.g. by raising a "point of order" from Robert's Rules, but the chair is responsible for interpretation of such principles, which in some cases may specify a greater degree of complexity than is required in order to accomplish SPEC's business. In such interpretations the chair shall use fairness as the primary criterion, and simplicity of process as a secondary criterion. The chair's interpretations may be over-ruled by the OSSC using its usual voting rules.
Regular and special meetings: OSSC meetings are of two types: regularly scheduled, or special. Special meetings may be called by the Chair, the Vice-Chair, or any 3 members, to handle urgent issues.
Notice of meeting: At least two full business days' notice is required prior to OSSC meetings, unless every voting member agrees to shorter notice. For example, if the meeting is called for Tuesday at 11AM PST, the meeting notice needs to go out by 11AM PST on the previous Friday.
Venue: Meetings may be face-to-face or by teleconference.
Face-to-face meetings: Face-to-face meetings take place about 4 times per year. Regular face-to-face meetings are considered essential because the steering committee often:
- Drives for consensus on controversial issues.
- Functions as a court of appeal for issues elevated from subcommittees.
- Seeks solutions to complex issues, while upholding SPEC's principles of effective and fair benchmarking.
- Engages in negotiations that require substantial time, effort, and understanding of other participants' concerns.
Remote call-in access to face-to-face meetings is typically not provided, as that would defeat much of the purpose of face-to-face meetings. Candidates for OSSC are strongly encouraged to take account of the face-to-face meeting requirements prior to running for office.
Meeting protocol: For regularly scheduled meetings, the OSSC has adopted the following meeting protocol:
- Written proposals submitted in advance of the meeting will be given priority on the agenda.
- A written form of the proposal must accompany all non-trivial proposals.
- Guest presentations may be scheduled for no more than one hour in total for all guests in any one meeting.
- Meetings include:
- A short session to adjust the agenda
- Status reports from each active subcommittee or working group
- Old business
- Action item followup
- New business
- Notice of the time and place of the next meeting
- Announcements
- If time permits, the agenda may also include a section for informal discussions to allow the group to begin discussion on relevant topics that are likely to require action at future meetings. During informal discussions, no OSSC votes will be taken.
2.1.2 OSSC Web Pages and Email Subscription
Steering Committee web pages (including OSSC Wiki pages) may be read by all members of the Open Systems Group. Any OSG member may subscribe to the email list for the OSSC.
2.2 OSG Guidelines for Working Groups and Subcommittees
The goal of the OSG is to produce benchmark software to characterize the performance of modern computer systems. To accomplish this work, the OSSC will from time to time charter subgroups of the membership with the responsibility to research and produce relevant benchmarking products.
There are two types of groups that the OSSC may charter: working groups and subcommittees. The requirements and goals for each are generally distinct, though occasionally there may be some overlap. This section attempts to clarify the purpose, organization, and responsibilities of each and to provide guidelines for the conduct of these groups.
2.2.1 The Working Group
As technologies develop, the OSG needs to keep pace and look toward the development of benchmarks to address the new technologies or to update older benchmark suites. To begin this process, the OSSC will sponsor the formation of a "working group". There will be a steering committee vote and the OSSC will appoint a chair. Specific notice will be sent to the Open Systems Group General Membership on the formation of a new working group. Participation in a working group is open to all interested OSG sustaining members, and all OSG non-profit associates. A supporting contributor may be invited to participate in a single working group. A working group takes on the following responsibilities:
- Research the technologies of interest to determine the requirements for developing a benchmark to address system performance in that space.
- Provide periodic progress reports to the OSSC, the OSG General Membership and, if appropriate, to other groups within SPEC.
- Produce a final report or project plan that describes the group's recommendations for benchmark development and an estimate of the resources required to develop and release the benchmark within the time frame deemed critical by the group. A draft charter or mission statement for the benchmark should be included. Other things covered would include what to test, how to test, what to measure, why the benchmark is needed, and an overview of the aspects of the technology to be benchmarked.
Additionally the working group should work towards:
- Increasing the visibility of their work, with an eye toward attracting participation of existing SPEC members and/or new SPEC members.
- Soliciting candidate benchmark codes from various sources including existing SPEC members, academia, public domain, or other third parties.
- Providing prototype benchmark tools and candidate codes, if practical.
2.2.2 Subcommittee Formation
A subcommittee's role is to utilize the research and recommendations of the working group and produce finished benchmarks and run rules that can be released after completion of Benchmark Release Approval.
Once the working group's report has been delivered to the OSSC, the OSSC must evaluate the report and decide if a subcommittee should be chartered to take up the task of producing a benchmark for release. Alternately, the OSSC may assign the work to an existing subcommittee.
The OSSC provides the charter for the subcommittee, and must approve changes to it. (A working group produces a draft charter, which the OSSC adjusts as it deems appropriate.) The charter must be posted at one of the subcommitte's top-level web pages on SPEC's private server.
The OSSC must also determine if there is sufficient commitment from interested members to complete the benchmark. If sufficient resources can not be identified, the OSSC may either table the project or direct the working group to continue with the resources available until enough members can actively participate to justify the founding of a subcommittee.
A minimum of three active members is required for the formation of a subcommittee.
2.2.3 Joining a Subcommittee
Membership in any OSG subcommittee is open to any OSG sustaining member or OSG associate. Subcommittees may also include members who are non-voting supporting contributors.
Terminology note - Notice that it is therefore possible to be a member of a subcommittee without being a "sustaining member" of SPEC. For details of types of membership of SPEC, see section 1.6.
It is not required that one join a subcommittee as a voting member; informal participation by sustaining members and non-profit associates is also welcome. Informal participants may join in activities of the subcommittee, as they are able. This includes making themselves heard at meetings and contributing to the completion of the benchmark. They also have the opportunity to review and comment on benchmarks in development during the benchmark test phase and during the final call for comments.
2.2.4 Subcommittee Voting Rights
To join a subcommittee as a voting member, the joining party must:
- Be an OSG sustaining member or OSG associate.
- Agree in principle with the goals of the subcommittee as outlined in its Charter.
- Identify a main contact.
- If the subcommittee has been active for over three (3) months:
- Demonstrate its commitment during an initial four week period, meeting the contribution requirements of section 2.2.4
- During its initial four weeks, take responsibility for bootstrapping itself in the past efforts of the group. This includes reviewing meeting minutes and the group's email archive and obtaining assistance offline from the meetings to ensure they are up to speed on the issues.
- Attend two (2) consecutive meetings in addition to the requirements listed above.
In order to maintain voting rights in a subcommittee, a member must:
Contribute resources towards the actual work of developing the benchmark. The recommended minimum contribution is 48 person-hours per month This consists of approximately 40 hours for technical work on the benchmark plus 8 hours for meeting and conference calls. Contributions may be divided among one or more individual contributors.
The subcommittee may adjust these requirements, based on the subcommittee schedules, the effort required to meet them, and the number of members participating.
-
The tasks that an active subcommittee member may contribute to include:
- developing and maintaining candidate codes and workloads
- review of results
- developing and maintaining benchmark tools
- porting and testing the benchmarks and tools on distinct architectures or operating systems
- writing, editing, and testing documentation
- developing, editing, and staging webpages
- editing the run and reporting rules
- answering technical support questions
- performing the duties of organizational roles, such as Chair, Vice-Chair, Secretary, or Release Manager
Status as a voting member may change:
If a member misses two consecutive duly noticed meetings, including face-to-face meetings and conference calls, the member's voting rights will be suspended. Missing a multi-day face-to-face meeting counts as missing a single meeting. It is understood that sometimes at face-to-face meetings, members may have to "timeshare" themselves between multiple subcommittees. This can be done with approval of the affected subcommittee Chairs.
The suspension of voting rights will take place at the beginning of the next scheduled meeting. Voting rights are reinstated at the beginning of the second of two consecutive meetings that the member attends.
Additionally, the voting rights of a subcommittee member will be suspended at the beginning of a meeting if the subcommittee member withdraws its resources. Voting rights will be reinstated after 4 weeks of the member contributing resources (e.g. 48 hours) or after having caught up with the assigned tasks.
When a member wishes to regain voting rights, that member will send an email message to the Chair stating the work that they have done, and a commitment to continue providing the minimum contribution level.
Monthly reporting and effect on voting rights: Subcommittees are strongly encouraged to require submission of monthly status reports and may use these to help assess whether contribution requirements are being met. If so, the subcommittee must clearly define when reports are due, how they will be assessed, and how voting rights may be affected by them. The following process is recommended, but may be adjusted by a subcommittee. Any adjusted process must retain a mechanism for protest of assessments similar to item 6 below, to ensure protection of the rights of potentially disenfranchised members.
Reports are due on the last day of each month.
Satisfactory contribution reports list tangible (i.e., backed up by a deliverable and/or evidence) contributions made, such as: checked-in/posted code, checked-in/posted bug fixes, documented bug reports, documented code reviews, checked-in/posted porting, testing bug fixes/new versions of benchmarks and providing written test reports, performance analysis reports, benchmark documentation, documentation edits, run rule editing, technical analyses, minute taking, chairing/running/leading/facilitating meetings, SPEC web site content contributions, assisting users with benchmark questions and configuration issues, managing releases, configuration management, results reviews, and participating in working group meetings.
Result submission activities do not qualify as contributions. Activities which do not result in a tangible deliverable to the subcommittee generally do not qualify as contributions. For example, testing bug fixes/new versions of a benchmark under development do not qualify as a contribution unless the results of that testing are reported to the subcommittee.
Prior to the first meeting of a month, the Chair will send an email to assess which contribution reports meet subcommittee requirements. If any are negative (i.e. assessed as inadequate), the Chair will plainly state why. Members may correct deficiencies or file late reports prior to the subcommittee's first meeting of the month.
If two consecutive assessments are negative, a member's voting rights will be suspended as of the beginning of the next meeting after the Chair's email.
In the event of disagreement with the Chair's assessment of contributions, whether positive or negative, any member may protest the assessment, and a proposal may be made to overturn it. For such a proposal to be considered, it must be sent prior to the second meeting following the sending of the assessment, or must be entered at the beginnning of said second meeting; otherwise, the Chair's assessment stands. Proposals to overturn will be considered using the subcommittee's usual voting rules, with the proviso that members whose voting rights would be affected by passage of any such proposal(s) at that meeting are nevertheless entitled to vote on all such proposal(s) that are under consideration at that meeting.
Example: a member's April contributions are ruled inadequate on May 1. There is no protest. The same member's May contributions are ruled inadequate on June 1. The member's voting rights are suspended, unless a proposal is made and passed to overturn the second assessment. It is too late to protest the first one.
2.2.5 Subcommittee Organization and Voting
Upon subcommittee formation, the OSSC appoints an initial Chair. After 3 to 6 months, preferably at a face-to-face meeting, the subcommittee will elect a Chair and Vice-Chair, and optionally a Secretary and Release Manager.
Thereafter, subcommittee elections will take place at the SPEC annual face-to-face meeting.
Subcommittees must hold an election to fill a vacancy when an officer steps down, except for the case where the Chair steps down and a Vice-Chair has already been elected; in this case, the Vice-Chair shall become the Chair for the duration of the term, and an election for a new Vice-Chair must be held to fill the vacancy.
A list of the elected subcommittee officers must be readily available to any SPEC member, either via the internal subcommittee page or the SPEC Wiki.
Should active membership increase beyond 7-10 active members, the subcommittee may request the OSSC approve a "Steering Committee" structure. If approved, the subcommittee would then elect its own Steering Committee members to direct the activities of the group and facilitate decision-making.
Quorum: The subcommittee should select its quorum to be either 2/3rds of the active members or a simple majority of the active members. It is also required that quorum not drop below 3 members. Votes may be taken only if there is a quorum of the current voting subcommittee members present.
Voting: A subcommittee should drive for consensus among its participants whenever possible. However to ensure progress, formal member votes will need to taken. An active member may vote 'yes', 'no', 'abstain', or 'pass' when the vote is first called. If the member votes 'pass', the chair will return to that member to finalize their vote as either 'yes', no', or 'abstain' after the remaining members have voted. Votes of 'abstain' count only toward establishing quorum. The chair should see to it that members' reasons for voting 'no' or 'abstain' are recorded in the minutes. Such recording can help with the drive for consensus, as it allows others to respond to the specific concerns of those who do not vote 'yes'.
Passage of motions: The subcommittee selects whether passage of motions requires simple majority of the people voting yes' or 'no', or >2/3rds of the people voting 'yes' or 'no', known as the "SPEC Consensus Voting Rule". The advantage of simple majority is that it may allow progress to be made more quickly; the advantage of the consensus voting rule is that it may force deeper exploration of common ground. (The consensus voting rule may also sometimes lead to deadlock.) A proposal to switch voting rules is itself bound by the rule currently in effect. The table that follows details the number of votes required to pass a motion if the SPEC Consensus Voting Rule is in effect.
| SPEC Consensus Voting Rule | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Voting yes or no | 3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
| Votes required | 3 |
3 |
4 |
5 |
5 |
6 |
7 |
7 |
8 |
9 |
9 |
10 |
11 |
2.2.6 Subcommittee Election Procedures
- Elections shall be announced at least 4 weeks in advance, with date and time expressed using US Pacific Time (PDT or PST, according to the season). As mentioned above, elections take place at the annual face-to-face meeting and at certain other occasions as may be required. For such other occasions, face-to-face is preferred, but a teleconference may also be used.
- Nominations may be made via email to the subcommittee mailing list.
- A candidate may nominate himself or herself.
- In order to be considered, a nomination must be seconded and the candidate must agree to the nomination.
- At the beginning of the meeting where the election is to be held, the Chair shall confirm who are the active/voting members.
- At the meeting, an additional opportunity shall be provided to make nominations, to second nominations, and for candidates to accept or decline nominations.
- A formal motion shall be made and passed to close nominations.
- After the close of nominations, for contested elections, an opportunity shall be provided for short statements by all candidates. Candidates also have the option of stating their credentials via email.
- For non-contested elections (i.e. one nominee for the position), a voice vote may be taken. If all positions are uncontested, a voice vote may be taken for the entire slate.
- For contested elections, a subcommittee member who is not standing for the position in question shall preside during the voting for that position.
For contested elections, the voting procedure depends on whether the vote is at a face-to-face meeting:
- If the election is face-to-face, written (secret) ballot shall be used. The presiding officer shall count the votes and announce the winner, with a second member confirming.
- If the election is not face-to-face, then the ballot may be secret on request from any member. In such a case, the presiding officer for the election, and a member of the SPEC Office, shall count the votes and announce the winner.
- The winner is the candidate who receives the highest number of votes (i.e. a plurality is sufficient).
- In the event of a tie for highest number of votes, a runoff election shall be held to decide among the tied candidates. If necessary, this step shall be repeated. If the tie cannot be broken after 3 attempts at a runoff, then the Chair of the steering committee shall use a randomized method to determine the winner.
2.3 OSG Guidelines for Result Submission and Review
The following section address the guidelines for submitting benchmark results to the OSG for review and publication on SPEC's website.
Each benchmark suite produced by the OSG includes a set of specific run and reporting rules that the licensee must follow to produce a publishable result. These basic tenets of these rules require:
- Proper use of the SPEC benchmark tools as provided.
- Availability of an appropriate full disclosure report.
- Support for all of the appropriate protocols and standards.
Furthermore, SPEC expects that any public use of results from this benchmark suite shall be for systems and configurations that are appropriate for public consumption and comparison. Thus, it is also expected that:
- Hardware and software used to run this benchmark must provide a suitable environment for the application area(s) targeted by the benchmark.
- Optimizations utilized must improve performance for a larger class of workloads than just the ones defined by this benchmark suite.
- The tested system and configuration is generally available, documented, supported, and encouraged by the providing
vendor(s).
SPEC reserves the right to adapt the benchmark codes, workloads, and rules of its benchmarks as deemed necessary to preserve the goal of fair benchmarking. SPEC will notify members and licensees whenever it makes changes to a benchmark or its run and reporting rules.
2.3.1 Results Review
Each subcommittee is responsible for establishing a regular review process for result submissions for the benchmarks that it develops. Typically, this is a two-week cycle where any results received by the start of a review cycle are peer reviewed by the subcommittee members over the following two weeks. At the end of the review, if there are no open issues related to a given submission, the result is accepted and the full disclosure report is posted on the SPEC website. Once posted on the SPEC website (www.spec.org), the information contained in the result disclosure is considered public information.
If there are open issues for a given submission at the end of its review cycle that have not and can not be resolved to the satisfaction of the subcommittee, the following process must be followed:
- A proposal is made that the submission is not compliant and should not be published on SPEC's web site.
- The proposal is voted following the subcommittee's usual voting rules.
- If the proposal is approved, the submitter may acquiesce to the vote and withdraw the submission.
- Alternately, the submitter may require the subcommittee to forward the submission to the OSSC for review. In this case, the subcommittee chair or designee must contact the OSSC chair at the conclusion of the review meeting. The submission is moved under the category of Pending. Relevant supporting documentation and arguments should be forwarded by both sides to the OSSC. The OSSC is generally expected to complete its review by the next scheduled review meeting or teleconference of the subcommittee.
- The OSSC undertakes a second review of the results to determine whether the submission adheres to the letter and spirit of SPEC's run and reporting rules. If an OSSC review is undertaken, the OSSC must vote on a proposal to accept the submission for publication.
- Should the OSSC not accept the submission, the submitter may still make an appeal to the SPEC Board of Directors.
- The submitter may withdraw, correct, or replace the questioned submission at any time during this process.
If minor corrections are made to a submission, e.g. documentation of tuning options or product names, which do not affect performance, then the corrected result remains on the same publication schedule as the original submission. If major corrections are made, e.g. a new run, different performance results, or different system configuration, then the corrected result starts review over as a new submission.
Expedited review: Under the direction of its Chair, a subcommittee may allow an "expedited review" of a corrected result, for example, a 2-business-day review of a re-run. An expedited review is typically appropriate when minor changes are made, and only the minimum changes needed to meet SPEC's concerns. It is typically not appropriate when additional changes have been made. When deciding whether to allow an expedited review, the Chair's primary criterion should be whether the result can receive a full and fair review, taking into account the degree of change, the availability of reviewers, and the other workload in the subcommittee. The Chair's determination may be over-ruled by the usual voting rules of the subcommittee.
To assure fairness to the submitter, the OSG has established the following guidelines regarding the proper use of review materials:
New and unique optimization techniques that appear in a result disclosure should not be utilized by another member until the original submission has completed its review cycle. The subcommittee may decide to suspend a result in review should this guideline not be followed.
- Trying to surpass a result currently under review is considered common practice and is not prohibited.
- Published information found in press releases etc. can be used at any time.
Members participating in the review of results submissions are expected to be familiar with the Guidelines for Handling SPEC
Information, particularly section 7.0 dealing with Results Review (see Appendix B.)
2.3.2 Handling OSG Run Rule Violations in Published Results
The subcommittee may also [vote to] undertake a re-review of a previously published result, if significant questions as to its adherence to the applicable run and reporting rules arise. The OSSC reserves the ability to denote a publish result as non-compliant if it has been found in violation of the run and reporting rules.
The following process has been established to handle instances where run and reporting rule violations are discovered in results that have been published on SPEC's web site.
The subcommittee is responsible for identifying that a result published on the public web site is in violation of their benchmark run and reporting rules. Once a subcommittee identifies such a result, the result may re-enter a review process unless the submitter agrees with the finding of non-compliance.
If the subcommittee's review returns a finding that the result is not in compliance with the benchmark's run and reporting rules,
the following should occur.
The results are modified:
- To remove the metric values (summary & individual) and replace with "NC" (non-compliant) .
- Add a header to the disclosure stating that the result did not comply to the rules and the reason(s) why.
- The submitter, with the subcommittee's approval of the text, may add additional wording into a remedy section. The remedy section is expected to explain how the vendor has addressed the problem. For example, mentioning that a new result has been submitted on the same platform with the issue resolved is the expected typical usage of this section.
2.3.3 Handling Continued Availability in Published Results
The various SPEC benchmarks' Run and Reporting Rules have clauses that at a certain time after the first publication (on SPEC's web site or otherwise), all components of the System Under Test (SUT) must be available for general customer shipment. The purpose of this requirement is
- to ensure that the results refer to a real, available system;
- to enable others to verify reproduce-ability of the results.
For the case that this availability status cannot be maintained over a minimum time length, the following rules apply:
Either at the request of the test sponsor or as a result of a subcommittee resolution, the subcommittee can undertake a re-review of a previously published result if a performance-relevant component ceases to be available for 30 days or more within 90 days after initial overall availability (i.e. initial availability of all components of the SUT). If the subcommittee's review returns a finding that this non-availability condition holds for a performance-relevant component, the result is marked "Not Available" (NA) in the following way:
- The metric values (summary and individual) are removed and replaced with "NA" (not available).
- A header is added to the disclosure stating that the result is for a system that is, at a time specified in the header, not available.
- The test sponsor can, with the subcommittee's approval of the text, add additional wording into a remedy section. The remedy section can for example, identify the part that is not available, contain the reason for non-availability, and/or an estimate when all components of the SUT are expected to be available again. If and when the SUT becomes available, the submitter may notify the subcommittee that the product is available. With the Subcommittee's agreement, the NA marking will be removed from the result page, however the page will retain the notice that the page was marked NA along with the history of the NA marking. This restarts the clock for the continuous availability requirement covered in this section.
For results not published by SPEC, the subcommittee can ask the publisher to withdraw the publication.
"Performance-relevant" is defined in the various benchmarks' Run and Reporting Rules' reproduceability clauses.
Replacement by a new similar component with equivalent or better performance is considered acceptable and not a reason for an NA marking.
In cases where the Run Rules allow use of a beta product, this beta product or a later equivalent or better performing regular product (see clause above) must remain available.
Note: Effective for results published on or after July 1, 2002.
2.3.4 Regarding the calculation of a required date for availability
Availability has sometimes been referenced as "within 3 months" or "within 90 days" of first publication. Both of these phrases have sometimes introduced questions as to interpretation. To avoid confusion, benchmark run rules must shift to using "within 3 months", which shall be interpreted as follows:
- Determine the date of first publication.
- If SPEC is the first to publish the result, the date of publication can be found via the records maintained by SPEC's editorial staff as to when results move to www.spec.org.
- If some other entity publishes the result first, the publication date is considered to be the date according to the timezone in effect at the headquarters of the entity publishing the result.
- Increment the month number by 3, modulo 12, to calculate the required month.
- Keep the day number the same.
- In the event that the day number does not exist in the required month, select the highest day number that does exist.
- The product must be available on the calculated date (any time zone).
Examples:
- A result is published on April 15. The product must be available on July 15.
- A result is published on 30 November 2012. The product must be available on 28 February 2013.
- A result is published at an industry conference in London, England (GMT+0), by Company A just after 11:00 AM on January 15. Company A has its corporate headquarters in Christchurch, New Zealand (GMT+13, with Daylight Savings Time in effect) where the date at that moment is January 16. Therefore, the date of publication is considered to be January 16. The result uses software that becomes generally available via download through Company A's electronic store. At the moment that order fulfilment begins, it is still April 16th on part of the planet (e.g. in areas that use GMT-12). The product was available within the required time window.
2.3.5 SUT Availability for Historical Systems
In the interest of providing some perspective on the performance and/or power consumption for older or obsolete systems, SPEC may consider review and publication of historical results.
Such publication may be considered for systems where the general availability date is beyond general availability limits expressed in the run rules, or for systems which may no longer be supported by their original vendor.
The hardware and software availability dates published on the result page will normally reflect the original general availability of the platform. In some cases, the technical subcommittee may choose to allow publication using a date that reflects general availability of what the review committee considers to be the "core components", for example (depending on the benchmark and the historical question under consideration), the chipset and processor combination; or (as another example) the primary software under test.
For historical results, this entry must be included in the appropriate Notes section:
The system as described on this result page was formerly generally available, but is no longer generally available. It may or may not be supported. This benchmark result is intended to provide perspective on past [power and/or] performance for the hardware and software described on this result page.
In the above, include the phrase in [square brackets] for benchmarks that include a power component.
2.3.6 Corrections to published results
SPEC's Editorial Committee controls both the mechanics of publishing results (e.g. procedures for transferring raw experiment data to SPEC in the required formats) and the updating of already published results. The Editor is responsible for keeping SPEC licensees informed about how to accomplish these tasks, via maintenance of appropriate instruction pages on SPEC's server.
The key points of the Editorial process for correcting results can be summarized as follows:
Requests for corrections are made to the Editorial subcommittee, copying the relevant technical subcommittee.
Spelling and technical corrections are always encouraged. Result pages may, and should, be corrected to improve their accuracy and to improve customers' ability to reproduce results.
Questions of rule compliance are not decided by the Editor; compliance issues are first examined in accordance with SPEC's process for "Violations Determination, Penalties, and Remedies".
For systems tested and submitted in advance of their general availability date it is understood that descriptions may change (e.g. naming of features, setting of defaults, listings of included components).
Whenever the need for such a correction is realized, it should be brought to the Editor's attention.
Availability dates may be updated, so long as such changes do not introduce compliance issues (e.g. moving outside the 3-month window).
In some cases, ongoing development between original announcement and actual availability may lead to multiple updates. In such cases, the final description should be the description that best reflects how the customer would order or configure the SUT, as of its General Availability date.
Note that individual benchmark run rules, or individual testers, may record additional technical detail beyond those recognized by customers, such as notations of internal nightly builds. Such documentation may be useful, but is a lower priority than to accurately reflect the customer perspective.
For those results measured and/or submitted after the system's general availability date, the system description information should be accurate at the time the result is published.
"Re-branding" a product or company name in a result page after the fact is generally not considered a correction, and hence is not allowed.
For example, with a result submitted for a system originally sold as a 'Company ABC Model 2', a later request to change the result page to say 'Company XYZ Model 2' would not be seen as a correction.
However, in such situations, Company XYZ may choose to submit a new result page with this new 'Company XYZ Model 2' brand name; and if the systems truly are identical in all but name, Company XYZ may even choose to re-submit the original raw result files just with the updated brand names.
2.4 OSG Technical Support Guidelines
The members of the OSSC and its subcommittees are responsible for providing adequate resources to handle technical support questions regarding current benchmark suites. It is the goal of technical support to provide timely response to questions regarding the nature and usage of these benchmark suites to help facilitate testing by other SPEC benchmark licensees. The purpose of SPEC's technical support is NOT to provide OS, compiler or other vendor tool support or assistance. At the technical support person's discretion, the user may be referred to a third party, who may charge a fee for such support.
For each released benchmark, the SPEC administrator will maintain an email alias (i.e. cpu95support@spec.org). The committee responsible for that benchmark will establish a quarterly rotation schedule among its active members (based on license number) to serve as technical support person.
In special cases a member may be assigned to provide support for specific subset of technical support questions for an extended period or to handle support for a specific geography. The email alias includes the subcommittee chair, representatives from the company up for that quarter's rotation, members with special duties and OSG chair. The chair's responsibility is to see that questions are answered promptly and provide backup when the vendor on rotation becomes unavailable. The current support representative or chairs may assign a technical support call to another subcommittee member, if they believe vendor specific knowledge may be required to resolve the call.
The SPEC administrator remains the initial point of contact for technical support questions. The administrator will make the determination as to which email alias the problem is reported to.
2.5 OSG General Member Guidelines
The OSG is open to any organizations that have an interest in the use and development of standardized performance benchmarks for computer systems running open systems environments. By joining the OSG, members may:
- Receive copies of all currently released OSG benchmark suites.
- Freely distribute copies of OSG benchmark suites within their company, including wholly owned subsidiaries. Country export and any other restrictions must be adhered to as per the license agreement.
- Submit an unlimited number of benchmark results to the OSG for review and publication on SPEC's website.
- Join any or all the currently active subcommittees and working groups and become active members by contributing engineering resources to the development of new benchmark suites.
- Join any or all of the OSG email aliases and receive access to the members-only website.
- Attend any or all meetings of the OSSC, OSG subcommittees and working groups, and SPEC general membership, and open SPEC Board of Directors meetings.
- Stand for election to the OSSC according to the guidelines above.
- Receive access to preliminary (beta) versions of benchmarks currently in development for internal review and to provide feedback to the appropriate subcommittee.
- Vote on any proposed benchmark suite submitted for General Membership approval. Note: Associates may not vote on benchmark
releases.
Members are expected to uphold the policies and principles endorsed by SPEC and the Open System Group. This includes policies described in this document and amended by resolutions of the OSSC; SPEC's Articles of Incorporation and Bylaws; policies adopted by the SPEC Board of Directors; and any applicable set of benchmark run and reporting rules.
Members are also expected to remain members in good standing by the prompt remittance of annual dues. Membership privileges will be suspended if dues have not been received by the SPEC administrator by March 31st. Membership privileges will be restored once dues have been paid.
2.6 OSG General Membership Voting
As a sustaining member of SPEC and the OSG, each member has the right to participate in any ballots of the general membership. The OSG membership votes to elect members to the Open Systems Steering Committee (OSSC).
Elections for seats on the OSSC are held at the Annual Meeting in January. The OSSC has staggered two-year terms, with half of the steering committee elected each year. Any OSG member in good standing may vote.
2.7 Benchmark Release Approval
The process to approve the release of new benchmark suites consists of several steps. The subcommittee that developed the suite must finalize the benchmark and prepare a beta version that can be recommended for general membership review. When this beta is ready, the subcommittee must send out a final call for comments to the general membership. A member has four weeks from that time to review and comments on the benchmark. The member should send their comments back to the subcommittee and the OSSC.
At the conclusion of the final call for comments period, the subcommittee will take a final vote to recommend that the OSSC approve the benchmark for release. After the OSSC has received the subcommittee's report and recommendations, they will vote on whether approve the benchmark for release or send it back to the subcommittee for further work.
The OSSC may also request that a general membership vote be taken to approve the release of a new benchmark. In a general membership vote on an OSSC's recommended candidate benchmark(s), a quorum of the OSG membership must participate in the balloting. The OSG defines quorum as one third of the OSG general memberships. A member may vote `yes, `no', or `abstain' to be counted in the quorum.
Ballots of the membership must be returned to the OSSC within a time limit specified by the OSSC, which must be no less than 21 days or more than 45 days. The balloting period provides members with an additional opportunity to do their own analysis on the candidate benchmarks.
Ballots not returned to the OSSC in the time allotted are not counted. Members can send their ballots through the mail, email, or fax and need not be present at the SPEC meeting to cast their ballot. To be accepted into a benchmark suite, a benchmark must receive more 'yes' than 'no' votes and a minimum of three 'yes' votes. Votes of 'abstain' count only toward establishing quorum.
The OSSC may approve minor changes to an approved benchmark as recommended by the benchmark's subcommittee (e.g., fix bugs or make other modifications that do not alter function) without resubmitting it to the general membership for a vote. For urgent cause, the Board of Directors may authorize the OSSC to take action inconsistent with a prior vote.
Urgent cause may include cases where an approved benchmark either violates the letter or spirit of SPEC's Articles of Incorporation and Bylaws; infringes on copyrights, trademarks, or patents, or on the advice of SPEC's legal counsel.
With the approval of the Board of Directors, the benchmark approved within the OSG becomes an authorized SPEC Benchmark Suite release.
Appendix A. OSG Benchmark Suite Development Overview
The Open Systems Group has to date developed two classes of benchmark suites: component suites (CPU92, CPU95, JVM98) and system-level suites (SDM, SFS93, WEB96, SFS97).
Component suites consist of a set of individual benchmark programs that are chosen on the basis of their abilities to demonstrate overall processor or system performance. The intent is to give prospective users a fair measure of capabilities over a range of real-world applications. To accomplish this, the benchmarks must meet the criteria set by the subcommittee. For example, candidate benchmarks for a CPU suite should be relatively long running and realistic programs that test a wide range of applications and system resources (integer, floating-point, cache, memory system, etc.). Whenever possible, the OSG will seek benchmark candidates from the open source software domain. The metrics reported for a component suite generally includes timings and normalized ratios for the individual codes and roll-up metrics that usually consist of a geometric mean of the individual ratios.
System-level suites use a flexible test harness to present a generalized workload to a system to measure performance in a multi-user or muli-client server environment. This has included muli-client testing of NFS file servers and HTTP Web servers and multi-tasking testing of UNIX software development environments. Typically, a system-level suite will run several data points as part of the test with each point applying an increased load to the system until the system reaches its peak throughput. The metrics reported for System-level suites generally include the throughput and response times for each test point and the corresponding plot of these points. A peak throughput is frequently used as the overall metric reported for the test.
Appendix B. Guidelines for Handling SPEC Information
B.1.0 Introduction
SPEC is an industry consortium whose membership includes many competing for-profit corporations. Membership in SPEC is open to all persons who are directly and materially affected by SPEC's activities, and participation in SPEC activities is generally open to all members. However, in order to protect the interests of SPEC and its members, and to encourage free and open participation and discussion to further SPEC's objectives, public release of SPEC internal information may be prohibited without explicit approval from the SPEC organization concerned. Violators of this policy will be subject to the full force of the SPEC penalties process.
B.2.0 Anti-Trust Compliance
Since all SPEC meetings are to be conducted in compliance with all applicable laws, including anti-trust laws, the following policies shall be followed in the course of SPEC meetings, telephone meetings, and electronic exchanges: Any discussions that relate to the validity or cost of patent use should be avoided. Any discussions or any material relating to an ongoing litigation shall be avoided, except for the Board in executive session with advice of counsel to discuss matters affecting SPEC. Any discussions of pricing or issues that would violate US antitrust laws shall be avoided.
B.3.0 How Information is Classified and Unclassified
SPEC officers and the SPEC Administrator may explicitly designate certain information classified and mark it appropriately. Any member that wishes to be sure that its information is treated as SPEC Confidential should clearly label each page (electronic or hard copy) as such. It is the member's responsibility to assure that such information is appropriately labeled. Information not explicitly marked but which meets the descriptions in this policy must be taken as though it were so marked.
Classified information may become public through the authorized public release of the information by SPEC or by a member. If classified information is disclosed without authorization (leaked), that does not automatically remove its classification and relieve participants of the obligation to protect that information from further disclosure. However, if an unauthorized disclosure is widespread, then a SPEC officer may redesignate that information as public, stating the reason for so doing.
B.4.0 Classifications
SPEC identifies two levels of internal information which must be protected by SPEC members and associates, employees, and contractors, in accordance with these policies. "SPEC Confidential" is the ordinary classification of information which should not be disclosed outside SPEC. "SPEC Confidential Need-to-Know" refers to the most sensitive information described below.
B.4.1 SPEC Confidential Need-to-Know
This is information that must be restricted to only those individuals who must have it in order to carry out their SPEC responsibilities. In meetings, this information is discussed only when the board adjourns to executive session. Only the author of such a document, or the SPEC President, can change the classification or the approved reader list. Examples of Confidential Need-to-Know information includes:
- information concerning litigation
- certain matters that relate to the formation of contracts with third parties
- employee salaries, performance appraisals, and other personnel matters
- administrative (root) passwords of SPEC business computers
B.4.1.1 Document Handling Guidelines
Documents of this sensitivity class will always be clearly labeled "SPEC Confidential Need-to-Know" on each page. The documents may be numbered for control purposes.
SPEC Private documents must have a cover page which serves to:
- hide the first page of the document,
- document the approved reader list, and
- document the date the information is no longer SPEC Confidential, if any,
Such documents should always be carefully transported. Documents should be kept inside an envelope, clearly labeled with the approved reader's name, when not in active use. Only persons on the approved reader list are privy to the document contents.
Email should be used sparingly for communicating such information. Messages should be sent directly to the intended recipient(s) rather than to any mail alias. Messages should be retained no longer than absolutely necessary, and should be kept separate from other messages. Encryption should be employed where possible.
B.4.2 SPEC Confidential
The following information is "SPEC Confidential":
- unpublished benchmark results, including configuration details, under SPEC review;
- benchmarks under development, including run rules, code, tools, workloads, benchmark and workload analyses, and benchmark design;
- the content of SPEC meetings;
- messages received on SPEC members email aliases;
- passwords to the SPEC members web site, and to accounts on SPEC computers;
- the content of the SPEC members web site;
- SPEC financial data;
- SPEC email and phone lists.
Note that non-members of SPEC should not be on mailing aliases. Please note that SPEC offers various levels of membership to fill different needs for information access.
B.5.0 Who May Have Access to SPEC Confidential Information
Confidential information is provided to SPEC members through their designated primary SPEC representatives, and those representatives of SPEC subcommittees designated by the member's primary SPEC representative. It is the responsibility of those representatives to determine who within a member company should have access to the information in order to carry out the member's SPEC activities; and it is their responsibility to ensure that anyone who does have access to SPEC Confidential information understands and agrees to abide by these guidelines. A member's SPEC representatives should use whatever internal controls are appropriate in their company to ensure that the member complies with SPEC's policies on confidentiality.
B.6.0 Meetings
Information about the activities and developments of SPEC are intended to be readily available to all parties with legitimate interests in the performance of advanced computer systems. The Board is responsible to carry out this policy.
The annual meeting of the SPEC membership must be open to any interested party with a legitimate interest in the work of the Corporation, including the press.
Other meetings of each SPEC group and subcommittee must be open to representatives of every member of that group (OSG, HPG, and GWPG). SPEC members who are not members of a particular group may nevertheless generally attend other groups' meetings, to encourage wider participation and exchange of ideas, unless the group has specifically decided to restrict attendance for some or all meetings. Such decisions are subject to review by the Board.
B.6.1 Non-member participation
The Chair of any SPEC meeting may invite limited participation by non-members who have a legitimate interest in SPEC's work, or who may possess information that SPEC would like to learn from. Such external participants must agree to SPEC's Guest Participation Statement. Exceptions to this requirement may be granted only by the Board.
B.6.2 Press Participation
Press representatives who are not SPEC members are generally excluded from SPEC meetings other than the annual meeting and SPEC's Public Workshops.
For other meetings, a meeting Chair may open the meeting to a representative of the press (who is not a member of SPEC) only if the press representative agrees to the confidentiality requirements in SPEC's Guest Participation Statement. Exceptions to the requirement for the participation statement may be granted only by the Board.
On the rare occasions that the press is in attendance, the Chair should announce press attendance prior to the meeting. The meeting Chair should request that the Press member use the attendance opportunity as "background" information in technical areas of performance benchmarking. SPEC is primarily an engineering organization which depends on frank and open exchange and debate of views. Views expressed in SPEC meetings are not expected necessarily to reflect official positions of SPEC member companies and organizations. SPEC asks that the press show restraint in any coverage based on SPEC meetings, so as not to jeopardize our open exchange of views.
B.6.3 Audio/Visual Recording
SPEC committees may apply restrictions on the use of audio recording, video recording, or photography equipment where they may impede free discussion, or where they are disruptive. Such restrictions should be clearly identified, in advance, to attendees.
B.7.0 Results Review
Benchmark results submitted for SPEC review, as for publication on the SPEC web site, are considered SPEC Confidential. This includes the benchmark result, the configuration, availability dates, and tuning notes. Within a member company this information should be shared with only those people necessary to provide a thorough review of the results, per paragraph 5.0.
Information which the submitting company makes public is not considered confidential. E.g., a company may issue a press release or briefing listing performance of new systems and, around the same time, submit those benchmark results for SPEC publication. In this case details included in the full SPEC disclosure under review, which were not included in the company's public announcement are not considered SPEC Confidential, because the run rules of each benchmark require the full SPEC disclosure to be publicly available.
If results are later published, then the result is public. If results are withdrawn or disapproved during SPEC review and are not published, then they remain SPEC Confidential. E.g., if a company submits results that are disapproved for SPEC publication, you may not make public use of that fact.
New and unique optimization techniques that appear in a result disclosure should not be utilized by another member until the original submission has completed its review cycle. The subcommittee may decide to suspend a result in review should this guideline not be followed. Trying to surpass a result currently under review is considered common practice and is not prohibited. Published information found in press releases, public web sites, manuals, etc. can be used at any time.
B.7.1 New Benchmark Announcements
More restrictive confidentiality rules may be applied by the SPEC groups on the occasion of results review for SPEC's announcement of a new benchmark suite, in addition to all of the preceding rules.
B.8.0 SPEC Press Relations
Press releases from SPEC must be approved by the SPEC group concerned, and by the Board.
SPEC members may occasionally be interviewed by the press, e.g. on the subject of newly released benchmarks. Normally SPEC's PR consultant will participate in such interviews in order to see that SPEC is treated fairly, and to assure all SPEC members that information is being represented in an unbiased manner. The Board Press Committee may on a case by case basis, waive this requirement, based on such factors as: degree of controversy of the subject, potential for appearance of conflict of interest by the spokesman, legal advice, advice from SPEC's PR consultant, recommendation of steering committee and technical committees and their chairs, experience of spokesman, experiences with the reporter, etc.
Appendix C. Guidelines for General Availability
One of the principal tenets of SPEC benchmarks is that the implementation of the System under Test (SUT) must be generally available, documented and supported. The run and reporting rules for each benchmark includes this statement. The available, documented, and supported trio is frequently referred to with the single term: general availability.
The purpose of including general availability requirements is to ensure that the systems and their hardware and software components actually represent real products that solve real business and computational problems. It is also intended to ensure that the benchmarked system does not contain so-called benchmark specials which improve benchmark performance but fail one or more of the tests of general availability. These tests include:
Has the system/component shipped or has the vendor committed to shipping the finished product within a time specified by the specific benchmark's run rules, usually 3 to 6 months from the date the result was made public?
A finished product will have the vendor's endorsement that the product is suitable for production use in an environment comparable to the one represented by the benchmark. Typically this version will be designated as the final released version and not designated as a prototype or a preliminary release (i.e. alpha, beta, field test, etc.).
Shipping the product refers to the vendor's standard process for getting said finished product into the hands of the end-user base. This can include putting it in a box and physically transporting it to end-user sites or simply advising the customer base that the product can be obtained on demand, for example by having an announced URL where the product can be downloaded or purchase information can be found.
The commitment to ship a product within a specific time frame provided in a results disclosure must be taken as a public statement of intent. Any potential end-user should be able to confirm the vendor's intent to ship the finished product within the stated time frame and that information should not be considered secret once the result has been publically disclosed.
Shipping a product generally requires that one or more instances of the final product be in end-user hands or enroute to end-users with delivery imminent and based on shipping method used.
Is the product documented such that there are written descriptions on the product, how to install it, how to configure it and how and when to use it?
The product documentation should be consistent with the deployment of the product in the benchmark such that there are no undocumented features employed solely for the benchmark. Availability of documentation should be concurrent with product availability.
Is the level of support reasonable for the type of product or component, and consistent with its importance within the system?
Levels of support can clearly have a wide range depending on the type of product or product component. SPEC understands that the determination of whether a level of support is "reasonable" may require a discussion and a ruling by the relevant subcommittee or by the steering committee. In such discussions, SPEC shall consider:
- Are the provider(s) of support clearly identified?
- Have clear procedures and expectations been established regarding support?
- Are support requests answered?
- If problems of a critical nature arise (i.e. those that affect the central purpose of the component, with no convenient workaround), are there mechanisms to ensure timely problem resolution?
If the answers to the above questions are "yes", this is taken as evidence that the product or component is supported. Notes: (1) Individual benchmark run rules may impose additional support requirements. (2) Sometimes, a component might be considered to have achieved a level of maturity such that support requirements are minimal or non-existent. If so, one or more of the above tests may be waived.
Regarding "paid support" vs. "community support": The identification of a paid support vendor may clarify responsibility for the useful operation of a product or component, and may increase confidence in its suitability for production use.
On the other hand, the requirement for paid support may exclude products for which standard benchmark results might be of interest, some of which are adequately supported by unpaid alternative mechanisms, such as support forums, bug mailing lists, wikis, and IRC channels. Such alternatives are herein designated as "community support".
For community supported products or components, SPEC shall consider:
- Is there an active support mechanism? (For example, is there traffic over the last 90 days? Postings from the benchmarkers or their representatives do not count in determining whether the mechanism is active.)
- Is the user interface usable, so that new reports can be filed in a straightforward manner?
- Is it possible to observe that support requests are being addressed?
- Can users benefit from previous support requests, for example via a search function, or via an evolving, useful knowledge base?
- If problems of a critical nature arise (i.e. those that affect the central purpose of the component, with no convenient workaround), are there community members who maintain the application?
If the answers to the above questions are "yes", that is taken as evidence that an active support exists mechanism exists, even in the absence of a paid support contract.
Does the product meet the vendor's own definition of general availability, in addition to all the other tests specified by SPEC?
Many vendors have their own internal processes for product development that set specific requirements for products to be considered generally available. If so, those requirements must be met for the product to be considered generally available by SPEC, in addition to the tests listed here. If the vendor has no such definition, then all other tests specified by SPEC must still be met.
A vendor is not required to discuss or disclose its internal processes to SPEC. However the SPEC representative is responsible to identify the internal definition most closely matching SPEC's intent of general availability, and applying that definition to any statements to SPEC regarding general availability. Note that the vendor may use different terms, e.g. full customer availability, open shipping status, phase 4 exit, etc.
-
Does the vendor represent the product to customers as being generally available?
This is not a clear cut test, as products may have several different distribution channels, and a product may not become generally available in all channels simultaneously. However, it is an indication that a product may not be generally available if the vendor specifically says in some venue that it is not. E.g.,
- Web page saying product is not available.
- Web page listing other configurations of the product as available, which fails to list the published configuration. (E.g., web page offers 1.5 GHz and 2.0 GHz models for sale but doesn't mention the published 2.5 GHz model.)
- Sales person will not take orders for the product.
- Statement by official company spokesman saying product is not available.
It is an indication that a product may be generally available if the vendor specifically says so in some venue. E.g.,
- Web page saying product is available.
- Advertisement for the product
- Sales person or reseller, possibly in a different locale, will take orders for the product.
- Statement by official company spokesman saying product is available.
In case of conflicting information, SPEC will weigh the preponderance of the evidence with the standard that it show the product to be generally available in some channel.
The rules for each benchmark may have slightly different requirements related to general availability and may specify other types of benchmark specials to avoid. The overall goal of ensuring fair benchmarking for the vendors submitting results and the end-users reviewing those results remains the same.
Note: The term "benchmark special" refers to any optimization which only improves benchmark performance.
Appendix D. Template SPECbenchmarkYYYY Run and Reporting Rules
This section contains a set of template run and reporting rules that subcommittees should use as a starting point when developing or revising benchmark suites. It contains the most common rule categories and policies that are shared by almost all current SPEC OSG benchmarks and have been approved by the OSSC.
Template SPECbenchmarkYYYY Run and Reporting Rules
Version 0.1 Last modified: July 11, 2001
Table of Contents
- 1.0 Introduction
- 2.0 Run Rules
- 3.0 Reporting Rules
- 4.0 Submission Requirements for SPECbenchmarkYYYY
- 5.0 SPECbenchmarkYYYY Benchmark Kit
1.0 Introduction
This document specifies the guideline on how SPECbenchmarkYYYY is to be run for measuring and publicly reporting performance results. These rules have been established by the SPEC Benchmark_name Subcommittee and approved by the SPEC Open Systems Steering Committee. They ensure that results generated with this suite are meaningful, comparable to other generated results, and are repeatable (with documentation covering factors pertinent to duplicating the results).
Per the SPEC license agreement, all results publicly disclosed must adhere to these Run and Reporting Rules.
1.1 Philosophy
SPEC believes the user community will benefit from an objective series of tests, which can serve as common reference and be considered as part of an evaluation process.
SPEC is aware of the importance of optimizations in producing the best system performance. SPEC is also aware that it is sometimes hard to draw an exact line between legitimate optimizations that happen to benefit SPEC benchmarks and optimizations that specifically target the SPEC benchmarks. However, with the list below, SPEC wants to increase awareness of implementers and end users to issues of unwanted benchmark-specific optimizations that would be incompatible with SPEC's goal of fair benchmarking.
SPEC expects that any public use of results from this benchmark suite shall be for Systems Under Test (SUTs) and configurations that are appropriate for public consumption and comparison. Thus, it is also required that:
- Hardware and software used to run this benchmark must provide a suitable environment for supporting the specific application area addressed by this benchmark using the common accepted standards that help define this application space.
- Optimizations utilized must improve performance for a larger class of workloads than just the ones defined by this benchmark suite. There must be no benchmark specific optimizations.
- The SUT and configuration is generally available, documented, supported, and encouraged by the providers.
To ensure that results are relevant to end-users, SPEC expects that the hardware and software implementations used for running the SPEC benchmarks adhere to following conventions:
- Proper use of the SPEC benchmark tools as provided.
- Availability of an appropriate full disclosure report.
- Support for all of the appropriate protocols.
1.2 Caveat
SPEC reserves the right to investigate any case where it appears that these guidelines and the associated benchmark run and reporting rules have not been followed for a published SPEC benchmark result. SPEC may request that the result be withdrawn from the public forum in which it appears and that the benchmarker correct any deficiency in product or process before submitting or publishing future results.
SPEC reserves the right to adapt the benchmark codes, workloads, and rules of SPECbenchmarkYYYY as deemed necessary to preserve the goal of fair benchmarking. SPEC will notify members and licensees if changes are made to the benchmark and will rename the metrics (e.g. from SPECbenchmarkYYYY to SPECbenchmarkYYYYa).
Relevant standards are cited in these run rules as URL references, and are current as of the date of publication. Changes or updates to these referenced documents or URL's may necessitate repairs to the links and/or amendment of the run rules. The most current run rules will be available at the SPEC web site at http://www.spec.org. SPEC will notify members and licensees whenever it makes changes to the suite.
2.0 Run Rules
The production of compliant SPECbenchmarkYYYY test results requires that the tests be run in accordance with these run rules. These rules relate to the requirements for the System Under Test (SUT) and the testbed (i.e. SUT, clients, and network), including applicable protocols or other standards, operation, configuration, test staging, optimizations and measurement.
2.1 Protocols (if applicable)
As application area is defined by its protocol definitions, SPECbenchmarkYYYY requires adherence to the relevant protocol standards:
The protocols listed above imply the following:
Internet standards are evolving standards. Adherence to related RFC's (e.g. RFC 1191 Path MTU Discovery) is also acceptable, provided the implementation retains the characteristic of interoperability with other implementations.
2.2 General Availability
The entire testbed (SUT, clients, and network) must be comprised of components that are generally available, or shall be generally available within three months of the first publication of these results.
Products are considered generally available if they are orderable by ordinary customers and ship within a reasonable time frame. This time frame is a function of the product size and classification and common practice. Some limited quantity of the product must have shipped on or before the close of the stated availability window. Shipped products do not have to match the tested configuration in terms of CPU count, memory size, and disk count or size, but the tested configuration must be available to ordinary customers. The availability of support and documentation of the products must be coincident with the release of the products.
Hardware products that are still supported by their original or primary vendor may be used if their original general availability date was within the last five years. The five-year limit is waived for hardware used in client systems.
Software products that are still supported by their original or primary vendor may be used if their original general availability date was within the last three years.
Information must be provided in the disclosure to identify any component that is no longer orderable by ordinary customers.
2.3 Stable Storage (if applicable)
The SUT must utilize stable storage for the application specific reason. Application area systems are expected to safely store any application object it has accepted until application disposition of object. To do this, Application area systems must be able to recover the application objects without loss from multiple power failures (including cascading power failures), operating system failures, and hardware failures of components (e.g. CPU) other than the storage medium itself (e.g. disk, non-volatile RAM). At any point where the data can be cached, after the server has accepted the message and acknowledged its receipt, there must be a mechanism to ensure any cached message survives the server failure.
- Examples of stable storage include:
- Media commit of data; i.e. the message has been successfully written to the disk media; for example, the disk platter.
- An immediate reply disk drive with battery-backed on-drive intermediate storage or uninterruptible power system (UPS).
- Server commit of data with battery-backed intermediate storage and recovery software.
- Cache commit with uninterruptible power system (UPS).
- Examples which are not considered stable storage
- An immediate reply disk drive without battery-backed on-drive intermediate storage or uninterruptible power system (UPS).
- Cache commit without uninterruptible power system (UPS).
- Server commit of data without battery-backed intermediate storage and recovery software.
If an uninterruptible power system (UPS) is required by the SUT to meet the stable storage requirement, the benchmarker is not required to perform the test with an UPS in place. The benchmarker must state in the disclosure that an uninterruptible power system (UPS) is required. Supplying a model number of an appropriate UPS is encouraged but not required.
If a battery-backed component is used to meet the stable storage requirement, that battery must have sufficient power to maintain the data for at least 48 hours to allow any cached data to be committed to media and the system to be gracefully shut down. The system or component must also be able to detect a low battery condition and prevent the use of the component or provide for a graceful system shutdown.
2.4 Single Logical Server (if applicable)
The SUT must present to application area clients the appearance and behavior of a single logical server for each protocol. Specifically, the SUT must present a single system view, in that the results of any application area transaction from a client that change the state on the SUT must be visible to any/all other clients on any subsequent application area transaction. For example, provide application specific example.
2.5 Application Logging (if applicable)
For a run to be valid, the following attributes related to logging must hold true:
- Provide statement on when a log entry is to be written; include minimum required fields for each log entry expected.
- Provide statement(s)on any additional requirements, such as storage type, update frequency, log duration, and format requirements
2.6 Networking (if applicable)
For a run to be valid, the following attributes that relate to TCP/IP network configuration must hold true (following list provided as examples):
- Since SPECbenchmarkYYYY is a representation of application area, the connections between a load
generating client and the SUT must not use a TCP Maximum Segment Size (MSS) greater than 1460 bytes. This needs to
be accomplished by platform-specific means outside the benchmark code itself. The method used to set the TCP MSS must be
disclosed. MSS is the largest "chunk" of data that TCP will send to the other end. The resulting IP datagram is
normally 40 bytes larger: 20 bytes for the TCP header and 20 bytes for the IP header resulting in an MTU (Maximum Transmission
Unit) of 1500 bytes.
- The value of TIME_WAIT must be at least 60 seconds.
- On those systems that do not dynamically allocate TCP TIME_WAIT table entries, the appropriate system parameter must be
configured to ensure that user ports are not reused before TIME_WAIT period expires. This would be set on a per client and per
server node basis as applicable. Include benchmark specific basis for this calculation.
Note: SPEC intends to follow relevant standards wherever practical, but with respect to this performance sensitive parameter it is difficult due to ambiguity in the standards. RFC1122 requires that TIME_WAIT be 2 times the maximum segment life (MSL) and RFC793 suggests a value of 2 minutes for MSL. So TIME_WAIT itself is effectively not limited by the standards. However, current TCP/IP implementations define a de facto lower limit for TIME_WAIT of 60 seconds, the value used in most BSD derived UNIX implementations.
2.7 Initializing and Running Benchmark
To make an official SPECbenchmarkYYYY test run, the benchmarker must perform the following steps:
- List specific step required to produce a conforming test run, such as initialization sequence, requires system
states, any required settings or options.
2.8 Optimization
Benchmark specific optimization is not allowed. Any optimization of either the configuration or software used on the SUT must improve performance for a larger class of workloads than that defined by this benchmark and must be supported and recommended by the provider. Optimizations that take advantage of the benchmarks specific features are forbidden. Examples of inappropriate optimization could include, but are not limited to, a list of possible benchmark specific optimizations.
2.9 Measurement
The provided SPECbenchmarkYYYY tools must be used to run and produce measured SPECbenchmarkYYYY results. The SPECbenchmarkYYYY metric is a function of the SPECbenchmarkYYYY workload, the associated benchmark specific working set, and the defined benchmark specific criteria. SPECbenchmarkYYYY results are not comparable to any other application area performance metric.
2.9.1 Metric
SPECbenchmarkYYYY expresses performance in terms of SPECbenchmarkYYYY metric name. Provide overview description of what the metric represents.
2.9.2 Workload
SPECbenchmarkYYYY <include additional overview on the workload and provide reference to complete workload description(s)>.
2.9.3 Application's Working Set (if applicable)
It is the responsibility of the benchmarker to ensure that the elements of the application's working set are placed on the SUT so that they can be accessed properly by the benchmark. These messages/files and only these messages/files shall be used as the target working set. The benchmark performs internal validations to verify the expected results. No modification or bypassing of this validation is allowed.
The benchmark determines the initial working set size for test based on a function. Provide details of this function. Include any additional requirements related to the working set for the benchmark and related benchmarker responsibilities.
2.9.4 Quality of Service Criteria (if applicable)
The SPECbenchmarkYYYY benchmark has specific Quality of Service (QoS) criteria for response times, delivery times and error rates. These criteria are checked for by the benchmark tools.
- Provide list of all QOS and error criteria for the benchmark
2.10 Load Generators (if applicable)
The SPECbenchmarkYYYY benchmark requires the use of one or more client systems to act as load generators. One client system is designated the prime client and this system will be the one on which the command that initiates the benchmark run. Provide brief description of client types used. Please refer to the User Guide for more detail on these roles.
A server component of the SUT may not be used as a load generator when testing to produce valid SPECbenchmarkYYYY results. A server component may be used as the prime client, but this is not recommended.
In order to run the benchmark tools, the client systems must include any requirements such as software versions or other configuration requirements.
2.11 SPECbenchmarkYYYY Parameters
The SPECbenchmarkYYYY benchmark provides number parameter files that contain the testbed configuration and workload parameters. The file benchmark-resource filename contains the testbed (clients and SUT) configuration information that appears in the final report and must be modified to contain your site-specific information.
The file benchmark-fixed-resource filename contains the default workload parameters used to produce a compliant test result this file must not be altered. Modifying the benchmark-fixed-resource filename will not prevent the benchmark from running, but the results generated using the modified benchmark-fixed-resource filename file will always be marked non-compliant.
To help ensure that the content of the parameter files are correct and can be used to produce a compliant test run, benchmarkers are encouraged to use <whatever benchmark specific actions or tools are provided>.
The SPECbenchmarkYYYY User's Guide provides detailed documentation on the parameters in the benchmark-resource filename and benchmark-fixed-resource filename files.
3.0 Reporting Rules
In order to publicly disclose SPECbenchmarkYYYY results, the benchmarker must adhere to these reporting rules in addition to having followed the run rules above. The goal of the reporting rules is to ensure the SUT and testbed are sufficiently documented such that someone could reproduce the test and its results.
3.1 Metrics And Result Reports
The benchmark single figure of merit, SPECbenchmarkYYYY messages per minute, is the throughput measured during the run at the benchmark specific load level. A complete benchmark result is comprised of benchmark specific description, shown on the results reporting page. A detailed breakdown of each test is included on the reporting page.
The report of results for the SPECbenchmarkYYYY benchmark is generated in HTML by the provided SPEC tools. These tools may not be changed, except for portability reasons with prior SPEC approval. The tools perform error checking and will flag some error conditions as resulting in an "invalid run". However, these automatic checks are only there for debugging convenience, and do not relieve the benchmarker of the responsibility to check the results and follow the run and reporting rules.
The section of the output.raw file that contains actual test measurement must not be altered. Corrections to the SUT descriptions may be made as needed to produce a properly documented disclosure.
3.2 Results Disclosure and Usage
Any SPECbenchmarkYYYY result produced in compliance with these run and reporting rules may be publicly disclosed and represented as valid SPECbenchmarkYYYY results.
Any test result not in full compliance with the run and reporting rules must not be represented using the SPECbenchmarkYYYY metric name.
The metric SPECbenchmarkYYYY must not be associated with any estimated results. This includes adding, multiplying or dividing measured results to create a derived metric.
3.2.1 Fair Use of SPECbenchmarkYYYY Results
Consistency and fairness are guiding principles for SPEC. To assure these principles are sustained, the following guidelines have been created with the intent that they serve as specific guidance for any organization (or individual) who chooses to make public comparisons using SPEC benchmark results.
When any organization or individual makes public claims using SPEC benchmark results, SPEC requires that the following guidelines be observed:
- Reference to the SPEC trademark. Such reference may be included in a notes section with other trademark references (see
www.specbench.org/spec/trademarks.html for all SPEC trademarks and service marks).
- The SPEC web site (www.spec.org) or a suitable sub page is noted as the source for more information.
- If competitive comparisons are made the following rules apply:
- the results compared must be compliant with that SPEC benchmark's run and reporting rules,
- the basis for comparison must be stated,
- the source of the competitive data must be stated,
- the date competitive data was retrieved must be stated,
- all data used in comparisons must be publicly available (from SPEC or elsewhere),
- the benchmark must be currently accepting new submissions if previously unpublished results are used in the
comparison.
- Comparisons with or between non-compliant test results can only be made within academic or research documents or
presentations where the deviations from the rules for any non-compliant results have been disclosed.
The following paragraph(s) is an example of acceptable language when publicly using SPEC benchmarks for competitive comparisons:
SPEC(tm) and the benchmark name SPECweb(tm) are registered trademarks of the Standard Performance Evaluation Corporation. Competitive benchmark results stated above reflect results published on www.spec.org as of Jan 12, 2001. The comparison presented above is based on the best performing 4-cpu servers currently shipping by Vendor 1, Vendor 2 and Vendor 3. For the latest SPECweb99 benchmark results visit www.spec.org <<or more specifically: www.spec.org/osg/web99>>.
3.2.2 Research and Academic usage of SPECbenchmarkYYYY
SPEC encourages use of the SPECbenchmarkYYYY benchmark in academic and research environments. It is understood that experiments in such environments may be conducted in a less formal fashion than that required of licensees submitting to the SPEC web site or otherwise disclosing valid SPECbenchmarkYYYY results.
For example, a research environment may use early prototype hardware that simply cannot be expected to stay up for the length of time required to run the required number of points, or may use research software that are unsupported and are not generally available. Nevertheless, SPEC encourages researchers to obey as many of the run rules as practical, even for