SPEC logo

SPECweb99 Frequently Asked Questions

  1. What is SPECweb99?
  2. What does SPECweb99 measure?
  3. What features have been added to SPECweb99?
  4. What kind of workload is used in the SPECweb99 benchmark?
  5. Does this benchmark replace the SPECweb96 suite?
  6. Can SPECweb99 results be meaningfully compared to SPECweb96 results?
  7. When and where will SPECweb99 results be available?
  8. What is the difference between SPECweb99 and other web server benchmarks?
  9. What are the limitations of SPECweb99?
  10. Can I use SPECweb99 to determine the size of the server I need?
  11. How can I obtain the benchmark?
  12. What is included with SPECweb99?
  13. What additional software is required to run SPECweb99?
  14. What hardware is required to run the benchmark?
  15. What if I have a problem building/running the SPECweb99 benchmarks?
  16. How can I submit SPECweb99 results?
  17. Where are the SPECweb99 run rules?
  18. Where can I go for more information?





1. What is SPECweb99?

SPECweb99 is a software benchmark product developed by the Standard Performance Evaluation Corporation (SPEC), a non-profit group of computer vendors, systems integrators, universities, research organizations, publishers and consultants. It is designed to measure a system's ability to act as a web server for static and dynamic pages.

SPECweb99 is the successor to SPECweb96, and continues the tradition of giving Web users the most objective, most representative benchmark for measuring web server performance. SPECweb99 disclosures are governed by an extensive set of run rules to ensure fairness of results.

The benchmark runs a multi-threaded HTTP load generator on a number of driving "client" systems that will do static and dynamic GETs of a variety of pages from, and also do POSTs to, the SUT (System Under Test).

SPECweb99 provides the source code for an HTTP 1.0/1.1 load generator that will make random selections from a predetermined distribution. The benchmark defines a particular set of files to be used as the static files that will be obtained by GETs from the server, thus defining a particular benchmark workload.

The benchmark does not provide any of the web server software. That is left up to the tester. Any web server software that supports HTTP 1.0 and/or HTTP 1.1 can be used. However, it should be noted that variations in implementations may lead to differences in observed performance.

To make a run of the benchmark, the tester must first set up one or more networks connecting a number of the driving "clients" to the server under test. The benchmark code is distributed to each of the drivers and the necessary fileset is created for the server. Then a test control file is configured for the specific test conditions and the benchmark is invoked with that control file.

SPECweb99 is a generalized test, but it does make a good effort at stressing the most basic functions of a web server in a manner that has been standardized so that cross-comparisons are meaningful across similar test configurations.
 

1.1 What is SPECweb99 Release 1.01

This is a minor release which fixes several issues found in the client test harness after the release. There are code changes to the module HTTP/HT.c and manager only. These changes include:

Minor documentation edits have been made to clarify the benchmark's operation in light of the changes shown above as well as to make a few corrective edits that were missed in the initial document review. User's should review these modified sections:

SPEC requires that all SPECweb99 published after Nov 1, 1999 use the new Release 1.01. All licensees of SPECweb99 will be provided with an update kit which will include the source for HTTP/HT.c and all updated documentation.


 

1.2 What is SPECweb99 Release 1.02

SPECweb99 Release 1.02 is a minor release which fixes several issues found in the client test harness after release 1.01. There are code changes to the files abstract.c, configure.in, random.c, manager, and client.c. These changes include:

Documentation Changes:

SPEC requires that all SPECweb99 results submitted after September 18, 2000 use the new Release 1.02. All licensees of SPECweb99 will be provided with an update kit which will include all updated source and documentation.
 

2. What does SPECweb99 measure?

SPECweb99 measures the maximum number of simultaneous connections, requestingthe predefined benchmark workload, that a web server is able to support while still meeting specific throughput and error rate requirements. The connections are made and sustained at a specified maximum bit rate with a maximum segment size intended to more realistically model conditions that will be seen on the Internet during the lifetime of this benchmark.
 
 

3. What features have been added to SPECweb99?

SPECweb99 includes many major enhancements:

4. What kind of workload is used in the SPECweb99 benchmark?

The SPECweb99 workload simulates the accesses to a web service provider, where the server supports the home page for a number of different organizations. Each home page is a collection of files ranging in size from small icons to large documents or images. As in the real world, certain files within the home page are more popular than others. The dynamic GETs simulate the common practice of "rotating" advertisements on a web page. The POSTs simulate entry of user data into a log file on the server, such as might happen during a user registration sequence.

In this release, SPECweb99 has added dynamic services for which real-world data was available. Dynamic services are still developing too rapidly at this point to allow all of the latest features to be included. Dynamic content, consisting of 16% POSTs, 41.5% GETs, and 42% GETs with cookies, and 0.5% CGI GETs, comprises 30% of the overall workload.

The workload is based on analysis of server logs from a variety of popular Internet servers and some smaller Web sites. The workload defines four classes of files which are accessed. These have the following file sizes: less than 1 KB, 1 to 10 KB, 10 to 100 KB, and 100 KB to 1 MB. There are nine files in each class, with sizes distributed evenly through the range for that class.

The access patterns to the files were determined from the analysis of the web server logs. SPECweb99 directs 35% of its activity to the smallest class, 50% to the 1-to-10-KB class, 14% percent to the 10-to-100-KB class, and 1% to the largest files. Within each class there are non-linear distributions of accesses, reflecting the fact that certain files are more popular than others.

Recent studies have shown that file accesses on a server follow a Zipf distribution (see, for example, http://www.useit.com/alertbox/zipf.html). Thus, SPECweb99 directory and within-class accesses are generated using the Zipf distribution function. The resulting overall distribution is very close to actual measured distributions on real servers.
 
 

5. Does this benchmark replace the SPECweb96 suite?

Yes. SPEC is providing a six month transition period from the date of the SPECweb99 announcement (June 14, 1999). During this period, SPEC will accept, review and publish results from both benchmark versions. After this period, results from SPECweb96 will no longer be accepted by SPEC for publication.
 
 

6. Can SPECweb99 results be meaningfully compared to SPECweb96 results?

No. Although the benchmarks are similar, they cannot be compared, since SPECweb99 uses an entirely different metric than SPECweb96. SPECweb99 also has different file access distributions and a mix of different types of server queries. The dynamic part of the SPECweb99 workload has no SPECweb96 equivalent, so there is no way to make meaningful comparisons between the two.

7. When and where will SPECweb99 results be available?

Initial SPECweb99 results are available on SPEC's Web site: http://www.spec.org/web99/. Subsequent results will be posted on an ongoing basis following each two-week review cycle: results submitted by the semi-weekly deadline are reviewed by web committee members for conformance to the run rules, and if approved at the end of that period are then publicly released.

8. What is the difference between SPECweb99 and other web server benchmarks?

SPECweb99 is a standardized test. The SPEC membership - leading vendors, systems integrators, universities, research organizations, publishers and consultants - has agreed on a single benchmark with one standardized implementation and workload.

Before any SPECweb99 results are published, SPEC requires that the system under test and the methodology used adhere to agreed-upon standards (the run rules). All results available through SPEC include full disclosure information that reveals exactly what configurations have been used to obtain a particular result.

SPECweb99 results published on SPEC's web site provide standardized, comparable results for those who cannot run their own web server performance tests.

9. What are the limitations of SPECweb99?

SPECweb99 is a standardized benchmark, which means that it is an abstraction of the real world. For example, SPECweb99 does not attempt to model latency associated with obtaining data across a wide-area network (WAN) such as the Internet. This kind of behavior is difficult to simulate at this stage, since it requires elaborate hardware and software.

10. Can I use SPECweb99 to determine the size of the server I need?

SPECweb99 is not the right tool for sizing a server. SPECweb99 was not designed as a capacity planning tool. However, it does provide information on how web servers handle this specific workload. The workload uncovers several key components of a good web server, including LAN performance, processing power, and memory bandwidth, to name a few.

11. How can I obtain the benchmark?

SPECweb99 is scheduled for public release in June 1999. It will be available on CD-ROM for $800. A discount is available for universities and other non-profits as well as for SPECweb96 licensees. To order, contact SPEC's administrative office.

12. What is included with SPECweb99?

The benchmark comes with the code necessary to build and run the driver system(s), as well as sample API modules for the dynamic content and the code for building the dataset that will be requested from the web server. It is up to the tester to provide the web server and to install the dataset created by the SPEC tools.

SPECweb99 does not provide web server software, but any web server that is HTTP/1.0 or 1.1 compliant can be used. In addition to licensed server software, public domain server software can be used to run SPECweb99.

The SPECweb99 CD-ROM contains:

13. What additional software is required to run SPECweb99?

First, of course, you'll need properly running web server software on your server. On at least one client system, you'll need a full ANSI C programming environment to build the benchmark from the provided source code. Binaries will be provided for some system architectures.

14. What hardware is required to run the benchmark?

Web Server: computer(s) with enough stable storage for fileset and web server logs

Client(s): computer(s) with enough stable storage for client software

Network: any TCPIP compatible network connecting the web server and clients

15. What if I have a problem building/running the SPECweb99 benchmark?

There may have been some issues that have been raised about the benchmark since it was released. We are keeping a SPECweb99 issues repository.

If your issue is not amongst the known issues, then bring it to the attention of SPEC.

16. How can I submit SPECweb99 results?

Only SPECweb99 licensees can submit results. SPEC member companies submit results free of charge. Non-members may submit results for an additional fee. All results are subject to a two-week review by web committee members. Non-member submissions are also subject to a preliminary review. If they pass preliminary review, they may be submitted for the standard member review, and if approved will be published by SPEC upon payment of a fee. First-time submitters should contact SPEC's administrative office.

SPECweb99 submissions must include both the raw output file and the source code for the dynamic implementation used. Submitters may also be required to submit the last 20% of the web server log file upon request.

17. What rules govern SPECweb99?

The current version of the run rules can be found at http://www.spec.org/web99/docs/runrules.html.

18. Where can I go for more information?

The SPECweb99 white paper contains detailed design information on the benchmark and workload. as well as instructions for installing and running the benchmark.  See: http://www.spec.org/web99 for the white paper and other available information on SPECweb99.




Revised: Fri Aug 11 10:40:02 EDT 2000