SPECvirt_sc2013 Frequently Asked Questions

Version 1.1 - September 21, 2016

1. What is SPECvirt_sc2013 V1.1?

SPECvirt_sc2013 V1.1 is a software benchmark product developed by the Standard Performance Evaluation Corporation (SPEC), a non-profit group of computer vendors, system integrators, universities, research organizations, application software stacks. The benchmark is intended to be run by hardware vendors, virtualization software vendors, application software vendors, datacenter managers, and academic researchers.

2. Why did SPEC develop SPECvirt_sc2013 V1.1?

The release of SPECvirt_sc2013 V1.1 enhances security protocol support to the webserver workload, addresses minor defects in the reporter and syntax checker, and updates the Power and Temperature Daemon (PTD) to the latest version.

3. How does SPECvirt_sc2013 V1.1 compare to SPECvirt_sc2013 V1.0?

SPECvirt_sc2013 V1.1 is an update to SPECvirt sc2013 V1.0 and shares its benchmark architecture, workload implementation, harness, and run requirements. For application stacks that require Transport Layer Security (TLS), it supports TLSv1, TLSv1.1, and TLSv1.2 and adds newer ciphers. Existing support of SSLv3 remains available and is the default.

4. What does SPECvirt_sc2013 measure?

The benchmark presents an overall workload that achieves the maximum performance of the platform when running a set of four application workloads against one or more sets of Virtual Machines called "tiles". Scaling the workload on the SUT (System Under Test) consists of running an increasing number of tiles. Peak performance is the point at which the addition of another tile either fails the Quality of Service criteria or fails to improve the overall metric.

The benchmarker has the option of running with power monitoring enabled and can submit results to either the performance with SUT power category and/or performance with Server only power category.

5. What kinds of workloads are used in the SPECvirt_sc2013 benchmark?

The suite consists of several SPEC workloads that represent applications that industry surveys report to be common targets of virtualization and server consolidation. We modified each of these standard workloads to match an enterprise server consolidation scenario's resource requirements for CPU, memory, disk I/O, and network utilization for each workload. The SPEC workloads used are modified versions of SPECweb2005, SPECjAppServer2004, SPECmail2008, and SPEC CPU2006.

6. When and where are SPECvirt_sc2013 results available?

Initial SPECvirt_sc2013 results are available on SPEC's web site. Subsequent results are posted on an ongoing basis following each two-week review cycle: results submitted by the two-week deadline are reviewed by SPECvirt committee members for conformance to the run rules, and if accepted at the end of that period are then publicly released. Results disclosures are at: http://www.spec.org/virt_sc2013/results.

7. What are the limitations of SPECvirt_sc2013?

SPECvirt_sc2013 is a standardized benchmark, which means that it is an abstraction of the real world. For example all the database servers can use the same database archive to restore their copy of the database. This helps reduce the complexity of setting up the test.

8. Can I use SPECvirt_sc2013 to determine the size of the server I need?

SPECvirt_sc2013 results are not intended for use in sizing or capacity planning.

9. What is a tile?

In SPECvirt_sc2013, a tile is a single unit of work that is comprised of four application workloads that are driven across five distinct virtual machines plus a separate Database Server VM. The load on the SUT is scaled up by configuring additional sets of the VMs as described below and increasing the tile count for the benchmark.  For a SPECvirt_sc2013 tile, the workloads and their VMs are:

The Application Server VM for each tile requires an enterprise-class Database Server VM backend. Each Database Server VM is shared by up to four appserver VMs. For every four consecutive tiles, a separate Database Server VM is required. Only the last Database Server VM may be shared by fewer than four tiles if the number of tiles is not a multiple of four.

10. What is a fractional tile?

When the SUT does not have sufficient system resources to support the full load of an additional tile, the benchmark offers the use of a fractional load tile. A fractional tile consists of an entire tile with all VMs but running at a reduced percentage of its full load.

11. How can I obtain the benchmark?

SPECvirt_sc2013 is available via web download from the SPEC site at $3000 for new licensees and $1500 for academic and eligible non-profit organizations. The order form is at: http://www.spec.org/order.html.

12. What is included with SPECvirt_sc2013?

The benchmark includes the code necessary to run the driver system(s), the server-side file set generation tools, and dynamic content implementations. It is at the tester's discretion to choose the application stack.

The SPECvirt_sc2013 kit contains:

13. What hardware is required to run the benchmark?

See the Run Rules and the User's Guide for more detailed information and requirements.

14. What if I have a problem configuring or running the SPECvirt_sc2013 benchmark?

You can find more information on how to set up and run the benchmark in the User's Guide and the Client Harness User's Guide. You also can register at the SPECvirt Forum where you can post questions and review solutions to common problems. If you cannot resolve your issue using these methods, please send email to bring it to the attention of the SPEC Virtualization subcommittee.

At the Forum you also can find an ExampleVM environment for the SPECvirt_sc2013 V1.1 benchmark. This ExampleVM environment includes documentation and scripts to help configure the six virtual machines needed in a SPECvirt_sc2013 tile as well as a client virtual machine. Even if the configuration is not exactly the same as one your are trying to set up, having a working example for comparison is a valuable tool to aid you in setting up your own environment. See this topic for more information.

15. How can I submit SPECvirt_sc2013 results?

Only SPECvirt_sc2013 licensees can submit results. SPEC member companies submit results free of charge, and non-members may submit results for an additional fee. All results are subject to a two-week review by SPECvirt subcommittee members. First-time submitters should contact SPEC's administrative office.

SPECvirt_sc2013 submissions must include both the raw output file and configuration information required by the benchmark. During the review process, other information may be requested by the subcommittee. You can find submission requirements in the run rules.

16. Where are the SPECvirt_sc2013 run rules?

The current version of the run rules can be found at: http://www.spec.org/virt_sc2013/docs/SPECvirt_RunRules.html.

17. Where can I go for more information?

The SPECvirt_sc2013 Design Document contains design information on the benchmark and workloads. The Run and Reporting Rules, the User's Guide, and the Client Harness User's Guide contain instructions for installing and running the benchmark. See http://www.spec.org/osg/virtualization for the available information on SPECvirt_sc2013.

18. What control mechanism is used to drive the workloads?

SPEC developed a test harness driver to coordinate running the component workloads in one or more tiles on the SUT. The harness allows you to run and monitor the benchmark, collects measurement data as the test runs, post-processes the data at the end of the run, validates the results, and generates the test report.

19. What is the performance metric for SPECvirt_sc2013?

The benchmark supports three categories of results, each with its own primary metric. Results may be compared only within a given category; however, the benchmarker has the option of submitting results from a given test to one or more categories. The first category is Performance-Only and its metric is SPECvirt_sc2013 which is expressed as "SPECvirt_sc2013 @ <5*Number_of_Tiles + Number_of_DBservers> VMs" on the reporting page. SPECvirt_sc2013_PPW (performance with SUT power) and SPECvirt_sc2013_ServerPPW (performance with Server only power) are performance per watt metrics obtained by dividing the peak performance by the peak power of the SUT or Server, respectively, during the run measurement phase.

20. Does the benchmark support multiple servers?

No. Currently the benchmark is designed for a single host system.

21. Can I report results for open source software?

Yes, you can use open source products when running the benchmark as long as you comply with open source requirements specified in the Run Rules.

22. Are the results independently audited?

The SPEC Virtualization subcommittee reviews all results but does not require that they be independently audited.

23. Can I announce my results before they are reviewed by the SPEC subcommittee?

No. SPEC must review and accept the result before it can be announced publicly.

24. Are results sensitive to components outside of the SUT -- e.g. client driver machines?

Yes, the client driver machines must be configured properly to accommodate the workloads. You may use one or more physical systems for client load drivers, clients may be virtualized, and the tile may be driven by multiple clients. Note that client resource requirements for SPECvirt_sc2013 are higher than for SPECvirt_sc2010. See the User's Guide for more information regarding hardware and software requirements for the clients.

25. Does SPECvirt_sc2013 have a power measurement component associated with the benchmark?

SPECvirt_sc2013 implements the SPECpower methodology for power measurement. The benchmarker has the option of running with power monitoring enabled and can submit results to any of three categories: * performance only (SPECvirt_sc2013) * performance/power for the SUT (SPECvirt_sc2013_PPW) * performance/power for the Server-only (SPECvirt_sc2013_ServerPPW)

You can find more information on power measurement in the Client Harness User's Guide and Run Rules.

26. Can I compare the results of SPECvirt_sc2013 workloads to the results of the SPEC benchmarks from which they were derived? For example, can I compare a SPECweb2005 result to the result of the SPECvirt_sc2013 web server component?

No, they are not. Several substantive changes have been made that make the SPECvirt_sc2013 workloads unique.

27. Can I compare SPECvirt_sc2013 results in different categories?

No. Results between the different SPECvirt_sc2013 categories cannot be compared.

28. Can I compare SPECvirt_sc2013 with other virtualization benchmarks?

No. SPECvirt_sc2013 is unique and not comparable to other benchmarks.

29. What is a "compliant" result of SPECvirt_sc2013?

A compliant benchmark result meets all the requirements of the SPECvirt_sc2013 run rules for a valid result. In addition to the run and reporting rules, several validation and tolerance checks are built-in to the benchmark. If you intend to publicly use the SPECvirt_sc2013 metrics, the result must be compliant and accepted by SPEC.

30. Can I run other workload levels?

Yes, for non-compliant runs only. You may set the load level for each or all workloads to be heavier or lighter as your needs dictate. You can set these load levels by changing parameters in the Control.config file and possibly each workload's configuration file.

31. How long does it take to run the benchmark?

The run time is approximately three hours with default settings.

32. How can I use the benchmark to research performance related to a specific component of the benchmark such as the memory, storage, hypervisor, or the application server VM?

SPECvirt has been implemented as a standardized end-to-end benchmark designed to stress all layers of a system that handles a workload representative of server consolidation. Performance critical components include the server hardware (Processors, Memory, Network, Storage, etc.), the virtualization technology (hardware virtualization, operating system virtualization, and hardware partitioning), the guest (VM) operating systems, and the guest application software stacks. Selection and tuning of any of these components can have significant effects on the overall performance of the SUT.

The best way to differentiate the performance characteristics of different versions or products for a specific element of a system is to hold all other elements constant and change only component you are interested in. For example if you want to see the effects of RAID 5 vs RAID 10, then keep the other elements of the server, virtualization products, and the guest VMs the same and install copies of the VMs on the RAID 5 storage and RAID 10 storage while keeping other storage elements such as number of LUNs the same and run your tests. Similarly if you want to compare versions of hypervisors, then you need to keep the rest of the platform constant. If you change other elements such as the software running on the VMs, it can significantly impact the overall results.

33. What types of virtualization platforms are supported by SPECvirt_sc2013?

SPECvirt_sc2013 supports hardware virtualization, operating system virtualization, and hardware partitioning. The benchmark does not address multiple host performance or application virtualization.

34. What skills do I need to run SPECvirt_sc2013?

The documentation assumes that you have familiarity with virtualization concepts and implementations. You require experience with the installation, configuration, management, and tuning of your selected hypervisor platform. You must know how to use your virtualization platform to create, administer, and modify virtual machines and allocate system resources including:

Product and service names mentioned herein may be the trademarks of their respective owners.

Copyright © 2013-2016 Standard Performance Evaluation Corporation (SPEC).

All rights reserved.