Performance Summary

Validation, Errors, and Issues

• Stopping condition reason: Benchmark stopped because number of AIs reporting results reached the configured value. Result received from AIs:10, _rules_reported_ais:10, _ais_issued:12:NOK
• 0 total validation errors found
• 0 YCSB validation errors found
• 0 KMeans validation errors found
• 0 YCSB AIs removed due to 60/40 rule
• 0 KMeans AIs removed due to 60/40 rule
• 0 YCSB AIs with zero runs
• 0 KMeans AIs with zero runs
• 0 YCSB AIs that failed to provision
• 0 KMeans AIs that failed to provision

Glossary of "Run code": 0 no error, 3 if an AI failed to provision, 2 if an AI that was provisioned but did not report any runs when the benchmark terminated - the AIs that are provisioning towards the end of elasticity + scalability phase may have zero runs, 1 for all other errors including AIs that were provisioned but did not report any result or reported results after QoS limits.

AIs that were issued but their provisioning was inprogress when the benchmark terminated are not shown.

Instance Configuration Hadoop Master
Parameter	Value
instance_type_name	m1.cmedium3
cpu	2
memory	12GB
disk_size	40GB
disk_backed	block storage
image_name	cb_speccloud_kmeans_hadoop_271_centos-raw
Notes	Hadoop 2.7.1 / Centos 7.1

Instance Configuration Hadoop Slave
Parameter	Value
instance_type_name	m1.clarge3
cpu	4
memory	20GB
disk_size	80GB
disk_backed	block storage
image_name	cb_speccloud_kmeans_hadoop_271_centos-raw
Notes	Hadoop 2.7.1 / Centos 7.1

Instance Configuration YCSB
Parameter	Value
instance_type_name	m1.cmedium3
cpu	2
memory	12GB
disk_size	40GB
disk_backed	block storage
image_name	cb_speccloud_cassandra_2111_centos-raw
Notes	Cassandra 2.11.1 / Centos 7.1

Instance Configuration Cassandra
Parameter	Value
instance_type_name	m1.cmedium3
cpu	2
memory	12GB
disk_size	40GB
disk_backed	block storage
image_name	cb_speccloud_cassandra_2111_centos-raw
Notes	Cassandra 2.11.1 / Centos 7.1

Cloud High Level Information
Parameter	Value
Name	Openstack
Type	private
Sut_Type	whitebox
Virtualized	KVM
Manager_Protocol	Red Hat Enterprise Linux
Manager_Version	Openstack Platform 7
Hardware_Availability	May-2012
Software/Service_Availability	Aug-2015
Geographic_Distribution	1
Notes	Cloud infrastructure and management nodes all reside in a single datacenter located in Austin, TX.

Cloud Under Test Software Levels
Parameter	Value
Python_version	2.7.6
JVM_version	1.7.0_45
Hadoop_version	2.7.1
YCSB_version	0.4.0
HiBench_version	2.2
Cassandra_version	2.1.11
Notes	N/A

Benchmark Harness Node
Parameter	Value
Vendor	Dell
Platform	PowerEdge R720xd
CPU_Description	Intel Xeon, E5-2640 0
CPU_Frequency_MHz	2500
RAM_Memory	48 GB, ECC, DDR3, 1600MHz RDIMM
Local_Storage	8x 1TB, SATA, RAID50, 7200 RPM
Quantity	1
Availability_Date	May-2012
OS_Version	RHEL 7.1
OS_Availability_Date	March-2015
Network_Adapter_1	1 x 1 x Broadcom Corporation NetXtreme BCM5720 4P
Network_Adapter_2	2 x Intel X520 10Gb 2P SFP+
Network_port_type_1	1000BASE-T
Network_port_type_2	10GBASE-SR
Network_speed_port_1	1 Gb/s
Network_speed_port_2	10 Gb/s
Notes	Solution Admin Host (SAH) hosts the benchmark driver harness

Cloud Manager Node
Parameter	Value
Hw_Vendor_node1,3	Dell
Hw_Platform_node1,3	PowerEdge R720xd
CPU_Type_node1,3	Intel Xeon, E5-2640 0
CPU_Count_node1,3	4
CPU_Cores_Per_node1,3	6
CPU_Frequency_MHz_node1,3	2500
RAM_Memory_GB_node1,3	96,48
Primary_Cache_KB_node1,3	190 KB I + 1500 KB D per core, on chip
Secondary_Cache_KB_node1,3	1500 KB I+D per core, on chip
RAM_Memory_Type_node1,3	ECC DDR3 1333 MHz
Hw_Availability_Date_node1,3	May-2012
Local_Storage_node1,3	24x1 TB HDD, RAID 50, 7200 RPM
Quantity_node1,3	2
Virtualized_node1,3	baremetal
Network_Adapter_node1,3_1	1 x Broadcom Corporation NetXtreme BCM5720 4P
Network_Adapter_node1,3_2	2 x Intel X520 10Gb 2P SFP+
Network_port_type_node1,3_1	1000BASE-T
Network_port_type_node1,3_2	10GBASE-SR
Network_speed_port_node1,3_1	1 Gb/s
Network_speed_port_node1,3_2	10 Gb/s
OS_Version_node1,3	RHEL 7.1
OS_Availability_Date_node1,3	March-2015
Hw_Vendor_node2	Dell
Hw_Platform_node2	PowerEdge R720xd
CPU_Type_node2	Intel Xeon, E5-2670 v2
CPU_Count_node2	2
CPU_Cores_Per_node2	10
CPU_Frequency_MHz_node2	2500
RAM_Memory_GB_node2	96
Primary_Cache_KB_node2	310 KB I + 2500 KB D per core, on chip
Secondary_Cache_KB_node2	2500 KB I+D per core, on chip
RAM_Memory_Type_node2	ECC DDR3 1600 MHz
Hw_Availability_Date_node2	May-2012
Local_Storage_node2	24x1 TB HDD, RAID 50, 7200 RPM
Quantity_node2	1
Virtualized_node2	baremetal
Network_Adapter_node2_1	1 x Broadcom Corporation NetXtreme BCM5720 4P
Network_Adapter_node2_2	2 x Intel X520 10Gb 2P SFP+
Network_port_type_node2_1	1000BASE-T
Network_port_type_node2_2	10GBASE-SR
Network_speed_port_node2_1	1 Gb/s
Network_speed_port_node2_2	10 Gb/s
OS_Version_node2	RHEL 7.1
OS_Availability_Date_node2	March-2015
Notes	3x Cloud Manager (Openstack controller) nodes

Cloud Computes Nodes
Parameter	Value
type 1_Compute_nodes	5
type 1_Hw_Vendor	Dell., Inc.
type 1_Hw_Platform	PowerEdge R720xd
type 1_Quantity	5
type 1_CPU_Type	Intel Xeon, E5-2640 0
type 1_CPU_Count	10
type 1_CPU_Cores_Per	6
type 1_CPU_Frequency_MHz	2500
type 1_RAM_Memory_GB	192
type 1_Primary_Cache_KB	190 KB I + 1500 KB D per core, on chip
type 1_Secondary_Cache_KB	15000 KB I+D per core, on chip
type 1_RAM_Memory_Type	ECC DDR3 1333 MHz
type 1_Hw_Availability_Date	May-2012
type 1_Local_Storage	24x 1 TB HDD, SATA, RAID 50, 7200 RPM
type 1_Virtualized	baremetal
type 1_Network_Adapter_1	1 x Broadcom Corporation NetXtreme BCM5720 4P
type 1_Network_Adapter_2	2 x Intel X520 10Gb 2P SFP+
type 1_Network_port_type_1	1000BASE-T
type 1_Network_port_type_2	10GBASE-SR
type 1_Network_speed_port_1	1 Gb/s
type 1_Network_speed_port_2	10 Gb/s
type 1_OS_Version	RHEL 7.1
type 1_OS_Availability_Date	March-2015
type 1_Notes	N/A
type 2_Compute_nodes	2
type 2_Hw_Vendor	Dell., Inc.
type 2_Hw_Platform	PowerEdge R720xd
type 2_CPU_Type	Intel Xeon, E5-2670 v2
type 2_CPU_Count	4
type 2_CPU_Cores_Per	8
type 2_CPU_Frequency_MHz	2500
type 2_RAM_Memory_GB	192
type 2_Primary_Cache_KB	310 KB I + 2500 KB D per core, on chip
type 2_Secondary_Cache_KB	25000 KB I+D per core, on chip
type 2_RAM_Memory_Type	ECC DDR3 1600 MHz
type 2_Hw_Availability_Date	May-2012
type 2_Local_Storage	12x1 TB, RAID 0, 7200 RPM
type 2_Quantity	2
type 2_Virtualized	baremetal
type 2_Network_port_1	1 Gbps
type 2_Network_port_2	10 Gbps
type 2_OS_Version	RHEL 7.1
type 2_OS_Availability_Date	Mar-2015

Cloud Schematic

Application Instance

A group of instances created to run a single workload together. An application instance comprises a workload driver instance and set of instances, which are stressed by the workload driver. SPEC Cloud IaaS 2016 benchmark uses multiple application instances during specific phases to determine elasticity and scaling.

Baseline phase

In baseline, peak performance for each workload is determined in five separate test runs. During each workload run, instances are provisioned, data set is generated, load generator is started, results are accumulated, and the instances are destroyed. The workloads are run in a sequential fashion. Data from the baseline phase is used to establish parameters for the Elasticity + Scalability phase.

Benchmark phases

Benchmark has two phases, namely baseline and elasticity + scalability.

Blackbox Cloud

A cloud-provider provides a general specification of the SUT, usually in terms of how the cloud consumer may be billed. The exact hardware details corresponding to these compute units may not be known. This will typically be the case if the entity benchmarking the cloud is different from a cloud provider.

Cloud (system or service)

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

Cloud Consumer

A person or organization that is a customer of a cloud; note that a cloud customer may itself be a cloud and that clouds may offer services to one another.

Cloud Provider

An organization that provides cloud services to customers who pay for only the computing time and services actually used.

Elasticity

Elasticity measures whether the work performed by application instances scales linearly in a cloud. That is, for statistically similar work, the performance of N application instances in a cloud must be the same as the performance of application instances during baseline phase when no other load is introduced by the tester. Elasticity is expressed as a percentage (out of 100).

Elasticity + scalability phase

In the Elasticity + Scalability phase, new application instances are created, and they run the workloads concurrently to determine the elasticity and scalability metrics. The benchmark reports are generated at the end of elasticity + scalability phase.

Infrastructure as a Service (IaaS)

The Service Provider gives the Cloud-Consumer the capability to the provision processing, storage, network, and basic computing resources. They can also deploy and run arbitrary operating systems. The End-Consumer does not manage or control the underlying physical cloud infrastructure, but has control over the operating system, assigned storage, deployed applications, and limited control of select networking components (e.g., host firewalls).

Instance

An instance is an abstracted execution environment which presents an operating system (either discrete or virtualized). The abstracted execution environment presents the appearance of a dedicated computer with CPU, memory and I/O resources available to the operating system. In SPEC Cloud, an instance consists of a single OS and the application software stack that supports a single SPEC Cloud component workload. There are several methods of implementing an instance, including physical machines, virtual machines, or containers. An instance is created or destroyed using an API provided by an IaaS cloud.

Instance image

An image on the disk from which an instance is provisioned. Common formats for instance image include QCOW2 (Qemu copy on write 2), RAW, or AMI (Amazon machine image)

Physical machine

A set of connected components consisting on one or more general purpose processors (CPUs), memory, network connectivity, and mass storage either local (disk) or remote (network attached, block storage). The physical machine can have its own physical packaging or be installed as a blade in a blade chassis. An example would be a multi-core server with 4 GB of memory, 250 GB disk, and 1 Gb/s network adapter.

Provisioning Time

The measured time needed to bring up a new instance, or add more resources (like CPU or storage) to an existing instance. [Reference: CloudWhitePaper]

Within this document, this will be divided into two measurements:

1) Instance: The time from request to create a new instance until that instance responds to a netcat probe on port 22.

2) Application instance: The time from request to create a new instance until the associated cluster reports readiness to accept client requests. YCSB (Yahoo! Cloud Serving Benchmark) is ready when all nodes in the underlying database are part of the Cassandra cluster. K-Means is ready when the all nodes in HDFS are ready and part of the HDFS cluster.

Private Cloud

The cloud infrastructure is provisioned for exclusive use by a single organization comprising single or multiple consumers (e.g., business units). It may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off premises.

Public Cloud

The cloud infrastructure is provisioned for open use by the general public. It may be owned, managed, and operated by a business, academic, or government organization, or some combination of them. It exists on the premises of the cloud provider.

Quality of Service (QoS)

The minimum percent (e.g., 95%) of collected values that complete within a predefined threshold in each measurement category. The Run and Reporting Rules document contains specific QoS limitations for each workload and benchmark.

Response Time

The time between when the work item request is issued until the corresponding completion condition. This definition is identical to the YCSB Latency metric.

Scalability

Scalability measures the total amount of work performed by application instances running in a cloud. The aggregate work performed by one or more application instances should linearly scale in an ideal cloud. It answers the question: How much more gets done if N instead of one (1) application instances? Ideally, each additional application instance will contribute 100% of its individual capability, but this may not be the case as the cloud gets loaded with multiple application instances or other workloads (for public clouds). Scalability is reported as a unit-less number @ the number of valid application instances. The number is an aggregate of workloads metrics across all application instances running in a cloud normalized by workload metrics in a reference cloud platform. The reference platform metrics are an average of workload metrics taken across multiple cloud platforms during the benchmark development.

SUT

One or more cloud services under test. This includes all hardware, network, base software and management systems used for the cloud service.

Variability

The difference between different runs of the benchmark. Typically, variability arises due to factors such as multi-tenancy and time of day execution, and may be more pronounced on public clouds.

Whitebox Cloud

The SUT's exact engineering specifications including all hardware and software are known and under the control of the tester. This will typically be the case for private clouds.