SlideShare a Scribd company logo
SPEC Cloud™ IaaS 2016 Benchmark
Salman Baset, IBM - @salman_baset
https://www.spec.org/benchmarks.html#cloud
Topics
• Standard Performance Corporation (SPEC)
• What are cloud characteristics?
• What is a cloud benchmark?
• SPEC Cloud IaaS 2016 benchmark, the first industry standard
benchmark
– Workloads
– Setup
– Metrics
– Compliant run
– Stopping conditions
– Published results
• What’s next in SPEC Cloud* benchmarks?
2
SPEC Background
• Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation
formed to establish, maintain and endorse a standardized set of relevant
benchmarks that can be applied to the newest generation of high-performance
computers.
– http://www.spec.org/
• Well-known benchmarks such as SPEC CPU 2006, SPEC Virt, SPEC Power…
• Cloud Subcommittee released the first industry standard benchmark (SPEC Cloud
IaaS 2016) for infrastructure-as-a-service cloud in May 2016.
– http://spec.org/benchmarks.html#cloud
3
SPEC Cloud IaaS Benchmark and OpenStack
• Measuring cloud performance in general and OpenStack
performance in particular is hard.
– Too many configuration options
– Too many metrics
– Difficult to repeat experiments and compare results
• SPEC Cloud IaaS 2016 can be used to measure and publish
performance (and related configuration) of each OpenStack
release.
– Each release can be measured by the benchmark
– Such results can potentially be published before an OpenStack summit
or during key note, giving credence to the release.
– All results are peer-reviewed before publishing.
4
What is a cloud?
• Public cloud
• Private cloud
• Hybrid cloud
5
What is a cloud?
• SPEC OSG Cloud Subcommittee Glossary
• Blackbox Cloud
• A cloud-provider provides a general specification of the system
under test (SUT), usually in terms of how the cloud consumer
may be billed. The exact hardware details corresponding to these
compute units may not be known. This will typically be the case
if the entity benchmarking the cloud is different from a cloud
provider, e.g., public clouds or hosted private clouds.
• Whitebox Cloud
• The SUT’s exact engineering specifications including all hardware
and software are known and under the control of the tester. This
will typically be the case for private clouds.
Source: https://www.spec.org/cloud_iaas2016/docs/faq/html/glossary.html
6
What are cloud characteristics?
7
SPEC RESEARCH GROUP - CLOUD WORKING GROUP
https://research.spec.org/working-groups/rg-cloud-working-
group.html
READY FOR RAIN? A VIEW FROM SPEC RESEARCH ON
THE FUTURE OF CLOUD METRICS
https://research.spec.org/fileadmin/user_upload/documents/
rg_cloud/endorsed_publications/SPEC-RG-2016-
01_CloudMetrics.pdf
Elasticity
8
- THE DEGREE TO WHICH A SYSTEM IS ABLE TO ADAPT
TO WORKLOAD CHANGES BY PROVISIONING AND DE-
PROVISIONING RESOURCES IN AN AUTONOMIC
MANNER, SUCH THAT AT EACH POINT IN TIME THE
AVAILABLE RESOURCES MATCH THE CURRENT DEMAND
AS CLOSELY AS POSSIBLE
Source: READY FOR RAIN? A VIEW FROM SPEC RESEARCH ON THE FUTURE OF CLOUD METRICS, SPEC RG Cloud
Working Group
Scalability
9
- THE ABILITY OF THE SYSTEM TO SUSTAIN INCREASING
WORKLOADS BY MAKING USE OF ADDITIONAL
RESOURCES, AND THEREFORE, IN CONTRAST TO ELASTICITY,
IT IS NOT DIRECTLY RELATED TO HOW WELL THE
ACTUAL RESOURCE DEMANDS ARE MATCHED BY THE
PROVISIONED RESOURCES AT ANY POINT IN TIME.
Source: READY FOR RAIN? A VIEW FROM SPEC RESEARCH ON THE FUTURE OF CLOUD METRICS, SPEC RG Cloud Working Group
SPEC Cloud - Scalability and Elasticity Analogy
Climbing a mountain
10
c c{
{
{
{
{
Elasticity – time for each step
IDEAL
Scalability
• Mountain: Keep on climbing
• Cloud: keep on adding load without errors
Elasticity
• Mountain: Each step takes identical time
• Cloud: performance within limits as load increases
{
{
{
Cloud Infinitely Scalable and Elastic?
• "Public clouds can have ‘infinite’ scale.”
– In reality
• Resource limitation: Limited instances can be created within
availability zone
• Budget limitation: for testing
• Performance variation: due to multiple tenants
• “Private clouds can be fully elastic”
– In reality
• Performance variation: may have better performance under no
load, unless configured correctly
11
What is a good IaaS cloud benchmark?
• Measures meaningful metrics that are comparable are reproducible
• Measures cloud “elasticity” and “scalability”
• Benchmark IaaS clouds, not the workloads!
• Measures performance of public and private infrastructure-as-a-service (IaaS)
clouds
• Measure provisioning and run-time performance of a cloud
• Uses workloads that resemble “real” workloads
– No micro benchmarks
• Places no restriction on how a cloud is configured.
• Places no restriction on the type of instances a tester can use (e.g., no
restriction on VM/baremetal, CPU, memory, disk, network)
– However, once a tester selects instance types for the test, they cannot be changed
– Minimum machines part of cloud: 4 (with appropriate specs for these machines)
• Allows easy addition of new workloads
12
SPEC Cloud IaaS 2016 Benchmark
• The first industry standard benchmark that measures
the scalability, elasticity, mean instance provisioning
time of clouds, and more
• Uses real workloads in multi-instance configuration
13
SPEC Cloud IaaS 2016 - Workloads
14
YCSB / Cassandra
Framework used by a common set
of workloads for evaluating
performance of different key-value
and cloud serving stores.
KMeans
-Hadoop-based CPU intensive
workload
-Chose Intel HiBench
implementation
YCSB Application
Instance
comprising
7 instances
KMeans Application
Instance
comprising
6 instances
No restriction on instance configuration,
i.e., cpu, mem, disk, net
How does SPEC Cloud IaaS 2016 measure
Scalability and Elasticity?
• Run each application YCSB and Kmeans instance once
(baseline phase)
• Create multiple YCSB and Kmeans application instances over
time (elasticity + scalability phase) with statistically similar
work
– The more application instances are created and perform work, the
higher the scalability
– The low variability in performance during elasticity + scalability phase
relative to baseline phase, the higher the elasticity
15
Benchmark phases
• Baseline phase
– Determine the performance of a single application instance for
determining QoS criteria.
– The assumption is that only a single application instance is run
in the cloud (white-box) or in the user account (black-box)
• Elasticity + Scalability phase
– Determine cloud elasticity and scalability results when multiple
workloads are run
– Stops when QoS criteria as determined from baseline is not
met, or maximum AIs as set by a tester are reached
– Scalability results are normalized against a “reference platform”
16
Elasticity + Scalability phase – pictorial representation
c c
Time
AIindex
Stopping condition (QoS / user specified limits)
c c
c c
AI
Provisioned
2nd run
complete
1st run
complete
YCSB K-Means
17
• Each block indicates the time for a valid
run within an application instance
• Application instances (comprising
multiple instances are provisioning on
the left)
SPEC Cloud™ IaaS 2016 Benchmark - Setup
CloudBench
Baseline
driver
Elasticity +
Scalability
driver
Report
generator
Cloud SUT
Group of boxes represents an
application instance.
Different color within a group
indicates workload generators/
different VM sizes.
Benchmark harness
Benchmark harness. It comprises of Cloud Bench (cbtool)
and baseline/elasticity drivers, and report generators. cbtool
creates application instances in a cloud and collects metrics.
cbtool is driven by baseline and scalability driver to execute
the baseline and elasticity + scalability phases of the
benchmark, respectively.
For white-box clouds the benchmark harness is outside the
SUT. For black-box clouds, it can be in the same location or
campus.
18
Metrics Definitions for SPEC Cloud™ IaaS 2016
Benchmark (1/2) – Primary Metrics
• Scalability
– measures the total amount of work performed by
application instances running in a cloud. Scalability is
reported as a unit-less number.
– Example: Support through of two application instances is
5000 ops/s, then normalized to reference platform,
scalability is 2.0
• Elasticity
– measures whether the work performed by application
instances scales linearly in a cloud. It is expressed as a
percentage (out of 100).
– Example: self-referential. How bad a cloud performs when
the offered load by the benchmark increases
19
Metrics Definitions for SPEC Cloud™ IaaS 2016
Benchmark (2/2) – Secondary Metrics
• Mean Instance Provisioning Time
– measures the interval between instance creation request sent by the
benchmark harness and the moment when the instance is reachable on
port 22 (SSH port).
• AI Run Success
– measures the percentage of runs across all application instances that
successfully completed.
• AI provisioning success
– Measures the percentage of application instances that provisioning
successfully.
• Phase start and end date/time and region (not a metric but
reported as part of fair usage)
– Date/time when the elasticity phase was started, and the region in which
the test was performed.
20
Stopping Conditions
• 20% of the AIs fail to provision
• 10% of the AIs have any errors reported in any of the AI runs
• 50% of the AIs have QoS condition violated across any run
– QoS conditions are defined per workload and are relative to baseline
– YCSB: 50% of AIs have one or more run where
• Throughput is less than 33% of baseline throughput
• 99% Insert or Read latency for any run is 3.33 times greater than Insert and Read
latency in baseline
– K-Means: 50% of AIs have one or more run where
• Complete time is 3.33 times greater than completion time in baseline phase
• Maximum number of AIs as set by the cloud provider has
been provisioned.
21
Published Results
• https://www.spec.org/cloud_iaas2016/results/
22
SPEC Cloud IaaS 2016 Update and Future Cloud
Benchmarks (get involved)
• Update for SPEC Cloud IaaS 2016 coming soon
– Updates for OpenStack Newton
– Easy writing of adapters for new clouds (lib cloud support)
– Miscellaneous bug fixes
• Gen Involved: Future SPEC Cloud* benchmarks for
– Object storage
– Serverless systems
– …
23
https://www.spec.org/benchmarks.html#cloud

More Related Content

What's hot

Design and Implementation of Incremental Cooperative Rebalancing
Design and Implementation of Incremental Cooperative RebalancingDesign and Implementation of Incremental Cooperative Rebalancing
Design and Implementation of Incremental Cooperative Rebalancing
confluent
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Josef Adersberger
 
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
HostedbyConfluent
 
OSMC 2021 | Use OpenSource monitoring for an Enterprise Grade Platform
OSMC 2021 | Use OpenSource monitoring for an Enterprise Grade PlatformOSMC 2021 | Use OpenSource monitoring for an Enterprise Grade Platform
OSMC 2021 | Use OpenSource monitoring for an Enterprise Grade Platform
NETWAYS
 
Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul
Better Kafka Performance Without Changing Any Code | Simon Ritter, AzulBetter Kafka Performance Without Changing Any Code | Simon Ritter, Azul
Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul
HostedbyConfluent
 
Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)
Introduction to ksqlDB and stream processing (Vish Srinivasan  - Confluent)Introduction to ksqlDB and stream processing (Vish Srinivasan  - Confluent)
Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)
KafkaZone
 
Sharing is Caring: Toward Creating Self-tuning Multi-tenant Kafka (Anna Povzn...
Sharing is Caring: Toward Creating Self-tuning Multi-tenant Kafka (Anna Povzn...Sharing is Caring: Toward Creating Self-tuning Multi-tenant Kafka (Anna Povzn...
Sharing is Caring: Toward Creating Self-tuning Multi-tenant Kafka (Anna Povzn...
HostedbyConfluent
 
The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...
The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...
The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...
confluent
 
Kafka Security 101 and Real-World Tips
Kafka Security 101 and Real-World Tips Kafka Security 101 and Real-World Tips
Kafka Security 101 and Real-World Tips
confluent
 
Westpac Bank Tech Talk 1: Dive into Apache Kafka
Westpac Bank Tech Talk 1: Dive into Apache KafkaWestpac Bank Tech Talk 1: Dive into Apache Kafka
Westpac Bank Tech Talk 1: Dive into Apache Kafka
confluent
 
Reliability Guarantees for Apache Kafka
Reliability Guarantees for Apache KafkaReliability Guarantees for Apache Kafka
Reliability Guarantees for Apache Kafka
confluent
 
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
confluent
 
Kafka summit SF 2019 - the art of the event-streaming app
Kafka summit SF 2019 - the art of the event-streaming appKafka summit SF 2019 - the art of the event-streaming app
Kafka summit SF 2019 - the art of the event-streaming app
Neil Avery
 
Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...
Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...
Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...
Guozhang Wang
 
Building an Event-oriented Data Platform with Kafka, Eric Sammer
Building an Event-oriented Data Platform with Kafka, Eric Sammer Building an Event-oriented Data Platform with Kafka, Eric Sammer
Building an Event-oriented Data Platform with Kafka, Eric Sammer
confluent
 
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...
Big Data Spain
 
Kafka Summit SF 2017 - One Data Center is Not Enough: Scaling Apache Kafka Ac...
Kafka Summit SF 2017 - One Data Center is Not Enough: Scaling Apache Kafka Ac...Kafka Summit SF 2017 - One Data Center is Not Enough: Scaling Apache Kafka Ac...
Kafka Summit SF 2017 - One Data Center is Not Enough: Scaling Apache Kafka Ac...
confluent
 
Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform
Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data PlatformStream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform
Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform
confluent
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No Keeper
C4Media
 
"What database can tell about application issues? What application can tell a...
"What database can tell about application issues? What application can tell a..."What database can tell about application issues? What application can tell a...
"What database can tell about application issues? What application can tell a...
Fwdays
 

What's hot (20)

Design and Implementation of Incremental Cooperative Rebalancing
Design and Implementation of Incremental Cooperative RebalancingDesign and Implementation of Incremental Cooperative Rebalancing
Design and Implementation of Incremental Cooperative Rebalancing
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
 
OSMC 2021 | Use OpenSource monitoring for an Enterprise Grade Platform
OSMC 2021 | Use OpenSource monitoring for an Enterprise Grade PlatformOSMC 2021 | Use OpenSource monitoring for an Enterprise Grade Platform
OSMC 2021 | Use OpenSource monitoring for an Enterprise Grade Platform
 
Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul
Better Kafka Performance Without Changing Any Code | Simon Ritter, AzulBetter Kafka Performance Without Changing Any Code | Simon Ritter, Azul
Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul
 
Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)
Introduction to ksqlDB and stream processing (Vish Srinivasan  - Confluent)Introduction to ksqlDB and stream processing (Vish Srinivasan  - Confluent)
Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)
 
Sharing is Caring: Toward Creating Self-tuning Multi-tenant Kafka (Anna Povzn...
Sharing is Caring: Toward Creating Self-tuning Multi-tenant Kafka (Anna Povzn...Sharing is Caring: Toward Creating Self-tuning Multi-tenant Kafka (Anna Povzn...
Sharing is Caring: Toward Creating Self-tuning Multi-tenant Kafka (Anna Povzn...
 
The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...
The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...
The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...
 
Kafka Security 101 and Real-World Tips
Kafka Security 101 and Real-World Tips Kafka Security 101 and Real-World Tips
Kafka Security 101 and Real-World Tips
 
Westpac Bank Tech Talk 1: Dive into Apache Kafka
Westpac Bank Tech Talk 1: Dive into Apache KafkaWestpac Bank Tech Talk 1: Dive into Apache Kafka
Westpac Bank Tech Talk 1: Dive into Apache Kafka
 
Reliability Guarantees for Apache Kafka
Reliability Guarantees for Apache KafkaReliability Guarantees for Apache Kafka
Reliability Guarantees for Apache Kafka
 
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
 
Kafka summit SF 2019 - the art of the event-streaming app
Kafka summit SF 2019 - the art of the event-streaming appKafka summit SF 2019 - the art of the event-streaming app
Kafka summit SF 2019 - the art of the event-streaming app
 
Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...
Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...
Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...
 
Building an Event-oriented Data Platform with Kafka, Eric Sammer
Building an Event-oriented Data Platform with Kafka, Eric Sammer Building an Event-oriented Data Platform with Kafka, Eric Sammer
Building an Event-oriented Data Platform with Kafka, Eric Sammer
 
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...
 
Kafka Summit SF 2017 - One Data Center is Not Enough: Scaling Apache Kafka Ac...
Kafka Summit SF 2017 - One Data Center is Not Enough: Scaling Apache Kafka Ac...Kafka Summit SF 2017 - One Data Center is Not Enough: Scaling Apache Kafka Ac...
Kafka Summit SF 2017 - One Data Center is Not Enough: Scaling Apache Kafka Ac...
 
Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform
Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data PlatformStream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform
Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No Keeper
 
"What database can tell about application issues? What application can tell a...
"What database can tell about application issues? What application can tell a..."What database can tell about application issues? What application can tell a...
"What database can tell about application issues? What application can tell a...
 

Viewers also liked

DATA CENTER
DATA CENTER DATA CENTER
DATA CENTER
Shekar Reddy
 
Open Source Cloud Technologies
Open Source Cloud TechnologiesOpen Source Cloud Technologies
Open Source Cloud Technologies
Salman Baset
 
Unraveling Docker Security: Lessons From a Production Cloud
Unraveling Docker Security: Lessons From a Production CloudUnraveling Docker Security: Lessons From a Production Cloud
Unraveling Docker Security: Lessons From a Production Cloud
Salman Baset
 
Dissecting Open Source Cloud Evolution: An OpenStack Case Study
Dissecting Open Source Cloud Evolution: An OpenStack Case StudyDissecting Open Source Cloud Evolution: An OpenStack Case Study
Dissecting Open Source Cloud Evolution: An OpenStack Case Study
Salman Baset
 
A Survey of Container Security in 2016: A Security Update on Container Platforms
A Survey of Container Security in 2016: A Security Update on Container PlatformsA Survey of Container Security in 2016: A Security Update on Container Platforms
A Survey of Container Security in 2016: A Security Update on Container Platforms
Salman Baset
 
Cloud SLAs: Present and Future
Cloud SLAs: Present and FutureCloud SLAs: Present and Future
Cloud SLAs: Present and Future
Salman Baset
 
Microservice architecture
Microservice architectureMicroservice architecture
Microservice architecture
Slim Ouertani
 
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
Zach Hill
 
Docker containers & the Future of Drupal testing
Docker containers & the Future of Drupal testing Docker containers & the Future of Drupal testing
Docker containers & the Future of Drupal testing
Ricardo Amaro
 
How To Train Your APIs
How To Train Your APIsHow To Train Your APIs
How To Train Your APIs
Ashley Roach
 
Drupal workshop ist 2014
Drupal workshop ist 2014Drupal workshop ist 2014
Drupal workshop ist 2014
Ricardo Amaro
 
Building a REST API Microservice for the DevNet API Scavenger Hunt
Building a REST API Microservice for the DevNet API Scavenger HuntBuilding a REST API Microservice for the DevNet API Scavenger Hunt
Building a REST API Microservice for the DevNet API Scavenger Hunt
Ashley Roach
 
Introduction to Infrastructure as Code & Automation / Introduction to Chef
Introduction to Infrastructure as Code & Automation / Introduction to ChefIntroduction to Infrastructure as Code & Automation / Introduction to Chef
Introduction to Infrastructure as Code & Automation / Introduction to Chef
Nathen Harvey
 
Drupalcamp es 2013 drupal with lxc docker and vagrant
Drupalcamp es 2013  drupal with lxc docker and vagrant Drupalcamp es 2013  drupal with lxc docker and vagrant
Drupalcamp es 2013 drupal with lxc docker and vagrant
Ricardo Amaro
 
Priming Your Teams For Microservice Deployment to the Cloud
Priming Your Teams For Microservice Deployment to the CloudPriming Your Teams For Microservice Deployment to the Cloud
Priming Your Teams For Microservice Deployment to the Cloud
Matt Callanan
 
Docker security: Rolling out Trust in your container
Docker security: Rolling out Trust in your containerDocker security: Rolling out Trust in your container
Docker security: Rolling out Trust in your container
Ronak Kogta
 
DOXLON November 2016 - Data Democratization Using Splunk
DOXLON November 2016 - Data Democratization Using SplunkDOXLON November 2016 - Data Democratization Using Splunk
DOXLON November 2016 - Data Democratization Using Splunk
Outlyer
 
Docker Security
Docker SecurityDocker Security
Docker Security
BladE0341
 
S.R.E - create ultra-scalable and highly reliable systems
S.R.E - create ultra-scalable and highly reliable systemsS.R.E - create ultra-scalable and highly reliable systems
S.R.E - create ultra-scalable and highly reliable systems
Ricardo Amaro
 
Drupal workshop fcul_2014
Drupal workshop fcul_2014Drupal workshop fcul_2014
Drupal workshop fcul_2014
Ricardo Amaro
 

Viewers also liked (20)

DATA CENTER
DATA CENTER DATA CENTER
DATA CENTER
 
Open Source Cloud Technologies
Open Source Cloud TechnologiesOpen Source Cloud Technologies
Open Source Cloud Technologies
 
Unraveling Docker Security: Lessons From a Production Cloud
Unraveling Docker Security: Lessons From a Production CloudUnraveling Docker Security: Lessons From a Production Cloud
Unraveling Docker Security: Lessons From a Production Cloud
 
Dissecting Open Source Cloud Evolution: An OpenStack Case Study
Dissecting Open Source Cloud Evolution: An OpenStack Case StudyDissecting Open Source Cloud Evolution: An OpenStack Case Study
Dissecting Open Source Cloud Evolution: An OpenStack Case Study
 
A Survey of Container Security in 2016: A Security Update on Container Platforms
A Survey of Container Security in 2016: A Security Update on Container PlatformsA Survey of Container Security in 2016: A Security Update on Container Platforms
A Survey of Container Security in 2016: A Security Update on Container Platforms
 
Cloud SLAs: Present and Future
Cloud SLAs: Present and FutureCloud SLAs: Present and Future
Cloud SLAs: Present and Future
 
Microservice architecture
Microservice architectureMicroservice architecture
Microservice architecture
 
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
 
Docker containers & the Future of Drupal testing
Docker containers & the Future of Drupal testing Docker containers & the Future of Drupal testing
Docker containers & the Future of Drupal testing
 
How To Train Your APIs
How To Train Your APIsHow To Train Your APIs
How To Train Your APIs
 
Drupal workshop ist 2014
Drupal workshop ist 2014Drupal workshop ist 2014
Drupal workshop ist 2014
 
Building a REST API Microservice for the DevNet API Scavenger Hunt
Building a REST API Microservice for the DevNet API Scavenger HuntBuilding a REST API Microservice for the DevNet API Scavenger Hunt
Building a REST API Microservice for the DevNet API Scavenger Hunt
 
Introduction to Infrastructure as Code & Automation / Introduction to Chef
Introduction to Infrastructure as Code & Automation / Introduction to ChefIntroduction to Infrastructure as Code & Automation / Introduction to Chef
Introduction to Infrastructure as Code & Automation / Introduction to Chef
 
Drupalcamp es 2013 drupal with lxc docker and vagrant
Drupalcamp es 2013  drupal with lxc docker and vagrant Drupalcamp es 2013  drupal with lxc docker and vagrant
Drupalcamp es 2013 drupal with lxc docker and vagrant
 
Priming Your Teams For Microservice Deployment to the Cloud
Priming Your Teams For Microservice Deployment to the CloudPriming Your Teams For Microservice Deployment to the Cloud
Priming Your Teams For Microservice Deployment to the Cloud
 
Docker security: Rolling out Trust in your container
Docker security: Rolling out Trust in your containerDocker security: Rolling out Trust in your container
Docker security: Rolling out Trust in your container
 
DOXLON November 2016 - Data Democratization Using Splunk
DOXLON November 2016 - Data Democratization Using SplunkDOXLON November 2016 - Data Democratization Using Splunk
DOXLON November 2016 - Data Democratization Using Splunk
 
Docker Security
Docker SecurityDocker Security
Docker Security
 
S.R.E - create ultra-scalable and highly reliable systems
S.R.E - create ultra-scalable and highly reliable systemsS.R.E - create ultra-scalable and highly reliable systems
S.R.E - create ultra-scalable and highly reliable systems
 
Drupal workshop fcul_2014
Drupal workshop fcul_2014Drupal workshop fcul_2014
Drupal workshop fcul_2014
 

Similar to SPEC Cloud (TM) IaaS 2016 Benchmark

Spirent CloudScore
Spirent CloudScoreSpirent CloudScore
Spirent CloudScore
Malathi Malla
 
Performance Benchmarking of Clouds Evaluating OpenStack
Performance Benchmarking of Clouds                Evaluating OpenStackPerformance Benchmarking of Clouds                Evaluating OpenStack
Performance Benchmarking of Clouds Evaluating OpenStack
Pradeep Kumar
 
AskTom: How to Make and Test Your Application "Oracle RAC Ready"?
AskTom: How to Make and Test Your Application "Oracle RAC Ready"?AskTom: How to Make and Test Your Application "Oracle RAC Ready"?
AskTom: How to Make and Test Your Application "Oracle RAC Ready"?
Markus Michalewicz
 
MySQL Replication Performance in the Cloud
MySQL Replication Performance in the CloudMySQL Replication Performance in the Cloud
MySQL Replication Performance in the Cloud
Vitor Oliveira
 
TechTalk_Cloud Performance Testing_0.6
TechTalk_Cloud Performance Testing_0.6TechTalk_Cloud Performance Testing_0.6
TechTalk_Cloud Performance Testing_0.6
Sravanthi N
 
An Evaluation of OpenStack Deployment Frameworks
An Evaluation of OpenStack Deployment FrameworksAn Evaluation of OpenStack Deployment Frameworks
An Evaluation of OpenStack Deployment Frameworks
shane_gibson
 
A survey-report-on-cloud-computing-testing-environment
A survey-report-on-cloud-computing-testing-environmentA survey-report-on-cloud-computing-testing-environment
A survey-report-on-cloud-computing-testing-environment
shritosh kumar
 
Planning open stack-poc
Planning open stack-pocPlanning open stack-poc
Cloud_Testing_The_future_of_softwareV1.04
Cloud_Testing_The_future_of_softwareV1.04Cloud_Testing_The_future_of_softwareV1.04
Cloud_Testing_The_future_of_softwareV1.04
Mrityunjaya Hikkalgutti
 
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld
 
DEVNET-1166 Open SDN Controller APIs
DEVNET-1166	Open SDN Controller APIsDEVNET-1166	Open SDN Controller APIs
DEVNET-1166 Open SDN Controller APIs
Cisco DevNet
 
Interop ITX: Moving applications: From Legacy to Cloud-to-Cloud
Interop ITX: Moving applications: From Legacy to Cloud-to-CloudInterop ITX: Moving applications: From Legacy to Cloud-to-Cloud
Interop ITX: Moving applications: From Legacy to Cloud-to-Cloud
Susan Wu
 
Test Strategy For Future Cloud Architecture
Test Strategy For Future Cloud ArchitectureTest Strategy For Future Cloud Architecture
Test Strategy For Future Cloud Architecture
MaheshShri1
 
Elastic-Engineering
Elastic-EngineeringElastic-Engineering
Elastic-Engineering
Araf Karsh Hamid
 
Hewlett Packard Entreprise | Stormrunner load | Game Changer
Hewlett Packard Entreprise | Stormrunner load | Game ChangerHewlett Packard Entreprise | Stormrunner load | Game Changer
Hewlett Packard Entreprise | Stormrunner load | Game Changer
Jeffrey Nunn
 
Cloud-based performance testing
Cloud-based performance testingCloud-based performance testing
Cloud-based performance testing
abhinavm
 
Cloud Readiness : CAST & Microsoft Azure Partnership Overview
Cloud Readiness : CAST & Microsoft Azure Partnership OverviewCloud Readiness : CAST & Microsoft Azure Partnership Overview
Cloud Readiness : CAST & Microsoft Azure Partnership Overview
CAST
 
Cloudify workshop at CCCEU 2014
Cloudify workshop at CCCEU 2014 Cloudify workshop at CCCEU 2014
Cloudify workshop at CCCEU 2014
Uri Cohen
 
PaaS Lessons: Cisco IT Deploys OpenShift to Meet Developer Demand
PaaS Lessons: Cisco IT Deploys OpenShift to Meet Developer DemandPaaS Lessons: Cisco IT Deploys OpenShift to Meet Developer Demand
PaaS Lessons: Cisco IT Deploys OpenShift to Meet Developer Demand
Cisco IT
 
OCCIware@POSS 2016 - an extensible, standard XaaS cloud consumer platform
OCCIware@POSS 2016 - an extensible, standard XaaS cloud consumer platformOCCIware@POSS 2016 - an extensible, standard XaaS cloud consumer platform
OCCIware@POSS 2016 - an extensible, standard XaaS cloud consumer platform
Marc Dutoo
 

Similar to SPEC Cloud (TM) IaaS 2016 Benchmark (20)

Spirent CloudScore
Spirent CloudScoreSpirent CloudScore
Spirent CloudScore
 
Performance Benchmarking of Clouds Evaluating OpenStack
Performance Benchmarking of Clouds                Evaluating OpenStackPerformance Benchmarking of Clouds                Evaluating OpenStack
Performance Benchmarking of Clouds Evaluating OpenStack
 
AskTom: How to Make and Test Your Application "Oracle RAC Ready"?
AskTom: How to Make and Test Your Application "Oracle RAC Ready"?AskTom: How to Make and Test Your Application "Oracle RAC Ready"?
AskTom: How to Make and Test Your Application "Oracle RAC Ready"?
 
MySQL Replication Performance in the Cloud
MySQL Replication Performance in the CloudMySQL Replication Performance in the Cloud
MySQL Replication Performance in the Cloud
 
TechTalk_Cloud Performance Testing_0.6
TechTalk_Cloud Performance Testing_0.6TechTalk_Cloud Performance Testing_0.6
TechTalk_Cloud Performance Testing_0.6
 
An Evaluation of OpenStack Deployment Frameworks
An Evaluation of OpenStack Deployment FrameworksAn Evaluation of OpenStack Deployment Frameworks
An Evaluation of OpenStack Deployment Frameworks
 
A survey-report-on-cloud-computing-testing-environment
A survey-report-on-cloud-computing-testing-environmentA survey-report-on-cloud-computing-testing-environment
A survey-report-on-cloud-computing-testing-environment
 
Planning open stack-poc
Planning open stack-pocPlanning open stack-poc
Planning open stack-poc
 
Cloud_Testing_The_future_of_softwareV1.04
Cloud_Testing_The_future_of_softwareV1.04Cloud_Testing_The_future_of_softwareV1.04
Cloud_Testing_The_future_of_softwareV1.04
 
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
 
DEVNET-1166 Open SDN Controller APIs
DEVNET-1166	Open SDN Controller APIsDEVNET-1166	Open SDN Controller APIs
DEVNET-1166 Open SDN Controller APIs
 
Interop ITX: Moving applications: From Legacy to Cloud-to-Cloud
Interop ITX: Moving applications: From Legacy to Cloud-to-CloudInterop ITX: Moving applications: From Legacy to Cloud-to-Cloud
Interop ITX: Moving applications: From Legacy to Cloud-to-Cloud
 
Test Strategy For Future Cloud Architecture
Test Strategy For Future Cloud ArchitectureTest Strategy For Future Cloud Architecture
Test Strategy For Future Cloud Architecture
 
Elastic-Engineering
Elastic-EngineeringElastic-Engineering
Elastic-Engineering
 
Hewlett Packard Entreprise | Stormrunner load | Game Changer
Hewlett Packard Entreprise | Stormrunner load | Game ChangerHewlett Packard Entreprise | Stormrunner load | Game Changer
Hewlett Packard Entreprise | Stormrunner load | Game Changer
 
Cloud-based performance testing
Cloud-based performance testingCloud-based performance testing
Cloud-based performance testing
 
Cloud Readiness : CAST & Microsoft Azure Partnership Overview
Cloud Readiness : CAST & Microsoft Azure Partnership OverviewCloud Readiness : CAST & Microsoft Azure Partnership Overview
Cloud Readiness : CAST & Microsoft Azure Partnership Overview
 
Cloudify workshop at CCCEU 2014
Cloudify workshop at CCCEU 2014 Cloudify workshop at CCCEU 2014
Cloudify workshop at CCCEU 2014
 
PaaS Lessons: Cisco IT Deploys OpenShift to Meet Developer Demand
PaaS Lessons: Cisco IT Deploys OpenShift to Meet Developer DemandPaaS Lessons: Cisco IT Deploys OpenShift to Meet Developer Demand
PaaS Lessons: Cisco IT Deploys OpenShift to Meet Developer Demand
 
OCCIware@POSS 2016 - an extensible, standard XaaS cloud consumer platform
OCCIware@POSS 2016 - an extensible, standard XaaS cloud consumer platformOCCIware@POSS 2016 - an extensible, standard XaaS cloud consumer platform
OCCIware@POSS 2016 - an extensible, standard XaaS cloud consumer platform
 

Recently uploaded

Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
flufftailshop
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Intelisync
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Jeffrey Haguewood
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Recommendation System using RAG Architecture
Recommendation System using RAG ArchitectureRecommendation System using RAG Architecture
Recommendation System using RAG Architecture
fredae14
 
Finale of the Year: Apply for Next One!
Finale of the Year: Apply for Next One!Finale of the Year: Apply for Next One!
Finale of the Year: Apply for Next One!
GDSC PJATK
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
Pravash Chandra Das
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
Shinana2
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 

Recently uploaded (20)

Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Recommendation System using RAG Architecture
Recommendation System using RAG ArchitectureRecommendation System using RAG Architecture
Recommendation System using RAG Architecture
 
Finale of the Year: Apply for Next One!
Finale of the Year: Apply for Next One!Finale of the Year: Apply for Next One!
Finale of the Year: Apply for Next One!
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 

SPEC Cloud (TM) IaaS 2016 Benchmark

  • 1. SPEC Cloud™ IaaS 2016 Benchmark Salman Baset, IBM - @salman_baset https://www.spec.org/benchmarks.html#cloud
  • 2. Topics • Standard Performance Corporation (SPEC) • What are cloud characteristics? • What is a cloud benchmark? • SPEC Cloud IaaS 2016 benchmark, the first industry standard benchmark – Workloads – Setup – Metrics – Compliant run – Stopping conditions – Published results • What’s next in SPEC Cloud* benchmarks? 2
  • 3. SPEC Background • Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. – http://www.spec.org/ • Well-known benchmarks such as SPEC CPU 2006, SPEC Virt, SPEC Power… • Cloud Subcommittee released the first industry standard benchmark (SPEC Cloud IaaS 2016) for infrastructure-as-a-service cloud in May 2016. – http://spec.org/benchmarks.html#cloud 3
  • 4. SPEC Cloud IaaS Benchmark and OpenStack • Measuring cloud performance in general and OpenStack performance in particular is hard. – Too many configuration options – Too many metrics – Difficult to repeat experiments and compare results • SPEC Cloud IaaS 2016 can be used to measure and publish performance (and related configuration) of each OpenStack release. – Each release can be measured by the benchmark – Such results can potentially be published before an OpenStack summit or during key note, giving credence to the release. – All results are peer-reviewed before publishing. 4
  • 5. What is a cloud? • Public cloud • Private cloud • Hybrid cloud 5
  • 6. What is a cloud? • SPEC OSG Cloud Subcommittee Glossary • Blackbox Cloud • A cloud-provider provides a general specification of the system under test (SUT), usually in terms of how the cloud consumer may be billed. The exact hardware details corresponding to these compute units may not be known. This will typically be the case if the entity benchmarking the cloud is different from a cloud provider, e.g., public clouds or hosted private clouds. • Whitebox Cloud • The SUT’s exact engineering specifications including all hardware and software are known and under the control of the tester. This will typically be the case for private clouds. Source: https://www.spec.org/cloud_iaas2016/docs/faq/html/glossary.html 6
  • 7. What are cloud characteristics? 7 SPEC RESEARCH GROUP - CLOUD WORKING GROUP https://research.spec.org/working-groups/rg-cloud-working- group.html READY FOR RAIN? A VIEW FROM SPEC RESEARCH ON THE FUTURE OF CLOUD METRICS https://research.spec.org/fileadmin/user_upload/documents/ rg_cloud/endorsed_publications/SPEC-RG-2016- 01_CloudMetrics.pdf
  • 8. Elasticity 8 - THE DEGREE TO WHICH A SYSTEM IS ABLE TO ADAPT TO WORKLOAD CHANGES BY PROVISIONING AND DE- PROVISIONING RESOURCES IN AN AUTONOMIC MANNER, SUCH THAT AT EACH POINT IN TIME THE AVAILABLE RESOURCES MATCH THE CURRENT DEMAND AS CLOSELY AS POSSIBLE Source: READY FOR RAIN? A VIEW FROM SPEC RESEARCH ON THE FUTURE OF CLOUD METRICS, SPEC RG Cloud Working Group
  • 9. Scalability 9 - THE ABILITY OF THE SYSTEM TO SUSTAIN INCREASING WORKLOADS BY MAKING USE OF ADDITIONAL RESOURCES, AND THEREFORE, IN CONTRAST TO ELASTICITY, IT IS NOT DIRECTLY RELATED TO HOW WELL THE ACTUAL RESOURCE DEMANDS ARE MATCHED BY THE PROVISIONED RESOURCES AT ANY POINT IN TIME. Source: READY FOR RAIN? A VIEW FROM SPEC RESEARCH ON THE FUTURE OF CLOUD METRICS, SPEC RG Cloud Working Group
  • 10. SPEC Cloud - Scalability and Elasticity Analogy Climbing a mountain 10 c c{ { { { { Elasticity – time for each step IDEAL Scalability • Mountain: Keep on climbing • Cloud: keep on adding load without errors Elasticity • Mountain: Each step takes identical time • Cloud: performance within limits as load increases { { {
  • 11. Cloud Infinitely Scalable and Elastic? • "Public clouds can have ‘infinite’ scale.” – In reality • Resource limitation: Limited instances can be created within availability zone • Budget limitation: for testing • Performance variation: due to multiple tenants • “Private clouds can be fully elastic” – In reality • Performance variation: may have better performance under no load, unless configured correctly 11
  • 12. What is a good IaaS cloud benchmark? • Measures meaningful metrics that are comparable are reproducible • Measures cloud “elasticity” and “scalability” • Benchmark IaaS clouds, not the workloads! • Measures performance of public and private infrastructure-as-a-service (IaaS) clouds • Measure provisioning and run-time performance of a cloud • Uses workloads that resemble “real” workloads – No micro benchmarks • Places no restriction on how a cloud is configured. • Places no restriction on the type of instances a tester can use (e.g., no restriction on VM/baremetal, CPU, memory, disk, network) – However, once a tester selects instance types for the test, they cannot be changed – Minimum machines part of cloud: 4 (with appropriate specs for these machines) • Allows easy addition of new workloads 12
  • 13. SPEC Cloud IaaS 2016 Benchmark • The first industry standard benchmark that measures the scalability, elasticity, mean instance provisioning time of clouds, and more • Uses real workloads in multi-instance configuration 13
  • 14. SPEC Cloud IaaS 2016 - Workloads 14 YCSB / Cassandra Framework used by a common set of workloads for evaluating performance of different key-value and cloud serving stores. KMeans -Hadoop-based CPU intensive workload -Chose Intel HiBench implementation YCSB Application Instance comprising 7 instances KMeans Application Instance comprising 6 instances No restriction on instance configuration, i.e., cpu, mem, disk, net
  • 15. How does SPEC Cloud IaaS 2016 measure Scalability and Elasticity? • Run each application YCSB and Kmeans instance once (baseline phase) • Create multiple YCSB and Kmeans application instances over time (elasticity + scalability phase) with statistically similar work – The more application instances are created and perform work, the higher the scalability – The low variability in performance during elasticity + scalability phase relative to baseline phase, the higher the elasticity 15
  • 16. Benchmark phases • Baseline phase – Determine the performance of a single application instance for determining QoS criteria. – The assumption is that only a single application instance is run in the cloud (white-box) or in the user account (black-box) • Elasticity + Scalability phase – Determine cloud elasticity and scalability results when multiple workloads are run – Stops when QoS criteria as determined from baseline is not met, or maximum AIs as set by a tester are reached – Scalability results are normalized against a “reference platform” 16
  • 17. Elasticity + Scalability phase – pictorial representation c c Time AIindex Stopping condition (QoS / user specified limits) c c c c AI Provisioned 2nd run complete 1st run complete YCSB K-Means 17 • Each block indicates the time for a valid run within an application instance • Application instances (comprising multiple instances are provisioning on the left)
  • 18. SPEC Cloud™ IaaS 2016 Benchmark - Setup CloudBench Baseline driver Elasticity + Scalability driver Report generator Cloud SUT Group of boxes represents an application instance. Different color within a group indicates workload generators/ different VM sizes. Benchmark harness Benchmark harness. It comprises of Cloud Bench (cbtool) and baseline/elasticity drivers, and report generators. cbtool creates application instances in a cloud and collects metrics. cbtool is driven by baseline and scalability driver to execute the baseline and elasticity + scalability phases of the benchmark, respectively. For white-box clouds the benchmark harness is outside the SUT. For black-box clouds, it can be in the same location or campus. 18
  • 19. Metrics Definitions for SPEC Cloud™ IaaS 2016 Benchmark (1/2) – Primary Metrics • Scalability – measures the total amount of work performed by application instances running in a cloud. Scalability is reported as a unit-less number. – Example: Support through of two application instances is 5000 ops/s, then normalized to reference platform, scalability is 2.0 • Elasticity – measures whether the work performed by application instances scales linearly in a cloud. It is expressed as a percentage (out of 100). – Example: self-referential. How bad a cloud performs when the offered load by the benchmark increases 19
  • 20. Metrics Definitions for SPEC Cloud™ IaaS 2016 Benchmark (2/2) – Secondary Metrics • Mean Instance Provisioning Time – measures the interval between instance creation request sent by the benchmark harness and the moment when the instance is reachable on port 22 (SSH port). • AI Run Success – measures the percentage of runs across all application instances that successfully completed. • AI provisioning success – Measures the percentage of application instances that provisioning successfully. • Phase start and end date/time and region (not a metric but reported as part of fair usage) – Date/time when the elasticity phase was started, and the region in which the test was performed. 20
  • 21. Stopping Conditions • 20% of the AIs fail to provision • 10% of the AIs have any errors reported in any of the AI runs • 50% of the AIs have QoS condition violated across any run – QoS conditions are defined per workload and are relative to baseline – YCSB: 50% of AIs have one or more run where • Throughput is less than 33% of baseline throughput • 99% Insert or Read latency for any run is 3.33 times greater than Insert and Read latency in baseline – K-Means: 50% of AIs have one or more run where • Complete time is 3.33 times greater than completion time in baseline phase • Maximum number of AIs as set by the cloud provider has been provisioned. 21
  • 23. SPEC Cloud IaaS 2016 Update and Future Cloud Benchmarks (get involved) • Update for SPEC Cloud IaaS 2016 coming soon – Updates for OpenStack Newton – Easy writing of adapters for new clouds (lib cloud support) – Miscellaneous bug fixes • Gen Involved: Future SPEC Cloud* benchmarks for – Object storage – Serverless systems – … 23 https://www.spec.org/benchmarks.html#cloud

Editor's Notes

  1. Scalability measures the total amount of work performed by application instances running in a cloud. The aggregate work performed by one or more application instances should linearly scale in an ideal cloud. Scalability is reported as a unit-less number. The number is an aggregate of workloads metrics across all application instances running in a cloud normalized by workload metrics in a reference cloud platform. The reference platform metrics are an average of workload metrics taken across multiple cloud platforms during the benchmark development. Elasticity measures whether the work performed by application instances scales linearly in a cloud. That is, for statistically similar work, the performance of N application instances in a cloud must be the same as the performance of application instances during baseline phase when no other load is introduced by the tester. Elasticity is expressed as a percentage (out of 100). The higher, the better. Elasticity is self-referential.