OCC Overview OMG Clouds Meeting 07-13-09 v3

www.opencloudconsortium.org
An Overview of the Open Cloud Consortium
Robert GrossmanOpen Cloud Consortium
OMG Cloud Computing Interoperability Workshop
July 13, 2009

This talk represents my personal opinions and does not necessarily represent a consensus position of the Open Cloud Consortium or positions of any of its members.
2

Part 1. Introduction
3

Two Types of Clouds
On-demand resources & services over a network at the scale of a data center
On-demand computing instances (IaaS)
IaaS: Amazon EC2, S3, etc.; Eucalyptus
supports many Web 2.0 applications/users
On-demand cloud services for large data cloud applications (Large Data PaaS)
GFS/MapReduce/Bigtable, Hadoop, Sector, …
Manage and compute with large data (say 100+ TB)
4

Delivering On-Demand Computing Instances
5
App
App
App
App
App
Table-based Data (BigTable) Services
Relational Data Services (Pig, etc.)
App
App
…
Cloud Compute Services (MapReduce, Variants & Generalizations)
App
App
Cloud Storage Services
For both architectures, you are interested in filling a data center.
App
App
App

Activities Currently Focused Around Five Use Cases
Moving an existing cloud application from Cloud 1 to Cloud 2 without changing the application.
Providing surge capacity for an application on Cloud 1 using any of the Clouds 2, 3, … (without changing the application).
Migrate / port
Surge / burst
Cloud 2
Cloud 1

Large Data Cloud Use Cases
Moving a large data cloud application from one large data cloud storage service to another.
Moving a large data cloud application from one large data cloud compute service to another.
App 1
App 2
Large Data Cloud Compute Services
Large Data Cloud Storage Services

Inter-Cloud Use Case
Inter-cloud communication between two HIPAA compliant clouds.
HIPPA Cloud 2
HIPAA Cloud 1

Part 2. The Open Cloud Consortium (OCC)
9

OCC is a member driven organization that:
Supports the development of standards.
Supports the development of benchmarks.
Supports reference implementations for cloud computing, preferably open source.
Sponsors workshops and other events related to cloud computing.
10

OCC Members
Current Members
Aerospace, Cisco, MIT Lincoln Labs, Northwestern University, Open Data Group, Sector Project, University of Chicago, University of Illinois at Chicago and Yahoo.
Current Contributing Members & Partners
CalIT2, Johns Hopkins University, National Lambda Rail

OCC Structure
OCC is a consortium operated by a not-for-profit.
OCC is member-driven.
OCC is oriented to companies and organizations, not individuals.
Governance structure is available from web site.
12

OCC Working Groups
Members set up working groups.
Large Data Clouds Working Group
Open Cloud Testbed Working Group.
Standard Cloud Performance Measurement (SCPM)
- Working collaboratively with CCIF
Information Sharing and Security Working Group.

Part 3. Large Data Clouds Benchmarks and Interoperability Working Group
14
*
*OCC is not affiliated with the Hadoop Project.

Cloud Benchmarks
To help inform purchase decisions.
To help inform the design of cloud applications.
To help inform cloud architecture decisions.

Benchmarks for Large Data Clouds
Until recently, the only benchmark used was Terasort (sorting 10 billion 100 byte records)
Replaced by Gray Sort and Minute Sort
Gray Sort tries to maximize TB / min sorted on 100 TB or more of data.
Hadoop holds the current Gray Sort and Minute Sort records.
Problem: sort is just one of the types of work load for analytic applications

MalStone
MalGen – generates synthetic data with realistic distributions.
MalStone A & B – “stylized” computations that can be used as benchmarks for architectures, software and systems for large data clouds.
Open source and available at code.google.com/p/malgen
17

MalStone B
18
entities
sites
dk-2
dk-1
dk
time

MalStone B Benchmark
19

ThriftStore
Cloud App 1
Cloud App 2
App n
…
ThriftStore
Hadoop DFS
KFS
Sector DFS
Each cloud application can access any of the large data cloud storage services without changing any of its APIs.
Based upon the open source Thrift project.

Part 4. Standard Cloud Performance Measurement (SCPM) Working Group
21

Establish Benchmarks Appropriate for Four Use Cases
Moving an application between two clouds.
Obtaining burst instances from multiple cloud service providers for a private/public hybrid application.
Moving a large data cloud application to another large data cloud storage service.
Moving a large data cloud application to another large data cloud compute service.

Hierarchy of Benchmarks
Benchmarks for basic operations
Kernel Benchmarks
Basic functions and routines
Stripped down versions of programs
Full applications
Goal is to develop small suite of these benchmarks for four use cases.

Please Get Involved…
The Standard Cloud Performance Measurement (SCPM) Working Group is just starting, please get involved.

Part 5. OCC Sponsored Events
25

OCC/CCIF Jointly Sponsored Events
International Summit on Governmental Cloud Computing
Ottawa's Westin Hotel on October 6, 2009
Cloud Computing and Its Applications 2009
Chicago on October 20-21, 2009

We Welcome New Members
Companies and organizations are welcome to join the Open Cloud Consortium (OCC)
www.opencloudconsortium.org/membership.html
Join one of our working groups
Large Data Clouds Working Group
Standard Cloud Performance Measurement (SCPM) Working Group
Information Sharing & Security Working Group

For More Information
info@opencloudconsortium.org