Cloud Standards in the Real World: Cloud Standards Testing for Developers

Open Cloud Standards 
In The Real World: 
Cloud Standards Testing
For Developers
A Project Of the National Science Foundation Cloud and Autonomic Computing 
Industry/University Cooperative Research Center At Texas Tech University
Alan Sill, Ph.D
Site Director
July 21, 2014OSCON Open Cloud Day

First, some preliminaries…
What it often looks like when developers encounter standards committees.
OSCON 2014 Open Cloud Day
July 21, 2014

What it ought to look like:
(Taken from an actual Cloud Plugfest.)
July 21, 2014

A New Research Effort
July 21, 2014

NSF CAC Current Participants:
July 21, 2014
CAC Industry Members:
Technical Partnerships:
Other CAC University Sites:
Others
to
come!
Texas Tech University Other CAC Sites
…
(Pending)(Existing)

New NSF
Award — 
April 2014

Background: Large-Scale Academic Computing
July 21, 2014
In Top500!
!
3rd in Texas and Big 12!
!
In Top 100 academic
institutions in the world !
Supports high-quality
academic research across
many fields of science,
business data analysis 
and engineering.
TTU High Performance
Computing Center:

Background: Big Data
July 21, 2014
2014:!
Phones: 100+ Gigabytes!
Science and Business:  
100s to 1000s of Petabytes

Experience: 
 
Related CyberInfrastructure
Projects

Experience In Global-Scale Computing
July 21, 2014

The Role of Standards for Risk Reduction and
Inter-operation in XSEDE
XSEDE: The Next Generation of
US Supercomputing Infrastructure
Cloud and grid standards now
power some of the largest
academic supercomputing
infrastructures in the world!

Blacklight

Shared
Memory

4k
Xeon
cores
!
Darter

24k
cores

!
Nautilus

Visualization

Data
Analytics

!
Keeneland

CPU/GPGPU
Stampede

460K
cores

w.
Xeon
Phi

>1000
users

Upgrade
in
2015
Wrangler

Date
Analytics
Trestles

IO-‐intensive

10k
cores

160
GB
SSD/Flash

!
Gordon

Data
intensive

64
TB
memory

300
TB
Flash
Mem
Open
Science
Grid

High
throughput

124
sites
Blue
Waters

Leadership
Class
SuperMIC

380
nodes
–
1PF

(Ivy
bridge,
Xeon
Phi,
GPU)
Over
13
million
service
units/day
typically

delivered
as
of
2014
across
all
XSEDE

supercomputing
sites
(about
3
million
core
hours/
day),
totaling
about
1.6
billion
core
hours
per
year
Yellowstone

Geosciences
US National Cyberinfrastructure
Promote an open,
robust,
collaborative, and
innovative
ecosystem
Adopt, create
and
disseminate
knowledge
Extend

the impact  
of cyber-
infrastructurePrepare the
current and
next
generation
Provide technical
expertise and
support services
Collaborate with
other CI groups
and projects
NSF Supercomputing 
Resources
July 21, 2014
FutureGrid*

CS
testbed
Maverick

Visualization

Data
Analytics
Comet

“Long
Tail
Science”

47k
cores/2
PF

High
throughput
ACI-‐REF

Campus
sharing
Grids
Credit: Irene Qualters, US
National Science Foundation

XSEDE Services Layer: Simple services combined in useful ways
–Resource
Namespace
Service
1.1

–OGSA
Basic
ExecuKon
Service

–OGSA
WSRF
BP
–
metadata
and
noKﬁcaKon

–OGSA-‐ByteIO

–GridFTP

–JSDL,
BES,
BES
HPC
Proﬁle

–WS
Trust
Secure
Token
Services

–WSI
BSP
for
transport
of
credenKals

–…
(more
than
we
have
room
to
cover
here)
Examples – (not
a complete list)
Bottom line: XSEDE represents a demonstration of the effective use of standards  
to provide interoperability within large-scale supercomputing cyberinfrastructure.
Used between
supercomputers and
all the way to
university campuses!

Distributed Across 124 Sites
Open Science Grid currently consists of over 124 geographical sites, operating on a wide variety of computing systems
600k - 800k jobs/day!

Virtual Organizations, July 18th 2011
Science VOs on the Open Science Grid
• Astrophysics
• Biochemistry
• Bioinformatics

• Earthquake
Engineering
• Genetics
• Gravitational-‐wave
physics

• Mathematics
• Nanotechnology
• Nuclear
and
particle
physics
… and many others!

www.egi.euEGI-InSPIRE RI-261323 www.egi.euEGI-‐InSPIRE
RI-‐261323
XSEDE/EGI BoF - XSEDE 2013
EGI international presence numbers
Storage Value
(yearly
increase)
Disk
(PB)

235
PB
(+69%)
Tape
(PB)

176
PB
(+32%)
Value
(yearly
increase)
CPU
cores 361,300
across
53
countries

(1.44
M
job/day)
Standards-based international collaboration

EGI Federated Cloud: a successful standards-based
international federated infrastructure
Credit: David Wallom
Chair EGI Federated Cloud Task Force
Members
•70 individuals
•40 institutions
•13 countries
Stakeholders
•23 Resource Providers
•10 Technology Providers
•7 User Communities
•4 Liaisons
Technologies
•OpenStack
•OpenNebula
•StratusLab
•CloudStack (in
evaluation)
•Synnefo
•WNoDeS
BSC
CNRS
LMU
OeRC
Masaryk
TUD
IFAE
Cyfronet
100%IT
CESNET
RADICAL
SRCE
DANTE
FZJ
GRNET
GWDG
STFC
SARA
KTH
INFNFCTSG
EGI.eu
Imperial
CESGACETA
IFCA
IGI
IPHC
IN2P3
SZTAKI
IISAS SixSq
Standards
•OCCI (control)
•OVF (images)
•X.509 (authN)
•CDMI (storage -
under development)
(Updated July 2014)
An
international
standards-
based 
federated
cloud
infrastructure:"
 
In production,  
in use and
growing!

www.egi.euEGI-InSPIRE RI-261323
Federated Cloud architecture
Open to new members: Join as user, or as an IaaS/PaaS/SaaS service provider: http://go.egi.eu/cloud
Cloud
hypervisor 
(e.g.
OpenStack,
OpenNebula,

EmotiveCloud,
Okeanos…)
Cloud
site 
academic/commercial
Domain
specific
services
in

Virtual
Machine
Images
FedCloud
User
interfaces
Standards
used
to
enable
federation

• OCCI:
VM
Image
management

• OVF:
VM
Image
format

• X509:
Authentication

• (CDMI:
Storage)
FedCloud

Operation

interfaces
• Information
system
(BDII)

• Monitoring
(SAM)

• Accounting
(APEL)

• AAI
(Perun)
Virtual
organisations
• GLUE2:
Resource
discovery

and
Description

• Others
in
development

Federation
monitoring

Examples of
CAC@TTU
Involvement In Cloud
Standards Testing

US NIST SAJACC Working Group 
Alan Sill, TTU and Eugene Luster, R2AD, co-chairs
NIST Cloud Computing and Mobility 
Forum and Workshop

March 25, 2014
SAJACC Phase II
Use Case Deﬁnition and Testing Project for
Cloud Standards,APIs and Cloud Software
Product Implementations

http://collaborate.nist.gov/twiki-cloud-computing/bin/view/CloudComputing/SAJACC
NIST SAJACC Public Process
July 21, 2014

European Commission“Standards & Interoperability for  
e-Infrastructure Implementation”Initiative
SIENA
(2010-‐2012)
was
a
Support
Ac*on
funded
by
the
European

Commission
under
FP7
(2007-‐13)
CapaciTes
programme.

This
project
created
the
ongoing
“CloudScape”
conference
series

and
led
to
other
related
projects,
such
as
CloudWatchHub
and
the

CloudScout
eﬀort
aimed
at
enhancing
the
adopTon
of
cloud

methods
by
SMEs.
A
coordinated
eﬀort...
...towards
the

delivery
of
a
future 
e-‐Infrastructures

Roadmap....
...aligned
with
the

needs
of
European

and
naVonal

iniVaVves
and
the

evolving
world.
IEG
–
Industry
Expert
Group
Partners
Interna*onal
Experts

REB
–
Roadmap
Editorial

Board
SLG
–
Special
Liaison
Group
Standards
Development
Organisa*ons
(SDOs)
Distributed
Compu*ng
Infrastructure
(DCI)
Ini*a*ves
Consolidated
Cloud
Compu*ng
Workshops
23

Open-access
academically
hosted Cloud
Interoperability
Testbed project

Target Cloud Standards-Related Organizations:
• It is often said that there are “too many standards organizations”.  
This is a lot like saying there is “too much software”.
• Each has its own area of specialty, its own contributor base, and its  
own method of funding to develop its work products.
• CAC will study products and effectiveness of each of these organizations.
July 21, 2014
…

Initial CAC@TTU Project Areas
1. Product and Standards Testing
• Cloud Performance Testbed
• Cloud Standards Testbed
• Cloud Interoperability Testbed
• Cloud Security Testbed <— (Future)
2. Design Labs
• Storage Design and Testing Lab
• Network Design and Testing Lab
3. Developer Events
• Cloud Plugfest Series
• Participation in technical partner events
• Organization of and participation in conferences
July 21, 2014
p CAC@TTU is new!
n More coming…

• Cloud Standards Testbed!
• Cloud Performance Testbed!
• Cloud Interoperability Testbed!
• Cloud Tester Benchmark Suite
Core Technology Efforts
*
* (In cooperation with The Aerospace Corporation and other CAC partners)

Continuing
series…
!
Oriented towards
REAL
DEVELOPMENT
!
Past events 
co-sponsored by
many open
source and
standards-related
organizations
including
 
OGF, DMTF,
SNIA, OASIS,
ETSI, OCEAN
and OW2
Developer-oriented in-
person standards and
software testing series
Easy to get involved
and join in events as
developers or project
researchers
Cloud Plugfest 11 just
completed!
http://cloudplugfest.org
Cloud Plugfest Developer Series:
July 21, 2014
More events in
planning pipeline.

CAC 
Cloud Standards
Testing and
Implementation Projects

Testing Cloud-Oriented Use Cases
§ Builds on results of publicly accessible documents and forums for general
interoperability testing, in addition to internal ones that may be set up by
companies and US agencies.
§ Up to now these have mostly been approached piecemeal.
§ CAC is carrying out an organized project within the center to extend these
techniques into tools that can be used for formal support of cloud
conformance, interoperability and requirements acceptance testing on an
ongoing basis.
§ This project builds on previous public eﬀorts such as SAJACC.
§ Our plan is to work with and contribute to public processes in open settings
while creating conditions in which our expertise and value can be applied for
center members to cloud standards and product testing.
July 21, 2014

Current Standards Being Tested
§ Cloud Data Management Interface (CDMI) from the Storage Networking
Industry Association (SNIA). (Reference implementation available.)
§ Open Cloud Computing Interface (OCCI) from the Open Grid Forum
(OGF). (Many implementations available.)
§ Cloud Infrastructure Management Interface (CIMI) and Open
Virtualization Format (OVF) from the Distributed Management Task Force
(DMTF). (CIMI implementations just becoming available; OVF
implementations well established.)
§ Topology and Orchestration Specification for Cloud Applications
(TOSCA) from the Organization for the Advancement of Structured
Information Standards (OASIS). (Several implementations available.)
§ Others under active evaluation.
July 21, 2014

Texas Tech Leadership Team
Alan Sill, Ph.D
Site Director
Yong Chen, Ph.D
Associate Site Director
Ravi Vadapalli, Ph.D
Associate Site Director

Experienced university-based
researchers carrying out
projects with real-world
applicability that draw on
decades of experience and
the best, newest techniques.
Publication oriented, but
available for practical
member-driven projects.
Full university support from
TTU and other participating
CAC university sites.
July 21, 2014
v
COMPUTER SCIENCE
Rattikorn Hewett, Ph.D.
Chairperson in the Department
of Computer Science
Areas of expertise » Security and privacy in cloud environments
» Intelligent data management & understanding
» Automated web service composition
Sunho Lim, Ph.D.
Assistant professor in the
Department of Computer Science
Areas of expertise » Wireless networks and mobile computing
» Mobile data management
» Mobile software
» Network security
Mohan Sridharan, Ph.D.
Areas of expertise » Learning and estimation in Big Data domains
» Stochastic machine learning
» Robotics and AI
Noe Lopez-Benitez, Ph.D.
Associate professor in the
Areas of expertise » Cloud computing
» Parallel and distributed computing
» High-performance computing
Yong Chen, Ph.D.
Areas of expertise » Cloud computing storage systems
» Data management and data centers
» Cloud programming and software environments
» Cloud platform architecture and virtualization
Eunseog Youn, Ph.D.
Areas of expertise » Large-scale machine learning
» Feature selection
» Text mining
» Bioinformatics
FACULTY MEMBERSHIGH PERFORMANCE COMPUTING CENTER
SCHOOL OF LAW
Victoria Sutton, Ph.D.
in the School of Law
Areas of expertise » Biosecurity
» Emerging technologies
» Energy
» Constitutional law
Many University Researchers, Deep Expertise

INDUSTRIAL ENGINEERING
Timothy Matis, Ph.D.
Department of Industrial
Engineering
Areas of expertise » Operations research
» Stochastic processes
» Queueing theory
» Ad-hoc communication networks
» Conceptual learning theories and virtual
learning environments
Areas of expertise » Mixed-integer programming, theory and
computation
» Nonlinear programming, theory and
computation
» Operations research probabilistic methods
» Applied management science
MATHEMATICS & STATISTICS
Philip W. Smith, Ph.D.
Senior director, High Performance
Computing Center
Areas of expertise » Optimization
» Design and implementation of parallel and
serial codes
» Code optimization
» Numerical analysis and linear algebra
MECHANICAL ENGINEERING
Fazle Hussain, Ph.D.
Professor in the Department of
Mechanical Engineering
Areas of expertise » Fluid Flow, turbulence, & transport phenomena
» Microseismology and inverse scattering
» Modeling cancer tissue growth and
targeted drug delivery
» Atmospheric boundary layers and wind
turbine aerodynamics
Siva Parameswarn, Ph.D.
Professor in the Department of
Mechanical Engineering
Areas of expertise » Modeling HF oscillatory ventilators
» Aerodynamics of ground vehicles
» Wake development behind wind turbines
PHYSICS
Ismael Regis de Farias Jr., Ph.D.
Department of Industrial Engineering
Alan Sill, Ph.D.
Adjunct Professor in the
Department of Physics
Areas of expertise » High performance computing
» Data-intensive scaled resource management
» Large-scale distributed computing
» Standards development
July 21, 2014

FACULTY MEMBERS
RAWLS COLLEGE OF BUSINESS
Peter H. Westfall, Ph.D.
Statistics in the Rawls College of
Business Administration
Zhangxi Lin, Ph.D.
Department of ISQS
Areas of expertise » Advanced analytics
» Business intelligence
» Statistics
» Biostatistics
» Bioinformatics
Areas of expertise » Big data analytics
» Business intelligence
» Information systems
» Electronic commerce
Mayukh Dass, Ph.D.
Assistant Professor of Marketing
in the Rawls College of Business
Areas of expertise » Brand analytics
» Business strategy analysis
» Dynamic environment modeling
» Network analysis
PETROLEUM ENGINEERING
Marshall Watson, Ph.D., P.E.
Roy Butler Chair and chairman
of the Bob L. Herd Department
of Petroleum Engineering
Mohamed Soliman, Ph.D., P.E.
George P. Livermore Chair and
Professor in the Bob L. Herd Dept.
of Petroleum Engineering
Bob L. Herd Department of
Petroleum Engineering
Ravi Vadapalli, Ph.D.
Adjunct Professor in the
Department of Petroleum
Engineering
Areas of expertise
» Production optimization
» Coal bed methane
» Unconventional reservoir analysis
Areas of expertise » Fracturing
» Reservoir engineering
» Well test analysis
» Conformance
Areas of expertise
» Reservoir simulation
» Advanced well completion
Areas of expertise » High performance computing
» Reservoir modeling
» Cancer radiotherapy informatics
» Cloud and autonomic computing
July 21, 2014

Links For Further Information and To Join:
§ NSF Cloud and Autonomic Computing Center main site:  
http://nsfcac.org
§ CAC@TTU information and membership materials: 
http://www.depts.ttu.edu/cac/
§ Cloud Plugfest developer series: http://cloudplugfest.org
§ NSF I/UCRC main site: http://www.nsf.gov/eng/iip/iucrc/
§ Cloud standards organization compilation:  
http://cloud-standards.org
§ NSF CAC@TTU contact email: cac.info@ttu.edu
July 21, 2014

Cloud Standards in the Real World: Cloud Standards Testing for Developers

Cloud Standards in the Real World: Cloud Standards Testing for Developers

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Cloud Standards in the Real World: Cloud Standards Testing for Developers

Similar to Cloud Standards in the Real World: Cloud Standards Testing for Developers (20)

More from Alan Sill

More from Alan Sill (15)

Recently uploaded

Recently uploaded (20)

Cloud Standards in the Real World: Cloud Standards Testing for Developers