State of the Stack April 2013

Randy Bias
Randy BiasCloud Pioneer, Founding Member of OpenStack Foundation, and Technology Disruptor
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution*
* All unlicensed or borrowed works retain their original licenses
State of the Stack - 2013
Game. Over. OpenStack is The Stack.
June 10th, 2013 - SNIA SPDEcon
(first edition delivered at OpenStack Summit, April 2013)
@randybias
Introduction
Who
3
OpenStack FoundationBoard of Directors
Prod. OpenStack pioneer, Cloudscaling:
Wins: KT, Internap, LivingSocial, Seagate (EVault),
IBS Datafort, major U.S. carriers, & others
Part of OpenStack community since July 2010 (launch)
Top 10 Cloud Computing Pioneer
I run an OpenStack product company
I believe the pioneers to emulate are:
I have run big data centers
100K+ sq ft, 1,000s of physical servers, 100s of switches
My Bias
4
5
1 What is OpenStack?
3 History & Momentum
4 Stackology - a stack taxonomy
5 Stacking it Up - a dive into the projects
6 Stack Gaps - what’s missing?
7 Stack Politics - who’s playing?
9 Summary
2 Why the Success?
8 Who’s using it and how?
What is OpenStack?
OpenStack From 10km
7
Networking
OPENSTACK
CLOUD OPERATING SYSTEM
Standard Hardware
Compute Storage
Your Applications
OpenStack Dashboard
OpenStack Shared Services
APIs
OpenStack Mission
8
"To produce the ubiquitous Open Source cloud computing
platform that will meet the needs of public and private
cloud providers regardless of size, by being simple to
implement and massively scalable."
Code Community
OpenStack Foundation Mission
9
The OpenStack Foundation is an independent body providing shared
resources to help achieve the OpenStack Mission by Protecting,
Empowering, and Promoting OpenStack software and the
community around it, including users, developers and the entire
ecosystem.
The ubiquitous cloud
computing platform
What it is
Some say ...
... it’s an Infra-as-a-Service (IaaS)
... it’s a cloud operating system
... it’s a tool for building private clouds
We say it’s “The Stack”
... think Linux
... think Java
... think ubiquitous open source cloud toolkit
... think Game Changer
10
The Battle is Over (open src)
11
OpenStack Launch
OpenStack CloudStack Eucalyptus OpenNebula
Source: trends.google.com
Battle is Nearly Over (closed src)
12
OpenStack vSphere vCloud
Source: trends.google.com
Linux 2000 vs. Linux 2009
13
Is this OpenStack’s Trajectory?
Operating system family market share
2000 2009
Unix Linux
Linux Unix
Source: Linux Magazine
http://www.linux-mag.com/id/7749/
Linux
UNIX
BSD
Windows/Other
Mixed
Fastest Growing Global Open Source Community
14
COMPANIES
TOTAL DEVELOPERS AVERAGE MONTHLY
CONTRIBUTORS
CODE CONTRIBUTIONS
929 245 3,241
189 TOP 10 COUNTRIES
9,000+
INDIVIDUAL MEMBERS
“OpenStack appears to be a more advanced or more modern open source project than
some of its predecessors because it's a highly coordinated effort.”
– Charlie Babcock
Information Week
COUNTRIES
100
United States, China, India,
Great Britain, Australia, France,
Russia, Canada, Ireland, Germany
15
Grizzly Stats
CONTRIBUTORS
PATCHES / DEV NEW DRIVERS TEST CLOUDS DEPLOYED DAILY
~14 15 700
517 (+56%)
TOP 10 CONTRIBUTING COMPANIES
7,620
PATCHES SUBMITTED
“OpenStack appears to be a more advanced or more modern open source project than
some of its predecessors because it's a highly coordinated effort.”
– Charlie Babcock
Information Week
NEW FEATURES
230
Red Hat, Rackspace, IBM, HP, Nebula,
Intel, eNovance, VMware, Cloudscaling,
DreamHost
Why the Success?
Who or What Should We Thank?
Rackspace for Letting Go
OpenStack Foundation & Community
Particularly, all of the companies who realized this could be big
Hype Curve
The OpenStack Infrastructure Team
Oslo Project (openstack-common)
A thankless job allowing shared code & cleaner projects
The Big Enterprises for Driving Interest
PTL Generational Shift
17
Infrastructure Team
Massive Effort -> Improved Quality
Gated Commits
All Code Has to Jump Through Gates
Tempest Test Framework
Code Reviews & Continuous
Integration
Jenkins, Gerrit
At scale: jenkins.openstack.org
18
1/4 pages
History &
Momentum
Jul
Inaugural
Design
Summit in
Austin
2010
20
OpenStack
launches
with 25+
partners
First ‘Austin’
code release
with 35+
partners
Oct Nov
First public
Design
Summit in
San Antonio
Austin
OpenStack Object Storage prod
OpenStack Compute dev preview
Launch!
2010 - The Launch Year
Source: Too many to list; blame me for inaccuracies
2011 - Growing Pains & Early Adopters
Feb
2nd Summit
21
Rackspace
announces plans
to launch
independent
Foundation in
2012
OctApr
3rd Summit
(Santa Clara)
adds
Conference
Governance moves forward with
project technical leads (PTL),
policy board elections (PPB)
Jul
First
Anniversary
Bexar
OpenStack Compute for mid-size prod
OpenStack Image Service added to core
Cactus
OpenStack Compute for larger-
scale prod
Sep
Diablo
Major stability release
First 6-mo cycle release
2011
Decision to shift from 3-mo to
6-mo dev cycle
Jan
1st Swift
Public Cloud
Internap w/ Cloudscaling
Happy
Birthday!
1st Nova
Public Cloud
Internap w/ Cloudscaling
Created
framework for
Foundation as
a community
Feb Apr Aug
19 companies
announce
public support
for Foundation
Framework &
documents ratified
by community
22
Drafting committee
formed – creating
legal documents
OpenStack
Foundation
“officially”
launches
Sep
Essex
OpenStack Identity in core
OpenStack Dashboard in core
Folsom
OpenStack Block
Storage in core
OpenStack
Networking in core
2012
May
HP
Cloud
Launch
(Beta)
Citrix Bails
(how’s that going for ya?)
Jan
AT&T Joins
OpenStack
Internal production (private)
Oct
Gartner
Report
(teeth gnashing followed)
Inaugural OpenStack
Foundation Board
meeting
VMware, Intel, & NEC
accepted as Gold members
Board
Elections
2012 - Rise of the Foundation & Prod Deployments
2013/2014 - Breakout Growth Years
Apr Oct
23
Q1
Havana
OpenStack Metering in integration
OpenStack Orchestration in integration
LBaaS?
“I” Release
2014
Grizzly
OpenStack Metering in incubation
OpenStack Orchestration in incubation
2013
First Summit
100% run and
funded by
Foundation
First
International
Summit
(APAC?)
In every single category, the
top 3 vendors support OpenStack
Incredible Industry Support
24
top 3 switch vendors top 3 storage vendors top 3 hypervisors
top 3 router vendors top 3 blade vendors top 3 linux vendors
top 3 x86 vendors
Developer Growth
25
Contributors per month (ohloh)
Developer Growth Comparison
Contributors per month (ohloh)
26
Dev Growth by Git Contributors
Qingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1
http://www.qyjohn.net/?p=3120
27
Accumulated Community
Qingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1
http://www.qyjohn.net/?p=3120
28
Growth by Domain (company - roughly)
Qingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1
http://www.qyjohn.net/?p=3120
29
For the CloudStack
projects, influence from
Citrix is quite obvious, over
45% of github.com
commits come from
accounts belonging to
citrix.com and cloud.com.
0
750
1500
2250
3000
Austin Santa Clara San Francisco Portland
Summit Growth
30
Established Marketing Reach
OpenStack.org 241k/visits month:
Software: 300K downloads
Membership: 9000+, Over 90% subscribe to newsletter
Relationships with Tier 1 publications and analysts
31
17,693
Followers
(+50% from 8/12)
Stackology
OpenStack From 10km
33
Networking
OPENSTACK
CLOUD OPERATING SYSTEM
Standard Hardware
Compute Storage
Your Applications
OpenStack Dashboard
OpenStack Shared Services
APIs
CLI toolsDashboard Other tools
ComputeNetworkingOrchestration
Hypervisor(s) Queuing
Database
/ KVS
/ Cache
External
Block
Provider
Physical
Network
Provider
Provisioning
Log
Aggregation
Health
Monitoring
etc.
REST Meter Data
REST
SQL, etc. Varies Varies Varies AMQP/0MQ
Topology &
Metadata
Metering
REST
REST REST
DNS
Image
Management
Identity
Management
REST
Block
Storage
Object
Storage
OpenStack (m)Architecture Slide
34
UI Layer
Elastic
Services
Layer
Otherstuff,youprobablyneed/want
Data &
Resource
Layer
Shared
Services
Layer
Project
Name
Description Layer
AWS
Equivalent
Codename
Dashboard Self-service, role-based web
interface for users and administrators
UI Console Horizon
Compute Provision and manage large pools of
on-demand computing resources
Elastic
Service
EC2 Nova
Block Storage Volumes on commodity storage gear,
and drivers for turn-key block storage
solutions
Elastic
Service
EBS Cinder
Object
Storage
Petabytes of reliable storage on
standard gear
Elastic
Service
S3 Swift
Networking L2-focused on-demand networking
with some L3 capabilities
Elastic
Service
VPC Quantum
Orchestration Application orchestration layer that
runs on top of and manages
OpenStack Compute
Elastic
Service
CloudFormation,
CloudWatch
Heat
Metering Centralized metering data for all
services for integration to external
billing
Shared
Service
N/A Ceilometer
Identity Multi-tenant authentication system
that ties to existing stores (e.g.
LDAP) and Image Service
Shared
Service
None Keystone
Image
Management
Upload, download, and manage VM
images for the compute service
Shared
Service
VM Import/
Export
Glance
35
6 month integrated release cycle
Every 6 months, we coordinate and integrate:
Thousands of patches & commits
Across hundreds of developers
And 9 “integrated” or “core” projects
Completely impossible without:
The OpenStack infrastructure team (CI, etc.)
Dedicated PTLs and individual developers
No other similar project does this
36
OpenStack is Well Organized
Qingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1
http://www.qyjohn.net/?p=3120
37
“[the coordinated
releases are] an
indicator that the
OpenStack
project is well
organized in terms
of sub-project
management.”
Stacking It Up
A Quick Note of Thanks
39
These diagrams would not have been
possible without the prior work of:
Ken Pepple, Solinea (@ken_pepple)
Dina Belova, Mirantis
... and the help of several Cloudscalers:
Eric Windisch (@ewindisch)
Joe Gordon (http://github.com/jogo)
Matt Joyce (@openfly, http://www.music-piracy.com)
Dan Sneddon (@dxs)
Joseph Glanville (@jpgvm)
Caveat Emptor
40
The focus for these diagrams was ease of
reading, not accuracy.
See Ken Pepple’s originals or the code if you need truth.
That being said, our team tried really hard for accuracy.
Blame me for any errors.
Architecture Diagrams Legend
41
OpenStack RPC
42
{
'oslo.version': '2',
'oslo.message': json ( {
'method': 'method_name',
'args': { 'keyword': 'value' }
} )
}
nova-api nova-scheduler
Remote Procedure Call
(invoked via (a)synchronous message passing)
OpenStack Compute (Nova)
43
Compute Thoughts
44
Nova still runs best w/ KVM
Do we need another hypervisor? What’s the biz case?
Multiple Availability Zones still not solved
Cells are for making one AZ bigger
complex, tight-coupling
We need a clean sharding mechanism for AZes
also what about Cinder and Quantum?
Integ. to Cinder & Quantum needs rethink
More information needs to be able to be passed back
Compute (Networking) Thoughts
45
nova-network still required
Quantum has been L2 focused & L3 gap still exists
centralized nova-networking is #fail
decentralized is more #fail
nova-conductor security for hypervisor obviated
nova-metadata-api & nova-network on every hypervisor?
security implications
reconciling Quantum and nova-network?
Quantum needs more L3 capability, but ...
OpenStack Block Storage (Cinder)
46
Block Storage Thoughts
47
Default “nova-volume” func. is too minimal
When people think Block Storage service they assume:
Persistent, Network-based, & Performant - it isn’t
Cinder scheduler needs info from Nova
Assuming you want to do anything interesting
Point of lock-in since default isn’t useful
You have to place a bet on a block storage solution
These are expensive, experimental or proprietary
OpenStack Networking (Quantum)
48
Networking Thoughts
49
Default networking functionality is minimal
The APIs have been L2-centric
L3 functionality is same as existed with nova-network
Same architecture, same basic layout, with all of the downsides
Needs a Quantum plugin for full func.
Can’t run more than one plugin at a time per function
Only truly baked plugin is probably Nicira?
Others in process, but it’s not clear how many production deployments there are
Good news is that this area is hot
So hopefully this is unstuck soon
OpenStack Object Storage (Swift)
50
Object Storage Thoughts
51
Swift has continued to lag OpenStack dev
3 yrs on, auditor is slow & does not prioritize replication
Container replication is a bad hack
Ugly stepchild of OpenStack
Keystone authentication woes (integration, performance)
Isn’t universally loved like Nova
Sad, since it was the more mature of the two projects at launch in 2010
Good news: this area has new players
EVault, Seagate, SwiftStack
OpenStack Image Mgmt (Glance)
52
Image Mgmt Thoughts
53
Still not clear why this is standalone project
Really a sub-function of Compute
Semi-pluggable (but not really)
Uses different backends for image storage
To be really useful it needs more features:
P2V, V2V, and other image conversion
Ability to slipstream PV drivers into images
Convert from popular formats: OVF, AMI, etc.
OpenStack Identity (Keystone)
54
Identity Mgmt Thoughts
55
Mixed identity / schizophrenia
Verifies identity, authorization, AND service registry
Service registry is one of many
Nova, Cinder, et al have their own internal registries
Slows everything down
See LivingSocial preso from Folsom Summit
See caching tricks with memcache some projects use
OpenStack needed to reinvent wheel here?
This could have just been LDAP with a schema + caching
OpenStack Dashboard (Horizon)
56
Source: Wikipedia (http://en.wikipedia.org/wiki/Model–view–controller)
Dashboard Thoughts
57
It’s gotten a lot better
Same UI for end-user and admin is bad idea
CloudStack did this and it was a mess
The workflows and views are too different
Security considerations exist
General lag: many things aren’t accessible
e.g. Heat
Need better docs on extending, w/o harm
Customers, product companies, SPs all want to modify
Allow for customizations, while supporting upgrades, etc.
OpenStack Metering (Ceilometer)
58
Metering Thoughts
59
Metering systems are hard
Bad or incomplete data for SPs is existential
Metering system should be *very* baked (is 1yr enough?)
No tokenized meter data
Instance hours not enough
How do you bill for Windows, Oracle, RHEL licenses?
Tokens stack: size of instance, OS and app licenses, etc.
Needs to get flow data from edge switches
Netflow and/or Sflow support for physical switches
Data from the vSwitches is not the best source in the real world
OpenStack Orchestration (Heat)*
60
* Source: http://www.slideshare.net/dbelova/openstack-heat-slides
Orchestration Thoughts
61
Huge potential
Adds additional AWS func: CloudWatch, CloudFormation
Provides clean templates for stacks, which means:
OpenStack on OpenStack (OoO) for testing, etc.
First primary project that rides “on top”
Clear differentiator over other projects
Initiative: Heat templates for Ref Arch
Vendors, customers, etc. could feed to prov systems:
Crowbar, Piston, Cloudscaling, etc.
Strengths:
De facto winner
Incredible community
Unstoppable velocity
Clear innovation curve
SWOT: OpenStack
62
Opportunities:
Build an SQL92 base for cloud
compute (see Threats)
Public cloud compatibility as first
order initiative
vCloud private cloud compatibility
as first order initiative
Weaknesses:
No benevolent dictator
Lack of IaaS experience for many
developers
Interoperability will be difficult
Not impossible, *difficult*
Threats:
Splintering, fragmentation, and
customization
Forking or ivory tower thinking
Stack Gaps
What’s In a Complete Cloud OS?
64
OpenStack
Relationship
Who?
OpenStack
Score
Ecosystem
Score
Vendor
Target*
User Interface(s) Horizon, CLI, ...
OpenStack, Vendor,
Ecosystem 4 2 4
Elastic Resource
Management
Nova, Swift,
Quantum, Cinder, ...
OpenStack, Vendor,
Ecosystem 4 1 4
Service Discovery
Scattered:
Nova, Keystone, ...
OpenStack, Vendor 2 0 4
Authentication,
Authorization, and
Access Controls (AAA)
Keystone (authen/
author), various
projects (ACLs)
OpenStack 2 0 4
HW/SW Life Cycle
Management
N/A Vendor, Ecosystem 1 2 4
Service Management N/A Vendor
0 1 4
Health & Logging N/A Vendor, Ecosystem
0 0 4
Topology & Inventory N/A Vendor
0 0 4
Hardware Certifications N/A
OpenStack, Vendor,
Ecosystem 1 1 2**
* We’re all _trying_ to close this gap ** It’s a hard problem no one will solve individually
Your Basic Choices
Download OpenStack and DIY
OpenStack Distributions
Turn-key Systems
powered by OpenStack
65
1
2
3
Stack Politiks
Types of OpenStack Players
67
Type Description Example
Hardware Vendor
Selling hardware that integrates or supports
OpenStack
Juniper, NetApp, Cisco, EMC
Component Vendor
Point solution, usually software, that
provides subset of OpenStack functionality or
supports it
Midokura, Nexenta
Distribution /
Packager
Basic packaging, some installation/setup,
etc.
RedHat, SUSE, Canonical
Turn-key System
Complete, integrated, OpenStack solution,
with value adds
Cloudscaling, Nebula, Piston
Service Companies
Professional or managed services to
customize or operate OpenStack
Mirantis, Metacloud,
Rackspace Private
Public Clouds Public IaaS HP, Rackspace Public
PaaS / ISVs Value add on top of OpenStack deployments
Scalr, ActiveState (Stackato),
CloudFoundry
Private Clouds Users Wikimedia, AT&T, Yahoo!
components -
compute
components -
storage
systems
Linux distros public clouds private clouds
PaaS / layered ISVsservice companies
components - networkhardware
Who’s Playing the OpenStack Game?
68
components -
compute
components -
storage
systems
Linux distros public clouds private clouds
PaaS / layered ISVsservice companies
components - networkhardware
Player Motivations
69
Sell Hardware
Sell SDN
Software
Sell Storage
Software
Sell HV
Software &
Support
Sell Turn-key
Systems &
Support
Sell Labor (T&M),
Monthly
Management, etc,
Sell Software on
Top of IaaS
Sell Support via
“owning” the
community
Sell Online Cloud
Resources
Use OpenStack for
Business Leverage
Who’s Using It?
First OpenStack Survey
71
414#survey#responses#
16%
7%
8%
4%
11%
17%
37%
More#than#10,000#employees#
5,001#to#10,000#employees#
1,001#to#5,000#employees#
501#to#1,000#employees#
101#to#500#employees#
217100#employees#
1720#employees#
Company Size
Information Technology 60%#
Academic / Research 15%#
Telecommunication 10%#
Industries
Government / Defense 3%#
CC Icons http://vathanx.deviantart.com/
175 29 28 23 18
56countries
Country
124$
100$
77$
151$
Service
Provider
Ecosystem
Vendor
Cloud
Consumer
Cloud
Operator
Type of Involvement
Deployments at a Glance
Type
35#
Hosted#
Private# 15#
Hybrid#
#
37#
Public#
106#
On#Premise#
Private#
Trunk&
8%&
Grizzly&
15%&
Folsom&
47%&
Essex&
25%&
Diablo&
5%&
Version
84
92
94
Production Proof of
Concept
Dev/QA
Stage
134#
94#
94#
89#
66#
46#
Dashboard
Object Storage
Snapshotting to new images
Live Migration
EC2 Compatibility API
S3 Compatibility API
Features
181#
171#
169#
153#
147#
121#
103#
20#
16#
Nova
Glance
Keystone
Horizon
Quantum
Cinder
Swift
Ceilometer
Heat
Components
197&Deployments&
Size of 98 Production Compute Systems
73
1"100$$
52%$
101"500$
18%$ 501"1,000$$
8%$
1,001"5,000$$
8%$
5,000"10,000$$3%$
>$10,000$$6%$
Unspecified$5%$
Other$
30%$
Instances
1"50$$
71%$
51"100$$
8%$
101"500$$
9%$
501"1,000$$2%$
>1,000$$4%$
unspecified$
6%$
Other$
29%$
Nodes
1"100$$
51%$
101"500$$
21%$ 501"1,000$$
4%$
1,001"5,000$$
12%$
5,001"10,000$
3%$
>10,000$$
4%$
unspecified$
5%$
Other$
16%$
Cores
Usage: KVM, LVM, OVS & SQL
74
KVM$
71%$
ESX$
8%$
Xen$
8%$
Xenserver$
5%$
Lxc$5%$
Hyperv$3%$
Other$
29%$
Hypervisors
LVM$
36%$
NFS$
19%$
Ceph$RBD$
13%$ Netapp$
10%$
GlusterFS$
8%$
SAN/HP$5%$
Windows$4%$
EMC$3%$
Solidfire$2%$
Other$
32%$
Open$
Vswitch$
39%$
Linux$
Bridge$
31%$
Cisco$
11%$ HyperUv$
5%$
Nicira$
5%$
Brocade$3%$
Ryu$2%$
big$switch$2%$
NEC$2%$
Other$
19%$
Storage Drivers
Network Drivers
SQL$
55%$
LDAP$
34%$ PAM$
8%$
KVS$
3%$
Other$11%$
Identity Drivers
150$
62$
33$
JSON$ XML$ Both$
API Format
Summary
OpenStack by TKO?
OH: “Finish him!”
We still have work to do
Your participation matters
Regardless of whether you: build, develop, or operate
Get involved
http://is.gd/openstack
76
Q & A
http://simplicityscales.com/
engineering blog
77
Randy Bias
CTO & Co-founder, Cloudscaling
Director, OpenStack Foundation
@randybias
Original Summit Slides are at:
http://engineering.cloudscaling.com/portland13
1 of 77

More Related Content

What's hot(20)

Cloud ComputingCloud Computing
Cloud Computing
NAILBITER595 views
Cloud Computing Technology Overview 2012Cloud Computing Technology Overview 2012
Cloud Computing Technology Overview 2012
Janine Anthony Bowen, Esq. 13.4K views
Application of Cloud ComputingApplication of Cloud Computing
Application of Cloud Computing
Boonlert Aroonpiboon10.5K views
Cloud Workshop - PresentationCloud Workshop - Presentation
Cloud Workshop - Presentation
NAILBITER514 views
Cloud PresentationCloud Presentation
Cloud Presentation
SantaMonicaChamber5.6K views
5 BENEFITS OF HYBRID CLOUD5 BENEFITS OF HYBRID CLOUD
5 BENEFITS OF HYBRID CLOUD
Tyrone Systems1.8K views
AWS LambdaAWS Lambda
AWS Lambda
Alexander Savchuk1.5K views
Cloud Computing Architecture Cloud Computing Architecture
Cloud Computing Architecture
Vasu Jain6.4K views
GREEN CLOUD COMPUTINGGREEN CLOUD COMPUTING
GREEN CLOUD COMPUTING
jauwad syed91 views
Hybrid cloud computing explainedHybrid cloud computing explained
Hybrid cloud computing explained
PMOfficers PMOAcademy2.7K views
OpenStack & business of cloudOpenStack & business of cloud
OpenStack & business of cloud
openstackindia1.4K views
What is cloud computingWhat is cloud computing
What is cloud computing
Dan Morrill1.5K views
Cloud Computing for Enterprise ArchitectsCloud Computing for Enterprise Architects
Cloud Computing for Enterprise Architects
Jean-François Caenen6.6K views

Viewers also liked(20)

Similar to State of the Stack April 2013(20)

State of the Stack v2State of the Stack v2
State of the Stack v2
Randy Bias10.5K views
Intro to OpenStackIntro to OpenStack
Intro to OpenStack
donnieh1379 views
Openstack Pakistan introOpenstack Pakistan intro
Openstack Pakistan intro
Affan Syed474 views
OpenStack for VMware AdministratorsOpenStack for VMware Administrators
OpenStack for VMware Administrators
Trevor Roberts Jr.305 views
OpenStack Ottawa Meetup - March 29th 2017OpenStack Ottawa Meetup - March 29th 2017
OpenStack Ottawa Meetup - March 29th 2017
Stacy Véronneau1.1K views
OpenStack - An OverviewOpenStack - An Overview
OpenStack - An Overview
graziol1.6K views
Open stack presentationOpen stack presentation
Open stack presentation
Frikha Nour209 views
Jonathan Bryce - OpenStackJonathan Bryce - OpenStack
Jonathan Bryce - OpenStack
Cloud Native Day Tel Aviv1.5K views
CloudStack newsCloudStack news
CloudStack news
ShapeBlue1.2K views
VMUG22 Filip Verloy VIOVMUG22 Filip Verloy VIO
VMUG22 Filip Verloy VIO
Filip Verloy753 views
Bird of Feather - OpenStackBird of Feather - OpenStack
Bird of Feather - OpenStack
Khairul Aizat Kamarudzzaman150 views
OpenStack IntroductionOpenStack Introduction
OpenStack Introduction
Roy Gilad2.4K views
OpenStack Winfest2011OpenStack Winfest2011
OpenStack Winfest2011
Open Stack2K views

Recently uploaded(20)

Web Dev - 1 PPT.pdfWeb Dev - 1 PPT.pdf
Web Dev - 1 PPT.pdf
gdsczhcet48 views
Java Platform Approach 1.0 - Picnic MeetupJava Platform Approach 1.0 - Picnic Meetup
Java Platform Approach 1.0 - Picnic Meetup
Rick Ossendrijver23 views

State of the Stack April 2013

  • 1. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution* * All unlicensed or borrowed works retain their original licenses State of the Stack - 2013 Game. Over. OpenStack is The Stack. June 10th, 2013 - SNIA SPDEcon (first edition delivered at OpenStack Summit, April 2013) @randybias
  • 3. Who 3 OpenStack FoundationBoard of Directors Prod. OpenStack pioneer, Cloudscaling: Wins: KT, Internap, LivingSocial, Seagate (EVault), IBS Datafort, major U.S. carriers, & others Part of OpenStack community since July 2010 (launch) Top 10 Cloud Computing Pioneer
  • 4. I run an OpenStack product company I believe the pioneers to emulate are: I have run big data centers 100K+ sq ft, 1,000s of physical servers, 100s of switches My Bias 4
  • 5. 5 1 What is OpenStack? 3 History & Momentum 4 Stackology - a stack taxonomy 5 Stacking it Up - a dive into the projects 6 Stack Gaps - what’s missing? 7 Stack Politics - who’s playing? 9 Summary 2 Why the Success? 8 Who’s using it and how?
  • 7. OpenStack From 10km 7 Networking OPENSTACK CLOUD OPERATING SYSTEM Standard Hardware Compute Storage Your Applications OpenStack Dashboard OpenStack Shared Services APIs
  • 8. OpenStack Mission 8 "To produce the ubiquitous Open Source cloud computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to implement and massively scalable." Code Community
  • 9. OpenStack Foundation Mission 9 The OpenStack Foundation is an independent body providing shared resources to help achieve the OpenStack Mission by Protecting, Empowering, and Promoting OpenStack software and the community around it, including users, developers and the entire ecosystem. The ubiquitous cloud computing platform
  • 10. What it is Some say ... ... it’s an Infra-as-a-Service (IaaS) ... it’s a cloud operating system ... it’s a tool for building private clouds We say it’s “The Stack” ... think Linux ... think Java ... think ubiquitous open source cloud toolkit ... think Game Changer 10
  • 11. The Battle is Over (open src) 11 OpenStack Launch OpenStack CloudStack Eucalyptus OpenNebula Source: trends.google.com
  • 12. Battle is Nearly Over (closed src) 12 OpenStack vSphere vCloud Source: trends.google.com
  • 13. Linux 2000 vs. Linux 2009 13 Is this OpenStack’s Trajectory? Operating system family market share 2000 2009 Unix Linux Linux Unix Source: Linux Magazine http://www.linux-mag.com/id/7749/ Linux UNIX BSD Windows/Other Mixed
  • 14. Fastest Growing Global Open Source Community 14 COMPANIES TOTAL DEVELOPERS AVERAGE MONTHLY CONTRIBUTORS CODE CONTRIBUTIONS 929 245 3,241 189 TOP 10 COUNTRIES 9,000+ INDIVIDUAL MEMBERS “OpenStack appears to be a more advanced or more modern open source project than some of its predecessors because it's a highly coordinated effort.” – Charlie Babcock Information Week COUNTRIES 100 United States, China, India, Great Britain, Australia, France, Russia, Canada, Ireland, Germany
  • 15. 15 Grizzly Stats CONTRIBUTORS PATCHES / DEV NEW DRIVERS TEST CLOUDS DEPLOYED DAILY ~14 15 700 517 (+56%) TOP 10 CONTRIBUTING COMPANIES 7,620 PATCHES SUBMITTED “OpenStack appears to be a more advanced or more modern open source project than some of its predecessors because it's a highly coordinated effort.” – Charlie Babcock Information Week NEW FEATURES 230 Red Hat, Rackspace, IBM, HP, Nebula, Intel, eNovance, VMware, Cloudscaling, DreamHost
  • 17. Who or What Should We Thank? Rackspace for Letting Go OpenStack Foundation & Community Particularly, all of the companies who realized this could be big Hype Curve The OpenStack Infrastructure Team Oslo Project (openstack-common) A thankless job allowing shared code & cleaner projects The Big Enterprises for Driving Interest PTL Generational Shift 17
  • 18. Infrastructure Team Massive Effort -> Improved Quality Gated Commits All Code Has to Jump Through Gates Tempest Test Framework Code Reviews & Continuous Integration Jenkins, Gerrit At scale: jenkins.openstack.org 18 1/4 pages
  • 20. Jul Inaugural Design Summit in Austin 2010 20 OpenStack launches with 25+ partners First ‘Austin’ code release with 35+ partners Oct Nov First public Design Summit in San Antonio Austin OpenStack Object Storage prod OpenStack Compute dev preview Launch! 2010 - The Launch Year Source: Too many to list; blame me for inaccuracies
  • 21. 2011 - Growing Pains & Early Adopters Feb 2nd Summit 21 Rackspace announces plans to launch independent Foundation in 2012 OctApr 3rd Summit (Santa Clara) adds Conference Governance moves forward with project technical leads (PTL), policy board elections (PPB) Jul First Anniversary Bexar OpenStack Compute for mid-size prod OpenStack Image Service added to core Cactus OpenStack Compute for larger- scale prod Sep Diablo Major stability release First 6-mo cycle release 2011 Decision to shift from 3-mo to 6-mo dev cycle Jan 1st Swift Public Cloud Internap w/ Cloudscaling Happy Birthday! 1st Nova Public Cloud Internap w/ Cloudscaling
  • 22. Created framework for Foundation as a community Feb Apr Aug 19 companies announce public support for Foundation Framework & documents ratified by community 22 Drafting committee formed – creating legal documents OpenStack Foundation “officially” launches Sep Essex OpenStack Identity in core OpenStack Dashboard in core Folsom OpenStack Block Storage in core OpenStack Networking in core 2012 May HP Cloud Launch (Beta) Citrix Bails (how’s that going for ya?) Jan AT&T Joins OpenStack Internal production (private) Oct Gartner Report (teeth gnashing followed) Inaugural OpenStack Foundation Board meeting VMware, Intel, & NEC accepted as Gold members Board Elections 2012 - Rise of the Foundation & Prod Deployments
  • 23. 2013/2014 - Breakout Growth Years Apr Oct 23 Q1 Havana OpenStack Metering in integration OpenStack Orchestration in integration LBaaS? “I” Release 2014 Grizzly OpenStack Metering in incubation OpenStack Orchestration in incubation 2013 First Summit 100% run and funded by Foundation First International Summit (APAC?)
  • 24. In every single category, the top 3 vendors support OpenStack Incredible Industry Support 24 top 3 switch vendors top 3 storage vendors top 3 hypervisors top 3 router vendors top 3 blade vendors top 3 linux vendors top 3 x86 vendors
  • 27. Dev Growth by Git Contributors Qingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1 http://www.qyjohn.net/?p=3120 27
  • 28. Accumulated Community Qingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1 http://www.qyjohn.net/?p=3120 28
  • 29. Growth by Domain (company - roughly) Qingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1 http://www.qyjohn.net/?p=3120 29 For the CloudStack projects, influence from Citrix is quite obvious, over 45% of github.com commits come from accounts belonging to citrix.com and cloud.com.
  • 30. 0 750 1500 2250 3000 Austin Santa Clara San Francisco Portland Summit Growth 30
  • 31. Established Marketing Reach OpenStack.org 241k/visits month: Software: 300K downloads Membership: 9000+, Over 90% subscribe to newsletter Relationships with Tier 1 publications and analysts 31 17,693 Followers (+50% from 8/12)
  • 33. OpenStack From 10km 33 Networking OPENSTACK CLOUD OPERATING SYSTEM Standard Hardware Compute Storage Your Applications OpenStack Dashboard OpenStack Shared Services APIs
  • 34. CLI toolsDashboard Other tools ComputeNetworkingOrchestration Hypervisor(s) Queuing Database / KVS / Cache External Block Provider Physical Network Provider Provisioning Log Aggregation Health Monitoring etc. REST Meter Data REST SQL, etc. Varies Varies Varies AMQP/0MQ Topology & Metadata Metering REST REST REST DNS Image Management Identity Management REST Block Storage Object Storage OpenStack (m)Architecture Slide 34 UI Layer Elastic Services Layer Otherstuff,youprobablyneed/want Data & Resource Layer Shared Services Layer
  • 35. Project Name Description Layer AWS Equivalent Codename Dashboard Self-service, role-based web interface for users and administrators UI Console Horizon Compute Provision and manage large pools of on-demand computing resources Elastic Service EC2 Nova Block Storage Volumes on commodity storage gear, and drivers for turn-key block storage solutions Elastic Service EBS Cinder Object Storage Petabytes of reliable storage on standard gear Elastic Service S3 Swift Networking L2-focused on-demand networking with some L3 capabilities Elastic Service VPC Quantum Orchestration Application orchestration layer that runs on top of and manages OpenStack Compute Elastic Service CloudFormation, CloudWatch Heat Metering Centralized metering data for all services for integration to external billing Shared Service N/A Ceilometer Identity Multi-tenant authentication system that ties to existing stores (e.g. LDAP) and Image Service Shared Service None Keystone Image Management Upload, download, and manage VM images for the compute service Shared Service VM Import/ Export Glance 35
  • 36. 6 month integrated release cycle Every 6 months, we coordinate and integrate: Thousands of patches & commits Across hundreds of developers And 9 “integrated” or “core” projects Completely impossible without: The OpenStack infrastructure team (CI, etc.) Dedicated PTLs and individual developers No other similar project does this 36
  • 37. OpenStack is Well Organized Qingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1 http://www.qyjohn.net/?p=3120 37 “[the coordinated releases are] an indicator that the OpenStack project is well organized in terms of sub-project management.”
  • 39. A Quick Note of Thanks 39 These diagrams would not have been possible without the prior work of: Ken Pepple, Solinea (@ken_pepple) Dina Belova, Mirantis ... and the help of several Cloudscalers: Eric Windisch (@ewindisch) Joe Gordon (http://github.com/jogo) Matt Joyce (@openfly, http://www.music-piracy.com) Dan Sneddon (@dxs) Joseph Glanville (@jpgvm)
  • 40. Caveat Emptor 40 The focus for these diagrams was ease of reading, not accuracy. See Ken Pepple’s originals or the code if you need truth. That being said, our team tried really hard for accuracy. Blame me for any errors.
  • 42. OpenStack RPC 42 { 'oslo.version': '2', 'oslo.message': json ( { 'method': 'method_name', 'args': { 'keyword': 'value' } } ) } nova-api nova-scheduler Remote Procedure Call (invoked via (a)synchronous message passing)
  • 44. Compute Thoughts 44 Nova still runs best w/ KVM Do we need another hypervisor? What’s the biz case? Multiple Availability Zones still not solved Cells are for making one AZ bigger complex, tight-coupling We need a clean sharding mechanism for AZes also what about Cinder and Quantum? Integ. to Cinder & Quantum needs rethink More information needs to be able to be passed back
  • 45. Compute (Networking) Thoughts 45 nova-network still required Quantum has been L2 focused & L3 gap still exists centralized nova-networking is #fail decentralized is more #fail nova-conductor security for hypervisor obviated nova-metadata-api & nova-network on every hypervisor? security implications reconciling Quantum and nova-network? Quantum needs more L3 capability, but ...
  • 46. OpenStack Block Storage (Cinder) 46
  • 47. Block Storage Thoughts 47 Default “nova-volume” func. is too minimal When people think Block Storage service they assume: Persistent, Network-based, & Performant - it isn’t Cinder scheduler needs info from Nova Assuming you want to do anything interesting Point of lock-in since default isn’t useful You have to place a bet on a block storage solution These are expensive, experimental or proprietary
  • 49. Networking Thoughts 49 Default networking functionality is minimal The APIs have been L2-centric L3 functionality is same as existed with nova-network Same architecture, same basic layout, with all of the downsides Needs a Quantum plugin for full func. Can’t run more than one plugin at a time per function Only truly baked plugin is probably Nicira? Others in process, but it’s not clear how many production deployments there are Good news is that this area is hot So hopefully this is unstuck soon
  • 51. Object Storage Thoughts 51 Swift has continued to lag OpenStack dev 3 yrs on, auditor is slow & does not prioritize replication Container replication is a bad hack Ugly stepchild of OpenStack Keystone authentication woes (integration, performance) Isn’t universally loved like Nova Sad, since it was the more mature of the two projects at launch in 2010 Good news: this area has new players EVault, Seagate, SwiftStack
  • 52. OpenStack Image Mgmt (Glance) 52
  • 53. Image Mgmt Thoughts 53 Still not clear why this is standalone project Really a sub-function of Compute Semi-pluggable (but not really) Uses different backends for image storage To be really useful it needs more features: P2V, V2V, and other image conversion Ability to slipstream PV drivers into images Convert from popular formats: OVF, AMI, etc.
  • 55. Identity Mgmt Thoughts 55 Mixed identity / schizophrenia Verifies identity, authorization, AND service registry Service registry is one of many Nova, Cinder, et al have their own internal registries Slows everything down See LivingSocial preso from Folsom Summit See caching tricks with memcache some projects use OpenStack needed to reinvent wheel here? This could have just been LDAP with a schema + caching
  • 56. OpenStack Dashboard (Horizon) 56 Source: Wikipedia (http://en.wikipedia.org/wiki/Model–view–controller)
  • 57. Dashboard Thoughts 57 It’s gotten a lot better Same UI for end-user and admin is bad idea CloudStack did this and it was a mess The workflows and views are too different Security considerations exist General lag: many things aren’t accessible e.g. Heat Need better docs on extending, w/o harm Customers, product companies, SPs all want to modify Allow for customizations, while supporting upgrades, etc.
  • 59. Metering Thoughts 59 Metering systems are hard Bad or incomplete data for SPs is existential Metering system should be *very* baked (is 1yr enough?) No tokenized meter data Instance hours not enough How do you bill for Windows, Oracle, RHEL licenses? Tokens stack: size of instance, OS and app licenses, etc. Needs to get flow data from edge switches Netflow and/or Sflow support for physical switches Data from the vSwitches is not the best source in the real world
  • 60. OpenStack Orchestration (Heat)* 60 * Source: http://www.slideshare.net/dbelova/openstack-heat-slides
  • 61. Orchestration Thoughts 61 Huge potential Adds additional AWS func: CloudWatch, CloudFormation Provides clean templates for stacks, which means: OpenStack on OpenStack (OoO) for testing, etc. First primary project that rides “on top” Clear differentiator over other projects Initiative: Heat templates for Ref Arch Vendors, customers, etc. could feed to prov systems: Crowbar, Piston, Cloudscaling, etc.
  • 62. Strengths: De facto winner Incredible community Unstoppable velocity Clear innovation curve SWOT: OpenStack 62 Opportunities: Build an SQL92 base for cloud compute (see Threats) Public cloud compatibility as first order initiative vCloud private cloud compatibility as first order initiative Weaknesses: No benevolent dictator Lack of IaaS experience for many developers Interoperability will be difficult Not impossible, *difficult* Threats: Splintering, fragmentation, and customization Forking or ivory tower thinking
  • 64. What’s In a Complete Cloud OS? 64 OpenStack Relationship Who? OpenStack Score Ecosystem Score Vendor Target* User Interface(s) Horizon, CLI, ... OpenStack, Vendor, Ecosystem 4 2 4 Elastic Resource Management Nova, Swift, Quantum, Cinder, ... OpenStack, Vendor, Ecosystem 4 1 4 Service Discovery Scattered: Nova, Keystone, ... OpenStack, Vendor 2 0 4 Authentication, Authorization, and Access Controls (AAA) Keystone (authen/ author), various projects (ACLs) OpenStack 2 0 4 HW/SW Life Cycle Management N/A Vendor, Ecosystem 1 2 4 Service Management N/A Vendor 0 1 4 Health & Logging N/A Vendor, Ecosystem 0 0 4 Topology & Inventory N/A Vendor 0 0 4 Hardware Certifications N/A OpenStack, Vendor, Ecosystem 1 1 2** * We’re all _trying_ to close this gap ** It’s a hard problem no one will solve individually
  • 65. Your Basic Choices Download OpenStack and DIY OpenStack Distributions Turn-key Systems powered by OpenStack 65 1 2 3
  • 67. Types of OpenStack Players 67 Type Description Example Hardware Vendor Selling hardware that integrates or supports OpenStack Juniper, NetApp, Cisco, EMC Component Vendor Point solution, usually software, that provides subset of OpenStack functionality or supports it Midokura, Nexenta Distribution / Packager Basic packaging, some installation/setup, etc. RedHat, SUSE, Canonical Turn-key System Complete, integrated, OpenStack solution, with value adds Cloudscaling, Nebula, Piston Service Companies Professional or managed services to customize or operate OpenStack Mirantis, Metacloud, Rackspace Private Public Clouds Public IaaS HP, Rackspace Public PaaS / ISVs Value add on top of OpenStack deployments Scalr, ActiveState (Stackato), CloudFoundry Private Clouds Users Wikimedia, AT&T, Yahoo!
  • 68. components - compute components - storage systems Linux distros public clouds private clouds PaaS / layered ISVsservice companies components - networkhardware Who’s Playing the OpenStack Game? 68
  • 69. components - compute components - storage systems Linux distros public clouds private clouds PaaS / layered ISVsservice companies components - networkhardware Player Motivations 69 Sell Hardware Sell SDN Software Sell Storage Software Sell HV Software & Support Sell Turn-key Systems & Support Sell Labor (T&M), Monthly Management, etc, Sell Software on Top of IaaS Sell Support via “owning” the community Sell Online Cloud Resources Use OpenStack for Business Leverage
  • 71. First OpenStack Survey 71 414#survey#responses# 16% 7% 8% 4% 11% 17% 37% More#than#10,000#employees# 5,001#to#10,000#employees# 1,001#to#5,000#employees# 501#to#1,000#employees# 101#to#500#employees# 217100#employees# 1720#employees# Company Size Information Technology 60%# Academic / Research 15%# Telecommunication 10%# Industries Government / Defense 3%# CC Icons http://vathanx.deviantart.com/ 175 29 28 23 18 56countries Country 124$ 100$ 77$ 151$ Service Provider Ecosystem Vendor Cloud Consumer Cloud Operator Type of Involvement
  • 72. Deployments at a Glance Type 35# Hosted# Private# 15# Hybrid# # 37# Public# 106# On#Premise# Private# Trunk& 8%& Grizzly& 15%& Folsom& 47%& Essex& 25%& Diablo& 5%& Version 84 92 94 Production Proof of Concept Dev/QA Stage 134# 94# 94# 89# 66# 46# Dashboard Object Storage Snapshotting to new images Live Migration EC2 Compatibility API S3 Compatibility API Features 181# 171# 169# 153# 147# 121# 103# 20# 16# Nova Glance Keystone Horizon Quantum Cinder Swift Ceilometer Heat Components 197&Deployments&
  • 73. Size of 98 Production Compute Systems 73 1"100$$ 52%$ 101"500$ 18%$ 501"1,000$$ 8%$ 1,001"5,000$$ 8%$ 5,000"10,000$$3%$ >$10,000$$6%$ Unspecified$5%$ Other$ 30%$ Instances 1"50$$ 71%$ 51"100$$ 8%$ 101"500$$ 9%$ 501"1,000$$2%$ >1,000$$4%$ unspecified$ 6%$ Other$ 29%$ Nodes 1"100$$ 51%$ 101"500$$ 21%$ 501"1,000$$ 4%$ 1,001"5,000$$ 12%$ 5,001"10,000$ 3%$ >10,000$$ 4%$ unspecified$ 5%$ Other$ 16%$ Cores
  • 74. Usage: KVM, LVM, OVS & SQL 74 KVM$ 71%$ ESX$ 8%$ Xen$ 8%$ Xenserver$ 5%$ Lxc$5%$ Hyperv$3%$ Other$ 29%$ Hypervisors LVM$ 36%$ NFS$ 19%$ Ceph$RBD$ 13%$ Netapp$ 10%$ GlusterFS$ 8%$ SAN/HP$5%$ Windows$4%$ EMC$3%$ Solidfire$2%$ Other$ 32%$ Open$ Vswitch$ 39%$ Linux$ Bridge$ 31%$ Cisco$ 11%$ HyperUv$ 5%$ Nicira$ 5%$ Brocade$3%$ Ryu$2%$ big$switch$2%$ NEC$2%$ Other$ 19%$ Storage Drivers Network Drivers SQL$ 55%$ LDAP$ 34%$ PAM$ 8%$ KVS$ 3%$ Other$11%$ Identity Drivers 150$ 62$ 33$ JSON$ XML$ Both$ API Format
  • 76. OpenStack by TKO? OH: “Finish him!” We still have work to do Your participation matters Regardless of whether you: build, develop, or operate Get involved http://is.gd/openstack 76
  • 77. Q & A http://simplicityscales.com/ engineering blog 77 Randy Bias CTO & Co-founder, Cloudscaling Director, OpenStack Foundation @randybias Original Summit Slides are at: http://engineering.cloudscaling.com/portland13