State of the Stack April 2013
Upcoming SlideShare
Loading in...5
×
 

State of the Stack April 2013

on

  • 77,340 views

A comprehensive review of OpenStack then and now, each project's architecture, and hard data on why the race for open cloud is over. (First edition delivered April 2013 at OpenStack Summit. This ...

A comprehensive review of OpenStack then and now, each project's architecture, and hard data on why the race for open cloud is over. (First edition delivered April 2013 at OpenStack Summit. This version is from SPDEcon on June 10, 2013.)

Statistics

Views

Total Views
77,340
Views on SlideShare
50,164
Embed Views
27,176

Actions

Likes
53
Downloads
865
Comments
5

140 Embeds 27,176

http://techcrunch.com 12657
http://www.google.com 3297
http://engineering.cloudscaling.com 2980
http://www.techcrunch.com 2757
http://blog.zenoss.com 2572
http://blog.cohesiveft.com 351
https://twitter.com 342
http://diversity.net.nz 255
http://cloudarchitectmusings.com 250
http://feeds.feedburner.com 215
http://www-ig-opensocial.googleusercontent.com 204
http://feedspot.com 177
http://www.kavistechnology.com 128
http://www.newsblur.com 92
http://www.scoop.it 80
http://tspace.web.att.com 77
http://www.pulse.me 57
http://newsblur.com 45
http://dev.newsblur.com 42
http://localhost 39
http://server.local 38
http://new.openspectruminc.com 37
http://engineering.cloudscaling.com.dev 28
http://blog.openspectruminc.com 26
http://flavors.me 22
http://wiki.swissre.com 20
http://www.hanrss.com 20
http://107.20.36.58 18
http://www.thatdrew.com 16
http://irq.tumblr.com 15
https://d2si.bluekiwi.net 14
https://abs.twimg.com 13
http://www.linkedin.com 12
http://www.plurk.com 12
https://confluence.coremedia.com 11
http://moderation.local 10
http://3216052632588610042_c6e9bfc7a7cc65b9e42dd99d71f9f3e11784d86b.blogspot.com 10
http://lector.johndong.com 8
http://feedproxy.google.com 8
http://www.bizboxusa.com 8
http://digg.com 8
http://ebulletin.collected.info 7
http://internetaddict.collected.info 6
http://cloud.feedly.com 6
http://tekdef.com 6
http://sadani.feedspot.com 5
http://l.lj-toys.com 5
http://www.mattoxaz.com 5
http://tech.findfollowtweet.com 5
http://translate.googleusercontent.com 5
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NoDerivs LicenseCC Attribution-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

15 of 5 Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    State of the Stack April 2013 State of the Stack April 2013 Presentation Transcript

    • CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution** All unlicensed or borrowed works retain their original licensesState of the Stack - 2013Game. Over. OpenStack is The Stack.June 10th, 2013 - SNIA SPDEcon(first edition delivered at OpenStack Summit, April 2013)@randybias
    • Introduction
    • Who3OpenStack FoundationBoard of DirectorsProd. OpenStack pioneer, Cloudscaling:Wins: KT, Internap, LivingSocial, Seagate (EVault),IBS Datafort, major U.S. carriers, & othersPart of OpenStack community since July 2010 (launch)Top 10 Cloud Computing Pioneer
    • I run an OpenStack product companyI believe the pioneers to emulate are:I have run big data centers100K+ sq ft, 1,000s of physical servers, 100s of switchesMy Bias4
    • 51 What is OpenStack?3 History & Momentum4 Stackology - a stack taxonomy5 Stacking it Up - a dive into the projects6 Stack Gaps - what’s missing?7 Stack Politics - who’s playing?9 Summary2 Why the Success?8 Who’s using it and how?
    • What is OpenStack?
    • OpenStack From 10km7NetworkingOPENSTACKCLOUD OPERATING SYSTEMStandard HardwareCompute StorageYour ApplicationsOpenStack DashboardOpenStack Shared ServicesAPIs
    • OpenStack Mission8"To produce the ubiquitous Open Source cloud computingplatform that will meet the needs of public and privatecloud providers regardless of size, by being simple toimplement and massively scalable."Code Community
    • OpenStack Foundation Mission9The OpenStack Foundation is an independent body providing sharedresources to help achieve the OpenStack Mission by Protecting,Empowering, and Promoting OpenStack software and thecommunity around it, including users, developers and the entireecosystem.The ubiquitous cloudcomputing platform
    • What it isSome say ...... it’s an Infra-as-a-Service (IaaS)... it’s a cloud operating system... it’s a tool for building private cloudsWe say it’s “The Stack”... think Linux... think Java... think ubiquitous open source cloud toolkit... think Game Changer10
    • The Battle is Over (open src)11OpenStack LaunchOpenStack CloudStack Eucalyptus OpenNebulaSource: trends.google.com
    • Battle is Nearly Over (closed src)12OpenStack vSphere vCloudSource: trends.google.com
    • Linux 2000 vs. Linux 200913Is this OpenStack’s Trajectory?Operating system family market share2000 2009Unix LinuxLinux UnixSource: Linux Magazinehttp://www.linux-mag.com/id/7749/LinuxUNIXBSDWindows/OtherMixed
    • Fastest Growing Global Open Source Community14COMPANIESTOTAL DEVELOPERS AVERAGE MONTHLYCONTRIBUTORSCODE CONTRIBUTIONS929 245 3,241189 TOP 10 COUNTRIES9,000+INDIVIDUAL MEMBERS“OpenStack appears to be a more advanced or more modern open source project thansome of its predecessors because its a highly coordinated effort.”– Charlie BabcockInformation WeekCOUNTRIES100United States, China, India,Great Britain, Australia, France,Russia, Canada, Ireland, Germany
    • 15Grizzly StatsCONTRIBUTORSPATCHES / DEV NEW DRIVERS TEST CLOUDS DEPLOYED DAILY~14 15 700517 (+56%)TOP 10 CONTRIBUTING COMPANIES7,620PATCHES SUBMITTED“OpenStack appears to be a more advanced or more modern open source project thansome of its predecessors because its a highly coordinated effort.”– Charlie BabcockInformation WeekNEW FEATURES230Red Hat, Rackspace, IBM, HP, Nebula,Intel, eNovance, VMware, Cloudscaling,DreamHost
    • Why the Success?
    • Who or What Should We Thank?Rackspace for Letting GoOpenStack Foundation & CommunityParticularly, all of the companies who realized this could be bigHype CurveThe OpenStack Infrastructure TeamOslo Project (openstack-common)A thankless job allowing shared code & cleaner projectsThe Big Enterprises for Driving InterestPTL Generational Shift17
    • Infrastructure TeamMassive Effort -> Improved QualityGated CommitsAll Code Has to Jump Through GatesTempest Test FrameworkCode Reviews & ContinuousIntegrationJenkins, GerritAt scale: jenkins.openstack.org181/4 pages
    • History &Momentum
    • JulInauguralDesignSummit inAustin201020OpenStacklauncheswith 25+partnersFirst ‘Austin’code releasewith 35+partnersOct NovFirst publicDesignSummit inSan AntonioAustinOpenStack Object Storage prodOpenStack Compute dev previewLaunch!2010 - The Launch YearSource: Too many to list; blame me for inaccuracies
    • 2011 - Growing Pains & Early AdoptersFeb2nd Summit21Rackspaceannounces plansto launchindependentFoundation in2012OctApr3rd Summit(Santa Clara)addsConferenceGovernance moves forward withproject technical leads (PTL),policy board elections (PPB)JulFirstAnniversaryBexarOpenStack Compute for mid-size prodOpenStack Image Service added to coreCactusOpenStack Compute for larger-scale prodSepDiabloMajor stability releaseFirst 6-mo cycle release2011Decision to shift from 3-mo to6-mo dev cycleJan1st SwiftPublic CloudInternap w/ CloudscalingHappyBirthday!1st NovaPublic CloudInternap w/ Cloudscaling
    • Createdframework forFoundation asa communityFeb Apr Aug19 companiesannouncepublic supportfor FoundationFramework &documents ratifiedby community22Drafting committeeformed – creatinglegal documentsOpenStackFoundation“officially”launchesSepEssexOpenStack Identity in coreOpenStack Dashboard in coreFolsomOpenStack BlockStorage in coreOpenStackNetworking in core2012MayHPCloudLaunch(Beta)Citrix Bails(how’s that going for ya?)JanAT&T JoinsOpenStackInternal production (private)OctGartnerReport(teeth gnashing followed)Inaugural OpenStackFoundation BoardmeetingVMware, Intel, & NECaccepted as Gold membersBoardElections2012 - Rise of the Foundation & Prod Deployments
    • 2013/2014 - Breakout Growth YearsApr Oct23Q1HavanaOpenStack Metering in integrationOpenStack Orchestration in integrationLBaaS?“I” Release2014GrizzlyOpenStack Metering in incubationOpenStack Orchestration in incubation2013First Summit100% run andfunded byFoundationFirstInternationalSummit(APAC?)
    • In every single category, thetop 3 vendors support OpenStackIncredible Industry Support24top 3 switch vendors top 3 storage vendors top 3 hypervisorstop 3 router vendors top 3 blade vendors top 3 linux vendorstop 3 x86 vendors
    • Developer Growth25Contributors per month (ohloh)
    • Developer Growth ComparisonContributors per month (ohloh)26
    • Dev Growth by Git ContributorsQingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1http://www.qyjohn.net/?p=312027
    • Accumulated CommunityQingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1http://www.qyjohn.net/?p=312028
    • Growth by Domain (company - roughly)Qingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1http://www.qyjohn.net/?p=312029For the CloudStackprojects, influence fromCitrix is quite obvious, over45% of github.comcommits come fromaccounts belonging tocitrix.com and cloud.com.
    • 0750150022503000Austin Santa Clara San Francisco PortlandSummit Growth30
    • Established Marketing ReachOpenStack.org 241k/visits month:Software: 300K downloadsMembership: 9000+, Over 90% subscribe to newsletterRelationships with Tier 1 publications and analysts3117,693Followers(+50% from 8/12)
    • Stackology
    • OpenStack From 10km33NetworkingOPENSTACKCLOUD OPERATING SYSTEMStandard HardwareCompute StorageYour ApplicationsOpenStack DashboardOpenStack Shared ServicesAPIs
    • CLI toolsDashboard Other toolsComputeNetworkingOrchestrationHypervisor(s) QueuingDatabase/ KVS/ CacheExternalBlockProviderPhysicalNetworkProviderProvisioningLogAggregationHealthMonitoringetc.REST Meter DataRESTSQL, etc. Varies Varies Varies AMQP/0MQTopology &MetadataMeteringRESTREST RESTDNSImageManagementIdentityManagementRESTBlockStorageObjectStorageOpenStack (m)Architecture Slide34UI LayerElasticServicesLayerOtherstuff,youprobablyneed/wantData &ResourceLayerSharedServicesLayer
    • ProjectNameDescription LayerAWSEquivalentCodenameDashboard Self-service, role-based webinterface for users and administratorsUI Console HorizonCompute Provision and manage large pools ofon-demand computing resourcesElasticServiceEC2 NovaBlock Storage Volumes on commodity storage gear,and drivers for turn-key block storagesolutionsElasticServiceEBS CinderObjectStoragePetabytes of reliable storage onstandard gearElasticServiceS3 SwiftNetworking L2-focused on-demand networkingwith some L3 capabilitiesElasticServiceVPC QuantumOrchestration Application orchestration layer thatruns on top of and managesOpenStack ComputeElasticServiceCloudFormation,CloudWatchHeatMetering Centralized metering data for allservices for integration to externalbillingSharedServiceN/A CeilometerIdentity Multi-tenant authentication systemthat ties to existing stores (e.g.LDAP) and Image ServiceSharedServiceNone KeystoneImageManagementUpload, download, and manage VMimages for the compute serviceSharedServiceVM Import/ExportGlance35
    • 6 month integrated release cycleEvery 6 months, we coordinate and integrate:Thousands of patches & commitsAcross hundreds of developersAnd 9 “integrated” or “core” projectsCompletely impossible without:The OpenStack infrastructure team (CI, etc.)Dedicated PTLs and individual developersNo other similar project does this36
    • OpenStack is Well OrganizedQingye Jiang (John) - Open Source IaaS Community Analysis CY13 - Q1http://www.qyjohn.net/?p=312037“[the coordinatedreleases are] anindicator that theOpenStackproject is wellorganized in termsof sub-projectmanagement.”
    • Stacking It Up
    • A Quick Note of Thanks39These diagrams would not have beenpossible without the prior work of:Ken Pepple, Solinea (@ken_pepple)Dina Belova, Mirantis... and the help of several Cloudscalers:Eric Windisch (@ewindisch)Joe Gordon (http://github.com/jogo)Matt Joyce (@openfly, http://www.music-piracy.com)Dan Sneddon (@dxs)Joseph Glanville (@jpgvm)
    • Caveat Emptor40The focus for these diagrams was ease ofreading, not accuracy.See Ken Pepple’s originals or the code if you need truth.That being said, our team tried really hard for accuracy.Blame me for any errors.
    • Architecture Diagrams Legend41
    • OpenStack RPC42{oslo.version: 2,oslo.message: json ( {method: method_name,args: { keyword: value }} )}nova-api nova-schedulerRemote Procedure Call(invoked via (a)synchronous message passing)
    • OpenStack Compute (Nova)43
    • Compute Thoughts44Nova still runs best w/ KVMDo we need another hypervisor? What’s the biz case?Multiple Availability Zones still not solvedCells are for making one AZ biggercomplex, tight-couplingWe need a clean sharding mechanism for AZesalso what about Cinder and Quantum?Integ. to Cinder & Quantum needs rethinkMore information needs to be able to be passed back
    • Compute (Networking) Thoughts45nova-network still requiredQuantum has been L2 focused & L3 gap still existscentralized nova-networking is #faildecentralized is more #failnova-conductor security for hypervisor obviatednova-metadata-api & nova-network on every hypervisor?security implicationsreconciling Quantum and nova-network?Quantum needs more L3 capability, but ...
    • OpenStack Block Storage (Cinder)46
    • Block Storage Thoughts47Default “nova-volume” func. is too minimalWhen people think Block Storage service they assume:Persistent, Network-based, & Performant - it isn’tCinder scheduler needs info from NovaAssuming you want to do anything interestingPoint of lock-in since default isn’t usefulYou have to place a bet on a block storage solutionThese are expensive, experimental or proprietary
    • OpenStack Networking (Quantum)48
    • Networking Thoughts49Default networking functionality is minimalThe APIs have been L2-centricL3 functionality is same as existed with nova-networkSame architecture, same basic layout, with all of the downsidesNeeds a Quantum plugin for full func.Can’t run more than one plugin at a time per functionOnly truly baked plugin is probably Nicira?Others in process, but it’s not clear how many production deployments there areGood news is that this area is hotSo hopefully this is unstuck soon
    • OpenStack Object Storage (Swift)50
    • Object Storage Thoughts51Swift has continued to lag OpenStack dev3 yrs on, auditor is slow & does not prioritize replicationContainer replication is a bad hackUgly stepchild of OpenStackKeystone authentication woes (integration, performance)Isn’t universally loved like NovaSad, since it was the more mature of the two projects at launch in 2010Good news: this area has new playersEVault, Seagate, SwiftStack
    • OpenStack Image Mgmt (Glance)52
    • Image Mgmt Thoughts53Still not clear why this is standalone projectReally a sub-function of ComputeSemi-pluggable (but not really)Uses different backends for image storageTo be really useful it needs more features:P2V, V2V, and other image conversionAbility to slipstream PV drivers into imagesConvert from popular formats: OVF, AMI, etc.
    • OpenStack Identity (Keystone)54
    • Identity Mgmt Thoughts55Mixed identity / schizophreniaVerifies identity, authorization, AND service registryService registry is one of manyNova, Cinder, et al have their own internal registriesSlows everything downSee LivingSocial preso from Folsom SummitSee caching tricks with memcache some projects useOpenStack needed to reinvent wheel here?This could have just been LDAP with a schema + caching
    • OpenStack Dashboard (Horizon)56Source: Wikipedia (http://en.wikipedia.org/wiki/Model–view–controller)
    • Dashboard Thoughts57It’s gotten a lot betterSame UI for end-user and admin is bad ideaCloudStack did this and it was a messThe workflows and views are too differentSecurity considerations existGeneral lag: many things aren’t accessiblee.g. HeatNeed better docs on extending, w/o harmCustomers, product companies, SPs all want to modifyAllow for customizations, while supporting upgrades, etc.
    • OpenStack Metering (Ceilometer)58
    • Metering Thoughts59Metering systems are hardBad or incomplete data for SPs is existentialMetering system should be *very* baked (is 1yr enough?)No tokenized meter dataInstance hours not enoughHow do you bill for Windows, Oracle, RHEL licenses?Tokens stack: size of instance, OS and app licenses, etc.Needs to get flow data from edge switchesNetflow and/or Sflow support for physical switchesData from the vSwitches is not the best source in the real world
    • OpenStack Orchestration (Heat)*60* Source: http://www.slideshare.net/dbelova/openstack-heat-slides
    • Orchestration Thoughts61Huge potentialAdds additional AWS func: CloudWatch, CloudFormationProvides clean templates for stacks, which means:OpenStack on OpenStack (OoO) for testing, etc.First primary project that rides “on top”Clear differentiator over other projectsInitiative: Heat templates for Ref ArchVendors, customers, etc. could feed to prov systems:Crowbar, Piston, Cloudscaling, etc.
    • Strengths:De facto winnerIncredible communityUnstoppable velocityClear innovation curveSWOT: OpenStack62Opportunities:Build an SQL92 base for cloudcompute (see Threats)Public cloud compatibility as firstorder initiativevCloud private cloud compatibilityas first order initiativeWeaknesses:No benevolent dictatorLack of IaaS experience for manydevelopersInteroperability will be difficultNot impossible, *difficult*Threats:Splintering, fragmentation, andcustomizationForking or ivory tower thinking
    • Stack Gaps
    • What’s In a Complete Cloud OS?64OpenStackRelationshipWho?OpenStackScoreEcosystemScoreVendorTarget*User Interface(s) Horizon, CLI, ...OpenStack, Vendor,Ecosystem 4 2 4Elastic ResourceManagementNova, Swift,Quantum, Cinder, ...OpenStack, Vendor,Ecosystem 4 1 4Service DiscoveryScattered:Nova, Keystone, ...OpenStack, Vendor 2 0 4Authentication,Authorization, andAccess Controls (AAA)Keystone (authen/author), variousprojects (ACLs)OpenStack 2 0 4HW/SW Life CycleManagementN/A Vendor, Ecosystem 1 2 4Service Management N/A Vendor0 1 4Health & Logging N/A Vendor, Ecosystem0 0 4Topology & Inventory N/A Vendor0 0 4Hardware Certifications N/AOpenStack, Vendor,Ecosystem 1 1 2*** We’re all _trying_ to close this gap ** It’s a hard problem no one will solve individually
    • Your Basic ChoicesDownload OpenStack and DIYOpenStack DistributionsTurn-key Systemspowered by OpenStack65123
    • Stack Politiks
    • Types of OpenStack Players67Type Description ExampleHardware VendorSelling hardware that integrates or supportsOpenStackJuniper, NetApp, Cisco, EMCComponent VendorPoint solution, usually software, thatprovides subset of OpenStack functionality orsupports itMidokura, NexentaDistribution /PackagerBasic packaging, some installation/setup,etc.RedHat, SUSE, CanonicalTurn-key SystemComplete, integrated, OpenStack solution,with value addsCloudscaling, Nebula, PistonService CompaniesProfessional or managed services tocustomize or operate OpenStackMirantis, Metacloud,Rackspace PrivatePublic Clouds Public IaaS HP, Rackspace PublicPaaS / ISVs Value add on top of OpenStack deploymentsScalr, ActiveState (Stackato),CloudFoundryPrivate Clouds Users Wikimedia, AT&T, Yahoo!
    • components -computecomponents -storagesystemsLinux distros public clouds private cloudsPaaS / layered ISVsservice companiescomponents - networkhardwareWho’s Playing the OpenStack Game?68
    • components -computecomponents -storagesystemsLinux distros public clouds private cloudsPaaS / layered ISVsservice companiescomponents - networkhardwarePlayer Motivations69Sell HardwareSell SDNSoftwareSell StorageSoftwareSell HVSoftware &SupportSell Turn-keySystems &SupportSell Labor (T&M),MonthlyManagement, etc,Sell Software onTop of IaaSSell Support via“owning” thecommunitySell Online CloudResourcesUse OpenStack forBusiness Leverage
    • Who’s Using It?
    • First OpenStack Survey71414#survey#responses#16%7%8%4%11%17%37%More#than#10,000#employees#5,001#to#10,000#employees#1,001#to#5,000#employees#501#to#1,000#employees#101#to#500#employees#217100#employees#1720#employees#Company SizeInformation Technology 60%#Academic / Research 15%#Telecommunication 10%#IndustriesGovernment / Defense 3%#CC Icons http://vathanx.deviantart.com/175 29 28 23 1856countriesCountry124$100$77$151$ServiceProviderEcosystemVendorCloudConsumerCloudOperatorType of Involvement
    • Deployments at a GlanceType35#Hosted#Private# 15#Hybrid##37#Public#106#On#Premise#Private#Trunk&8%&Grizzly&15%&Folsom&47%&Essex&25%&Diablo&5%&Version849294Production Proof ofConceptDev/QAStage134#94#94#89#66#46#DashboardObject StorageSnapshotting to new imagesLive MigrationEC2 Compatibility APIS3 Compatibility APIFeatures181#171#169#153#147#121#103#20#16#NovaGlanceKeystoneHorizonQuantumCinderSwiftCeilometerHeatComponents197&Deployments&
    • Size of 98 Production Compute Systems731"100$$52%$101"500$18%$ 501"1,000$$8%$1,001"5,000$$8%$5,000"10,000$$3%$>$10,000$$6%$Unspecified$5%$Other$30%$Instances1"50$$71%$51"100$$8%$101"500$$9%$501"1,000$$2%$>1,000$$4%$unspecified$6%$Other$29%$Nodes1"100$$51%$101"500$$21%$ 501"1,000$$4%$1,001"5,000$$12%$5,001"10,000$3%$>10,000$$4%$unspecified$5%$Other$16%$Cores
    • Usage: KVM, LVM, OVS & SQL74KVM$71%$ESX$8%$Xen$8%$Xenserver$5%$Lxc$5%$Hyperv$3%$Other$29%$HypervisorsLVM$36%$NFS$19%$Ceph$RBD$13%$ Netapp$10%$GlusterFS$8%$SAN/HP$5%$Windows$4%$EMC$3%$Solidfire$2%$Other$32%$Open$Vswitch$39%$Linux$Bridge$31%$Cisco$11%$ HyperUv$5%$Nicira$5%$Brocade$3%$Ryu$2%$big$switch$2%$NEC$2%$Other$19%$Storage DriversNetwork DriversSQL$55%$LDAP$34%$ PAM$8%$KVS$3%$Other$11%$Identity Drivers150$62$33$JSON$ XML$ Both$API Format
    • Summary
    • OpenStack by TKO?OH: “Finish him!”We still have work to doYour participation mattersRegardless of whether you: build, develop, or operateGet involvedhttp://is.gd/openstack76
    • Q & Ahttp://simplicityscales.com/engineering blog77Randy BiasCTO & Co-founder, CloudscalingDirector, OpenStack Foundation@randybiasOriginal Summit Slides are at:http://engineering.cloudscaling.com/portland13