State of the Stack v4 - OpenStack in All It's Glory


Published on

The almost annual State of the Stack, version 4, an end-to-end view of OpenStack. This edition focuses on what the challenges are within the community and how they can be addressed.

v1 of SOTS has over 90,000 views and is one of the highest viewed OpenStack presentations ever.

Published in: Technology

State of the Stack v4 - OpenStack in All It's Glory

  1. 1. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution* * All unlicensed or borrowed works retain their original licenses SOTS v4 State of the Stack May 20th, 2015 OpenStack Summit, Spring 2015 @randybias With significant help from many Cloudscalers and EMCers. Thank you!
  2. 2. The Randy Bias • Built big clouds; production clouds • An OpenStack Original • part of launch in 2010, on Foundation Board since formation • built some of the largest and earliest OpenStack clouds • Top <insert number here> cloud/twitter/pioneer/visionary • you pick… 2
  3. 3. What Winning Looks Like 3
  4. 4. Fastest Growing Open Src Community 4 COMPANIES TOTAL DEVELOPERS AVERAGE MONTHLY 
 CONTRIBUTORS TOTAL CODE CONTRIBUTIONS 3,654 >600 125,000+ 502 TOP 10 COUNTRIES 27,398 INDIVIDUAL MEMBERS COUNTRIES 140+ United States, India, China, United Kingdom, France, Russia, Canada, Germany, Japan, Australia
  5. 5. Now That’s Something 5 OpenStack CloudStack Eucalyptus VMware vSphere Source:
  6. 6. Active Contributors in The Community 6 Qingye Jiang (John) - Open Source IaaS Community Analysis CY15 - Q1
  7. 7. Monthly Commit Volume 7 Qingye Jiang (John) - Open Source IaaS Community Analysis CY15 - Q1
  8. 8. Accumulated Developers 8 Qingye Jiang (John) - Open Source IaaS Community Analysis CY15 - Q1
  9. 9. 9 Source: LinkedIn Groups via RedMonk analysis AAWS Community Network Azure OpenStack
  10. 10. Why Did We Win? 10 Explosive Growth
  11. 11. What’s Also Very Dangerous? 11 Explosive Growth
  12. 12. Let’s Review 12
  13. 13. 13 Infrastructure as a Service Compute Nova Ironic Magnum Network Neutron (LBaaS) (VPNaaS) (FWaaS) Storage Swift Cinder Manila Cloud Management Telemetry Ceilometer Deployment Triple O Orchestration Heat Test Suites Tempest Rally Advanced Services (Consume IaaS) Image Management: Glance Data Processing Sahara Key Management Barbican DNS Management Designate Database Management Trove Message Queue Zaqar Service Catalog Murano Workflow Management Mistral Policy Management Congress Common/Shared: Identity: Keystone Common Libraries: Oslo User/Admin UI API CLIKilo
  14. 14. OpenStack Interdependence 14
  15. 15. – Me / Ako / Moi / Yo * “OpenStack is at risk of collapsing under its own weight.” 15 *
  16. 16. Lots of Improvements • Product WG formed • Create an aggregation point for longer term planning, bring user feedback into process, prioritization of blueprints, lobbying TC and PTLs for work queues, “funding” of key blueprints, etc. • Integrated release & 6-month cycle reformed to “Big Tent” approach • No more forced 6-month integration, more project autonomy, encouragement of 3rd party integration testing of drivers, tagging for release, etc. 16
  17. 17. Product Working Group 17 User Committee N+3 members: 3 selected by the board, the TC and an additional nominated representative. An additional N members elected by the user community. Enterprise Focused teams to gather user requirements from segments and represent them Telco / OPNFV Application Ecosystem Large Deployments API Working Group Working Groups to address a particular requirement set. These WGs should have a target set of deliverables and conclude when those are met. Maintenance should be a function of the regular workflows. Logging Ops Tools Monitoring HPC Product Working Group Gather requirements from both sets of WGs (Segment and Requirement Oriented) above in the form of user stories, work with cross-project team to populate blueprints from user stories across projects, work to identify developers to help complete blueprints, communicate with project PTLs and core team to collect feedback on future directions, and compile this data into a multi-release roadmap that is publicly available. In summary, facilitate a feedback loop between projects, user community, and working groups. Multi-Release Roadmap New
  18. 18. “Big Tent” Release Cycle Reform 18 Solving for “How do we allow for the additional projects in the future without breaking down?” (Current) Tag Categories: Release Team Tag Description integrated-release Frozen tag, not given to new projects. Identifies projects that were integrated prior to Kilo. release: indepdent Projects with this tag “release as needed” and don’t have to coordinate with other projects. release: at-6mo-cycle-end Projects that commit to being a part of a coordinated release every 6 months. They can still have intermediate releases independent of the 6 month cycle “final” release. release: has-stable-branches Projects that have stable branches (from the last release in the cycle) release: managed Projects that agree to follow the processes/timelines outlined by the OpenStack Release Management Team team: diverse-affiliation This tag shows that the developer team for the project is from a diverse set of organizations (1 < 50% and 2 < 80%). This is tested every 6 months. Details at (Current) Tags:
  19. 19. It’s not enough … 19
  20. 20. Complexity Kills 20
  21. 21. Enterprise Cloud Maturity 21 Most enterprises in the “thick middle” of maturity cycle
  22. 22. The “Thick Middle” Struggling 22
  23. 23. Docker is Simpler .. And Adopted Faster: 3M downloads in H1’2014 100M downloads at end of 2014 23
  24. 24. Reality Check Time 24
  25. 25. Technology Adoption Curve 101 25
  26. 26. Technology Adoption Curve 101 25 We are roughly here!
  27. 27. Technology Adoption Curve 101 25 We are roughly here! … and headed for here!
  28. 28. 26
  29. 29. How Do We Know? • Growing skepticism from analysts, reporters, and pundits • Growing dissatisfaction with certain aspects of OpenStack • Lots of failures in the field, enough to be worrisome • Peak OpenStack? • 6K+ attendees, early signs of slow down in adoption? • Decide for yourself; I could be calling it early 27 1 2 3
  30. 30. The Growing Skepticism 28 Linthicum believes that despite the fact that OpenStack has "the only game in town" for open source, the implementation hasn't met up to all of the hoopla since its release. OpenStack can run a fine private cloud, if you have lots of people to throw at the project and are willing to do lots of coding, according to Alan Waite, a research director at Gartner. OpenStack has the following drawbacks as a platform on which to build a private cloud*:  1 Difficulty of implementation 2 Shortage of skills available in the market 3 Conflicting or uncoordinated project governance 4 Weak spots in some projects 5 Integration with existing infrastructure *Recent Q1’2015 Gartner Report
  31. 31. OpenStack Self-Improvement Survey • Intention: • determine if and where project dissatisfaction exists • report back to provide perspective on where we need to change • After 10 days: • 65+ respondents w/ 30 months average time with OpenStack • Survey: [ TAKE ME! ] 29
  32. 32. How Would Your Characterize Your Participation in OpenStack Land? 30 19% 16% 30% 35% OpenStack Developer OpenStack Operator/Administrator OpenStack End-User (MIA) Pundit, Analyst, Reporter, OpenStack Evangelist, or Groupie Other Average Time Working With OpenStack: 32 months
  33. 33. What is Your MOST favorite OpenStack Project? 31 Nova Swift Heat Keystone Neutron Ironic Cinder Designate Ceilometer Trove Barbican Glance Horizon Manila Oslo Sahara TripleO Zaqar Other 0 4 8 12 16 Responses
  34. 34. What is Your LEAST favorite OpenStack Project? 32 Ceilometer Neutron TripleO Cinder Horizon Oslo Glance Keystone Nova Heat Ironic Sahara Swift Trove Zaqar Barbican Designate Manila Other 0 4 8 12 16 Responses
  35. 35. User Survey Feedback • Neutron: • “Neutron is a lot more complex and harder to provide real HA” – Survey Respondent • “Complexity, availability and scalability remain some of the concerns [ of the operators during the Operator Meetup in March ]” – User Survey Team • Ceilometer: • “adoption has not been rising as quickly as expected … dozens of comments related to stability and reliability, particularly at scale.” – User Survey Team 33
  36. 36. User Survey: Neutron 34
  37. 37. Well run technology organizations will often throw away or re-architect v1 and even v2 products.  Do you think this is a good practice? 35 19% 81% Yes No
  38. 38. Why can’t we fix these? It’s been years now… 36
  39. 39. OpenStack Threats • Explosive growth drives complexity • Continued complexity slows adoption • Can’t simplify, kill, or re-architect “broken” projects • Rigid technical governance model (still too centralized) • Long term vision & product strategy doesn’t emerge (in process) 37
  40. 40. Towards Glory 38
  41. 41. Technology Adoption Curve 101 39
  42. 42. Technology Adoption Curve 101 39 Need to get here!!
  43. 43. Path to the Plateau of Productivity 40 Plan Item Objective #1) Streamlining Governance Model empower projects, scale TC, focus Product WG, focus Board and Foundation on marketing and interoperability #2) Allow Competition force poor projects to evolve or die, allow other projects, particularly non-Python to come under our “big tent” #3) Conform to Well Known APIs don’t create new APIs in places where they exist (e.g. OAuth 2.0) #4) Testable Reference Architectures allow for vertical and horizontal-specific OpenStack reference implementations and separate infrastructure from platform #5) Ruthless Simplification downloadable “OpenStack Basic IaaS” should be 1-click download and install to run a POC/trial on a simple stack (1 switch, 10 servers)
  44. 44. 41 18 Categories (including retired), 252 Projects The ASF Scales Source: To Manage This… You Need This.
  45. 45. Allow Competing Projects & Multiple Languages • Competition is good; pretending our shit doesn’t stink is bad • Poor projects must die; survival of the fittest works • There is already leeway for this: • “Where it makes sense, the project cooperates with existing projects rather than gratuitously competing or reinventing the wheel.”* • i.e. Competitive projects are OK as long as they have good reason • Python isn’t good for everything • A bigger tent means allowing non-Python projects • Swift is already experimenting with re-writing pieces in Go Language (golang) 42 Source:
  46. 46. –Thierry Carrez, Chairman of the TC, Foundation Release Manager “OpenStack is about community, common values, and a common governance model.” 43
  47. 47. Keystone API • Seriously … WTF? • There are dozens of well known, documented, scalable, tested, standard APIs for authN/authZ • OAuth1/2, SAML, Kerberos • There is no excuse for creating something from whole cloth • Google is secure as hell and they use OAuth 2.0 • You aren’t better at security than the Google team; sorry • We don’t apply this standard to our community (completely new Nova API anyone?) 44
  48. 48. Example Reference Architectures 45 OpenStack Interop Standard RA “Key” Components RA Optional Components Basic IaaS 1+ of Nova/Magnum/Ironic OAuth 2.0 Server (Keystone or other) Glance, Horizon Advanced IaaS OpenStack Basic IaaS Cinder, Swift Neutron (or an alternative?) OAuth 2.0 Server (Keystone or other) Glance, Horizon OpenStack App Services Zaqar, Trove, Designate, OAuth2.0 Horizon OpenStack App Management Heat, Murano, Mistral, Horizon, OAuth2.0 Horizon OpenStack for NFV Basic IaaS Pluggable SDN Controller w/ Neutron APIs OpenStack Public Cloud Advanced IaaS + OpenStack App Svcs + OpenStack App Mgmt ec2api, gce-api, etc.
  49. 49. P2 tests How it Might Work 46 Reference Architecture Default Config Opts Key + Optional Projects { Project 1 Project 2 Interoperability Test Suite Defined “Capabilities” (previously “DefCore”) RefStack P1 tests Capabilities Tests API Code Owner(s): Infrastructure Team & Working Groups TC, Board, Vertical/Horizontal Working Groups, Community, & Foundation PTLs & key committers/reviewers (more like Apache PMC??) Unit Tests
  50. 50. We Are t3h Borg. You Will Be Assimilated. 47 “Maybe I’m an idiot, but I have no idea what anyone is talking about. What is it? It’s complete gibberish. It’s insane. When is this idiocy going to stop?” Larry Ellison on Cloud ComputerWorld, July 2000 "This 'telephone' has too many shortcomings to be seriously considered as a means of communication. The device is inherently of no value to us.”, Western Union internal memo, 1876. Decca Records rejected the Beatles, saying "guitar groups are on the way out" and "The Beatles have no future in show business,"
  51. 51. We Can Do It! • Interrelated, but not interdependent projects • Testable reference architectures that are interoperable • Streamline governance • Survival of the fittest project and programming language • OpenStack is not specific code or APIs, it’s: • Community, common values, and common governance 48