Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Joint OpenStack Kubernetes Environment (preview)


Published on


Presented as a preview of my OpenStack BCN presentation of the same title at the Bay Area Meetup (

See the updated version at the OpenStack summit here:

Published in: Software
  • This was a very informative session. Thanks Rob for sharing these insights (and with a humorous touch!) I am unclear as to what problem we are solving with this OpenStack sandwiched between Kubernetes overlay and underlay. If we are targeting reducing the complexity for an Administrator, this does not cut it for day 0 or 1 or 2 ops. You hit on the head with your comment on OpenStack networking. I was a developer for Tandem/NonStop systems where we wrote context free aka stateless server code before it became a buzz. One of the challenges we, and our customers, constantly ran into was lack of skilled talent available in the market who could design and code per the NonStop paradigms of shared nothing, always available, massive parallel architecture. The second gap that I see with these cool technologies is attention to enterprise readiness. Not just the lifecycle of containers, but how am I going to manage them per my SLAs and policy, how am I going to make sure that they stay compliant irrespective of where they get deployed, and are always available. Delivering that business function comprising of many microservices spread across hybrid deployment models and executing to policies and meeting all SLAs is still some ways to go…If we can start breaking this complex problem into smaller pieces and deliver on it, then we would start solving the challenges faced by the Dev and Ops folks in a real enterprise world.
    Are you sure you want to  Yes  No
    Your message goes here
  • VIDEO + SLIDES. This topic drew a H-U-G-E crowd (100+) people. We talked about the practical technical and marketing implications of using Kubernetes as an underlay for OpenStack. What's an Underlay? That's in there too.
    Are you sure you want to  Yes  No
    Your message goes here

Joint OpenStack Kubernetes Environment (preview)

  1. 1. Will it blend? A Joint OpenStack Kubernetes Environment A pragmatic operational assessment of if and when Kubernetes can become an underlay for OpenStack.
  2. 2. TL;DR: Not today. In the future, yes.
  3. 3. Rob Hirschfeld (aka Zehicle online) In Community: OpenStack Board Member (4 years) Co-Chair of Kubernetes Cluster Ops SIG Founder of Digital Rebar & Crowbar Projects Professional: CEO of RackN - hybrid automation software Executive at Dell - scale data center ops Cloud Data Center Ops going back to 1999
  4. 4. What is Kubernetes? Container Orchestration / Container Scheduler API driven to provide restart, placement, network routing and life-cycle For Applications designed for Kubernetes Key Design Elements: Immutable Infrastructure (stateless ops) 12 Factor Configuration Service Oriented
  5. 5. Platform: Three Tier Application Client 0 Ready 1 Prereq 2 Control 3 Nodes etcd (cluster) etcd (cluster) etcd (cluster) API (cluster) API (cluster) API (cluster) Kubelet KubeCtl Container Manager 5 Apps Network CNI Host Network Host Storage Host Init Pod Pod Pod Pod 4 Add-Ons Certificate Authority Scheduler (leader) Heapster Infrastructure APIs Routers, Storage, LBs... Proxy ... Controller (leader) DNS Watcher ...
  6. 6. Together 4ever: API server + Kubelet Client 0 Ready 1 Prereq 2 Control 3 Nodes etcd (cluster) etcd (cluster) etcd (cluster) API (cluster) API (cluster) API (cluster) Kubelet KubeCtl Container Manager 5 Apps Network CNI Host Network Host Storage Host Init Pod Pod Pod Pod 4 Add-Ons Certificate Authority Scheduler (leader) Heapster Infrastructure APIs Routers, Storage, LBs... Proxy ... Controller (leader) DNS Watcher ...
  7. 7. Who Cares? Operators! Operational Success is Essential to Project Success Operators are not developers! Simple, Transparent and Stable are key concerns Becoming a super-user of the platform should not be required to run it Scale & Upgradability has both internal and external drivers Generally, Kubernetes has good operational fundamentals
  8. 8. First: We’re Confusing Technical and Marketing Marketing around Kubernetes under OpenStack is a “hot mess” ● People heard “Kubernetes is stable, OpenStack is not” ● Further confuses “OpenStack one platform message” Who is promoting doing this? Mirantis / CoreOS / Intel / Google Confusion with the Plain Old Container Install (“POCI”) message ● Canonical (Ubuntu Cloud Install), ● Rackspace (OpenStack Ansible) ● Cisco (Kolla)
  9. 9. We’re Talking Underlay, not Overlay We’re talking about installing Kubernetes first (aka underlay) and using it to manage the OpenStack control plane. This approach is not a win if we ● Disable Kubernetes management ● Still need outside management tooling For now, we’ll ignore that our user may actually want to use Kubernetes as the overlay. IMHO, a bad assumption. Physical Infrastructure Kubernetes Underlay OpenStack Kubernetes Overlay Here! Simplest conception of the K8s OpenStack Sandwich
  10. 10. Second: Why I’m scared This discussion keep kicking the operations & install problems down the field Kubernetes is much newer than OpenStack, so even less understood Yet more complexity and some very basic questions: ● Now we have a both a Kubernetes and OpenStack upgrade problem ● We still need tooling to manage OpenStack in Kubernetes ● We still need someone to package the containers ● Relies on Docker to keep systems running ● Storage and Networking are still being worked out
  11. 11. But, it’s going to happen anyway...
  12. 12. Key Principle: Containerization vs Kubernetes Containers can be treated as a) lightweight vms or 2) packaged daemon sets. ● Canonical builds their containers like persistent vms and configures with Juju ● Kolla & OSA treats containers as packaging and configures with Ansible Kubernetes accepts neither approach – they expect containers to be immutable and 12 factor configured ● Kubernetes manages the full container life-cycle ● Containers need to be able to handle being added, removed ● Services need to be able to handle IP address changes (or use DNS names)
  13. 13. Specific Technical Barriers Host / Pinned vs Managed Containers ● It is possible to disable Kubernetes management and pin containers ● This eliminates the desired benefit of using Kubernetes How to handle Layered SDN integrations? Who wins? How to handle expectations of container persistence? Assumptions of Exclusive Ownership / Administrative Control
  14. 14. General Challenges to Overcome Complexity ● Overall Complexity of More Components ● Need to Control Kubernetes & Kubernetes Stability ● Need for Multiple Tiers of Load Balancer ● IP Mobility in Service Registration & Message Bus ● Mixed Networking Models ● Utility Upgrades and maintenance of the Underlay ● Mixing Kubernetes workloads Networking Utility
  15. 15. There are REAL Potential Benefits Part of Kubernetes Ecosystem (which is likely bigger than OpenStack’s) Leverage Docker packaging efforts and reduce Python & O/S dependencies Upgrades would benefit from Kubernetes built-in processes Use of the Kubernetes job scheduler for maintenance “Free” fault tolerance of key components Easier install if Kubernetes already running on-site More constrained options for configuration and operation
  16. 16. How could this actually be done quickly? Focus on Control plane, leave the nodes alone. ● Workers are “pinned” and agents need administrative control Cherry pick services to move into Kubernetes ● Focus on web services ● Separate support services (data & message bus) Externalize data using service registry Physical Infrastructure Kubernetes Underlay OpenStack Mgmt OpenStack Nodes SupportServices Make sure OpenStack Projects can handle immutable container requirements
  17. 17. In summary, OpenStack operability is not solved via the underlay platform alone. Technical Leadership motivation required for OpenStack adopting Kubernetes architecture requirements. Serious messaging confusion in effort has to be resolved. However, this collaboration is required for OpenStack Because Kubernetes will have a larger footprint in Operations