This document discusses using Kubernetes as an underlay platform for OpenStack. Some key points: 1. Kubernetes is becoming more widely used and understood by operators compared to OpenStack. Using Kubernetes as an underlay could improve simplicity, stability, and upgrade processes for OpenStack. 2. There are still many technical challenges to address, such as networking, storage, tooling to manage OpenStack on Kubernetes, and ensuring containers meet Kubernetes' immutable infrastructure requirements. 3. Using Kubernetes as an underlay risks further confusing the messaging around OpenStack by implying Kubernetes is more stable or a replacement target. Clear communication will be important to avoid undermining OpenStack.

1. May 2017 Update:
Will it blend?
Joint
OpenStack
Kubernetes
Environment
A pragmatic operational assessment about how
Kubernetes can become an underlay for OpenStack.

2. TL;DR: Yes
and then Kubernetes
wins as the platform.
Video Demo: bit.ly/rebarhelm

3. Rob Hirschfeld (aka Zehicle online)
In Community: OpenStack Board Member (4 years)
Co-Chair of Kubernetes Cluster Ops SIG
Founder of Digital Rebar & Crowbar Projects
Professional: CEO of RackN - hybrid automation software
Executive at Dell - scale data center ops
Cloud Data Center Ops going back to 1999

4. Addressing Operators Needs
Operational Success is Essential to Project Success
Operators are not developers!
Simple, Transparent and Stable are key concerns
Becoming a super-user of the platform should not be required to run it
Scale & Upgradability has both internal and external drivers
Generally, Kubernetes has good operational fundamentals
Even more, we need more community operational practices for OpenStack

5. We’re Talking Underlay, not Overlay
We’re talking about installing Kubernetes first (aka
underlay) and using it to manage the OpenStack control
plane.
Objectives for Kubernetes Underlay:
● Must Work with Kubernetes Primatives
● Not a Dedicated Kubernetes
● Limited Outside Management
Physical Infrastructure
Kubernetes
Underlay
OpenStack
Kubernetes
Overlay
This Talk
Simplest conception of the
K8s OpenStack Sandwich

6. What is Kubernetes?
Container Scheduler (no, it’s not really Orchestration)
API driven to provide restart, placement, network routing and life-cycle
For Applications designed for Kubernetes
Key Design Elements: Immutable Infrastructure (stateless ops)
12 Factor Configuration
Service Oriented

7. What is Kubernetes: A Three Tier Application
Client
0
Ready
1
Prereq
2
Control
3
Nodes
etcd
(cluster)
etcd
(cluster)
etcd
(cluster)
API
(cluster)
API
(cluster)
API
(cluster)
Kubelet
KubeCtl
Container Manager
5
Apps
Network CNI
Host
Network
Host
Storage
Host
Init
Pod Pod Pod Pod
4
Add-Ons
Certificate
Authority
Scheduler
(leader)
Heapster
Infrastructure
APIs
Routers,
Storage,
LBs...
Proxy
...
Controller
(leader)
DNS Watcher ...

8. Together 4ever: API server + Kubelet
Client
0
Ready
1
Prereq
2
Control
3
Nodes
etcd
(cluster)
etcd
(cluster)
etcd
(cluster)
API
(cluster)
API
(cluster)
API
(cluster)
Kubelet
KubeCtl
Container Manager
5
Apps
Network CNI
Host
Network
Host
Storage
Host
Init
Pod Pod Pod Pod
4
Add-Ons
Certificate
Authority
Scheduler
(leader)
Heapster
Infrastructure
APIs
Routers,
Storage,
LBs...
Proxy
...
Controller
(leader)
DNS Watcher ...

9. Kubernetes = Rainbows?!

10. Why do we want Kubernetes as Underlay?
Community Perception Accuracy
1 OpenStack Operations is still not “solved” True (no change)
2 We already do most new deploys in containers True (was partially)
3 Kubernetes is awesome at containers True (was partially)
4 Kubernetes is simple, stable and secure (for operators) Partially (was false)
5 Kubernetes means easy Upgrades and High Availability Partially (was false)

11. There are REAL Potential Benefits
● Leverage Docker packaging efforts and reduce Python & O/S dependencies
● Upgrades would benefit from Kubernetes built-in processes
● Use of the Kubernetes job scheduler for maintenance
● “Free” fault tolerance of key components
● Easier install if Kubernetes already running on-site
● More constrained options for configuration and operation
BUT REALLY, IT’S ABOUT LOWER FRICTION AND COMMUNITY SIZE...
I expect more people will understand Kubernetes operations than OpenStack
operations because Kubernetes is 1) simpler and 2) cloud and physical.

12. Kubernetes
Underlay is coming,
So let’s get
pragmatic about it.
Leadership Kudos to
SAP, ATT Comummity Dev, & Port Direct

13. Issues: Marketing Message is Confusing
Marketing around Kubernetes under OpenStack is a “hot mess”
● People hear “Kubernetes is stable, OpenStack is not”
● Further confuses “OpenStack one platform message”
● Encourages Kubernetes as target instead of OpenStack
Confusion with the Plain Old Container Install (“POCI”) message
● Canonical (Ubuntu Cloud Install),
● Rackspace (OpenStack Ansible)
● Cisco (Kolla)
● Triple O

14. Key Principle: Containerization vs Kubernetes
Containers can be treated as a) lightweight vms or 2) packaged daemon sets.
● Canonical builds their containers like persistent vms and configures with Juju
● Kolla & OSA treats containers as packaging and configures with Ansible
Kubernetes accepts neither approach – they expect containers to be immutable
and 12 factor configured
● Kubernetes manages the full container life-cycle
● Containers need to be able to handle being added, removed
● Services need to be able to handle IP address changes (or use DNS names)

15. This work is progressing quickly!
Using Kubernetes v1.5+ Primatives
● Using Kubernetes Helm Charts
● Services are tagged to nodes
● Agents become Daemon sets
● Databases using Stateful sets
● Multiple container sources
Hard work remains….
● Networking, Configuraton & Storage
● OpenStack Projects must handle immutable
container requirements
Physical Infrastructure
Kubernetes + Helm
Underlay
OpenStack
Mgmt
OpenStack
Nodes
Other
Apps

16. Kubernetes
Workers
More Detail: Kubernetes Underlay of OpenStack
Physical Infrastructure
Kubernetes
Controllers
OpenStack
Mgmt
OpenStack
Nodes
Data
base
If you to really want to build this, give me a call - RackN has all the components
Msg
Bus
Software Defined NetworkingCeph Distributed Storage
Other
Workloads
Helm

17. Technical Challenges Remain
This discussion keep kicking the operations & install problems down the field
Kubernetes is much newer than OpenStack, so even less understood
Yet more complexity and some very basic questions:
● Now we have a both a Kubernetes and OpenStack upgrade problem
● We still need tooling to manage OpenStack in Kubernetes
● We still need someone to package the containers (+ multi-platform like ARM)
● Relies on Docker to keep systems running
● Storage and Networking are still being worked out

18. In summary,
OpenStack operability is not solved via the underlay platform alone.
Technical Leadership motivation required for OpenStack adopting
Kubernetes architecture requirements.
Serious messaging confusion in effort has to be resolved.
However, this collaboration is required for OpenStack
Because Kubernetes will have a larger footprint in Operations

19. By 2018, this
approach will be
THE install method
Rob Hirschfeld, @zehicle
RackN & Digital Rebar