This document discusses running Kubernetes on OpenStack at scale. It describes how OpenStack provides automated provisioning of infrastructure resources, while Kubernetes provides a container platform for consuming those resources. The advantages of combining these technologies include fully automated infrastructure, consistent management experience, isolation for workloads, and leveraging existing plugins. It provides an example architecture using Red Hat OpenShift on OpenStack with key components like Ceph storage, Neutron networking integrated via Kuryr, and Heat for orchestration.

1. Running Enterprise Kubernetes on
OpenStack at Scale
Jonathan Gershater, Red Hat - July 2019
Senior Principal Product Marketing Manager
1

2. CONFIDENTIAL DesignatorAGENDA
2
Motivation for Kubernetes on OpenStack
Role of OpenStack
Role of Kubernetes
Sample architecture
Agenda

3. CONFIDENTIAL Designator
Why IaaS and containers?
Automation at both layers
CONTAINERS
Consumption of resources
Able to easily access new
environments to quickly build
new apps and move on
IAAS
Exposition of resources
Provide necessary
environments to developers in
minutes, not weeks or months

4. CONFIDENTIAL DesignatorOPENSTACK
4
Exposition and consumption of resources
IaaS layer
Container
application
platform
layer

5. CONFIDENTIAL DesignatorOPENSTACK
5
Exposition of resources
OpenStack:
● Automated provisioning of virtual
machine, storage and network
● Rinse and repeat, automated
● Scales very well

6. CONFIDENTIAL DesignatorKUBERNETES
6
Consumption of resources
Container Platform:
● Processes distributed in cluster
● Self service portal
● Define and share applications

7. CONFIDENTIAL DesignatorOPENSTACK AND KUBERNETES
7
Putting it all together
Architectural tenets:
● Technical independence
● Contextual awareness
● Avoiding redundancy
● Simplified management
SaaSPaaSIaaS
Your Application

8. CONFIDENTIAL DesignatorADVANTAGES
8
Technical advantages
1. API software defined infrastructure at all levels
a. Scale in sync, policy driven, dynamic resource allocation, etc.
2. Fully automated infrastructure resources for OCP consumption
3. Provide consistent infrastructure management experience
4. Provide the right level of isolation for each workload
5. Greater density levels over bare metal
6. Simplified deployment of apps to production-ready platform
7. Leverage existing certified plugin ecosystem for OpenStack Platform (Cisco, Juniper
Contrail, Nuage, etc.)

9. CONFIDENTIAL Designator
OpenStack BareMetal service
IRONIC
9
● Enterprise-Ready Bare Metal as a Service
● Trusted, multi-tenant platform
● Integration with Neutron, Nova and Cinder for a BMaaS experience on par with
that of the virtual instances

10. CONFIDENTIAL Designator
OpenStack High Availability
HA
10
3x Controllers
● Pacemaker (cluster coordination)
○ RabbitMQ (Internal message bus)
○ Galera (Clustered MariaDB)
○ Virtual IP’s (to coincide with HAproxy)
● HAproxy (Load balancing)
○ All OpenStack Service API’s
● Ceph storage monitors (if used)
Three dedicated OpenStack controllers, running (by default) the following...

11. CONFIDENTIAL Designator
Kubernetes High Availability
HA
11
● The OpenStack compute
nodes and Ceph OSDs are
grouped into availability
zones on a per-rack basis.
● The virtual machines are
all members of the same
OpenStack tenant.
● Affinity rules spread the
virtual machines across
the physical compute
nodes by role.

12. CONFIDENTIAL Designator
Storage
CEPH STORAGE
12
● A minimum of three Ceph monitors and three or more Ceph OSD
nodes are needed to ensure high availability in production.
● Recommend each Ceph node on dedicated physical servers.

13. CONFIDENTIAL Designator
Kubernetes tenant networking
NETWORKING
13
Public network: This network is
reachable by the outside world.
It is an OpenStack provider
network that maps to a
physical network that exists in
the data centre.
D
eployment network: An
internal network created by the
tenant user. All kubernetes
instances are created on this
internal network.

14. CONFIDENTIAL Designator
Networking, Kuryr
AVOID NETWORKING DOUBLE ENCAPSULATION
14
● Combining networking solutions for each platform can increase
complexity and unwanted performance overhead.
● Do not run a Kubernetes SDN on top of an OpenStack SDN.
● Kuryr uses a CNI plugin
to integrate Neutron
and Kubernetes
● Kuryr controller watches
for OCP events and
manages OSP resources
for them
● Kuryr allows containers
and virtual machines to
exist on the same
network segment
● Kuryr eliminates need
for multiple network
overlays which can
improve performance
and simplify
management

15. CONFIDENTIAL Designator
Compute, Heat
COMPUTE
15
● Heat is OpenStack’s orchestration service. It can launch
composite cloud applications based on text-file templates that
can be managed as code. ● Heat provides a
scalable and reliable
interface for
automating
Kubernetes
installations.

16. CONFIDENTIAL DesignatorRED HAT IMPLEMENTATION
16
Architecture example: OpenShift on OpenStack
OpenShift container platform
standard hardware
OpenStack shared services
KVM Ironic
VM VM
Service Container Container
compute networking storage
Containers, Virtual Machines, and Bare-metal

17. CONFIDENTIAL Designator
OpenShift on OpenStack deeper dive
RED HAT IMPLEMENTATION
17

18. CONFIDENTIAL Designator
Summary
THANK YOU
18
● Applications deployed in an on-premises private cloud or in a co-location facility for various reasons (for example,
security and compliance, data affinity, performance, among others). The IT organizations responsible for operating the
private cloud desire it to be simple, agile, flexible, secure, cost efficient, and be a part of their overall Hybrid and Multi
cloud architecture.
● Red Hat OpenShift Container Platform, Red Hat OpenStack Platform, and Red Hat Ceph Storage are the key
architectural components of this solution. It can be easily extended to Hybrid and Multi-Cloud with OpenShift
Container Platform serving as the common container and kubernetes platform across all clouds.

19. linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
Red Hat is the world’s leading provider of
enterprise open source software solutions.
Award-winning support, training, and consulting
services make
Red Hat a trusted adviser to the Fortune 500.
Thank you
19