To kayobe or not to kayobe?

1. To Kayobe or not to Kayobe? Mark Goddard Manchester OpenStack Meetup 12th March 2018

2. Overview ● Kayobe ○ Deployment of containerised OpenStack to bare metal ● Background ● Existing Solutions ● Kayobe ● Demo

3. Background

4. Me --- name: Mark Goddard education: degree: Electronic & Communications Engineering institution: University of Bristol career: - tech: Video conferencing bridges company: Tandberg Cisco - tech: High Performance Ethernet switches company: Gnodal - tech: OpenStack & Software Defined Networking (SDN) company: Cray - tech: Scientific OpenStack company: StackHPC

5. StackHPC ● Bristol-based software consultancy ● “The convergence of HPC & cloud” ● Started ~2 years ago by Stig Telfer & John Taylor ● Current headcount: 9 ● Involved with University of Cambridge, Square Kilometre Array (SKA), Verne Global (hpcDIRECT), Data Direct Networks (DDN), Cray, others... ● https://stackhpc.com

6. A Kayobe is Born ● Square Kilometre Array (SKA) ● Science Data Processor (SDP) ● Performance Prototype Platform (P3) / ALaSKA ○ Hardware & software test bed ○ Two racks of bare metal compute ○ Heterogeneous nodes ■ High memory, NVMe, GPU ○ Lots of networks ■ 1G, 10G, 25G Ethernet ■ 100G EDR Infiniband ○ Managed by OpenStack ● Kayobe built to address pain points of existing tools

7. Existing Solutions

8. Surely OpenStack has enough deployment tools already? ● Yes, but... ● Existing tools have several pain points ● Many new deployment tools are based on Kubernetes

9. TripleO - OpenStack on OpenStack ● Basis of Red Hat’s OpenStack Platform (RHOS) ● Undercloud ○ Bare metal deployment of control plane ○ A separate, single node OpenStack installation ■ Heat, nova, neutron, ironic, etc. ● Overcloud ○ Multi-node, Highly Available (HA) OpenStack control plane ○ Multiple roles - controller, compute, blockstorage, etc. ○ Now using kolla container images ○ Deployed using a giant monolithic heat stack ■ OpenStack resources (networks, instances) ■ Service installation & configuration

10. TripleO - Common Pain Points ● Undercloud is complicated ● Monolithic heat stack ○ Places limits on scalability (~300 nodes?) ○ Problems are hard to diagnose ○ Deploy/reconfigure/upgrade are all-or-nothing (scary) ■ The entire heat stack is updated ■ Hard to prove it won’t change something unexpected ● Technologies are easy to add, less easy to remove ○ Heat, nova, Puppet, Pacemaker, mistral, Ansible (buried in heat templates…), Docker, Kubernetes (eventually) ● No isolation between services ○ Switch to containers should help here ● Puppet adds a layer of abstraction to config

11. Kubernetes-based Tools ● Deploy an OpenStack control plane using Kubernetes ● Kolla Kubernetes ● OpenStack helm ○ Seems to have the most momentum ○ Mostly single vendor (AT&T) ● TripleO (in future) ● Stackanetes (unmaintained) ● Fuel CCP (unmaintained)

12. What’s Wrong With Kubernetes? Disclaimer: I’ve not used any of these tools… ● A great way to run scalable, reliable microservices ● But... ● Don’t we have enough turtles already? ● How to deploy Kubernetes? ● OpenStack isn’t very “cloud native” ● Considerable scaffolding required to run OpenStack on Kubernetes

13. Kolla Ansible ● Containerised control plane using kolla images ● Ansible keeps things (relatively) simple ● Configure any* configuration option using native INI files ● Three principal operations ○ Deploy ○ Reconfigure ○ Upgrade ● Target specific services ○ kolla-ansible deploy --tags <tags> ● Target specific hosts ○ kolla-ansible upgrade --limit <hosts | groups>

14. What’s Wrong Missing from Kolla Ansible? ● Bare metal deployment of control plane* ○ Equivalent of TripleO’s undercloud ● Configuration of control plane host OS* ○ Kolla ansible largely expects hosts to be ready to deploy containers ● Bare metal compute node management * Limited support provided

15. Kayobe

16. Kayobe ● Etymology ○ Kolla On Bifrost (K-O-B) ● Adds some missing pieces to kolla ansible ○ Bare metal deployment of control plane using bifrost ○ K-a inventory generation & service placement ○ Configuration of control plane host OS ○ Management of physical network devices ○ Bare metal compute node management ○ A friendly openstack-like CLI ● Ansible playbooks & python CLI

17. Kayobe Software Stack kayobe CLI bifrost ironic kayobe playbooks kolla Docker images kolla ansible Docker containers

18. Kayobe Architecture Ansible control host [kayobe, kolla ansible] Seed host [bifrost, ironic] Control plane hosts [OpenStack] Bare metal compute hosts [user workloads] Controllers, network hosts, virtualised compute hypervisors, storage, etc. provisions provisions manages Network devices configures

19. Physical Network Device Management ● Uses Ansible’s network configuration modules ● Applies static network configuration ● Currently supports Dell OS 6, OS 9 & JunOS ● Network devices added to Ansible inventory ○ Can be used to configure neutron plugins e.g. genericswitch ● Configuration defined via host variables

20. Seed Host ● Host that runs bifrost services ● Roughly equivalent to TripleO’s undercloud ● Supports the same host configuration as the control plane ● Can be deployed as a VM ○ Allows for reuse of the hypervisor ○ Automation of: ■ Libvirt/KVM hypervisor ■ Seed VM provisioning

21. Bifrost ● Minimal* bare metal provisioning environment ○ OpenStack ironic ○ OpenStack ironic inspector ○ Dnsmasq ○ Nginx ○ MariaDB ○ RabbitMQ ○ Disk Image Builder (DIB) ● Deployed using ansible ● Discovers & inspects control plane nodes ● Deploys an image to a static inventory * Minimal for OpenStack

22. Host Configuration ● From provisioned image to container-ready ● Uses roles from Ansible Galaxy ● User bootstrapping ● Network configuration ○ Static IP address allocation ○ Interfaces (Ethernet, bridge, bond) ○ DNS ● NTP ● Disk wiping & LVM ● Docker storage ● User accounts ● Ceph block device preparation

23. Control Plane Service Placement ● Multiple standard host types supported ○ Controller ○ Compute (virtualised) ○ Storage ○ Monitoring ● Custom host types possible via group/host variables ○ e.g. Database, load balancer ● Discovered hosts mapped to a top level kayobe group ○ e.g. compute-01 is in the [compute] group ● Kayobe groups mapped to top level kolla ansible groups ○ e.g. hosts in the kayobe [compute] group should be added to the kolla ansible [compute] group ● Support for customisation of kolla ansible inventory ○ e.g. place MariaDB containers on hosts in the [database] group

24. Control Plane Service Placement control-01 controllers control nova-api, mariadb, rabbitmq, etc. compute-01 compute compute nova-compute, libvirt, etc. Control plane host [ironic inventory] kayobe top level group [kayobe host mapping] kolla ansible top level group [kayobe group mapping] kolla ansible service groups [kolla ansible inventory]

25. Usage Examples ● Provision control plane hosts ○ kayobe overcloud provision ● Configure host OS services on compute hosts ○ kayobe overcloud host configure --limit compute ● Reconfigure ironic ○ # Modify ironic.conf template ○ kayobe overcloud service reconfigure --kolla-tags ironic ● Upgrade keystone ○ kayobe overcloud service upgrade --kolla-tags keystone ● Snapshot the currently applied configuration ○ kayobe overcloud service configuration save

26. Demonstration!

27. Community

28. Kayobe Community ● StackHPC ● Square Kilometre Array (SKA) ○ Performance Prototype Platform (P3) ● Verne Global ○ hpcDIRECT ● Devoteam ○ Continuous Integration (CI) platform ● OpenStack Kolla community ○ Showed interest in collaboration during recent Project Teams Gathering (PTG) ● You?

29. Get Involved! ● Documentation: http://kayobe.readthedocs.io/en/latest/ ● Try it out: http://kayobe.readthedocs.io/en/latest/development/automated.html ● IRC: #openstack-kayobe

30. Thanks! Mark Goddard IRC: mgoddard | #openstack-kayobe @markgoddard mark@stackhpc.com