Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From Kubernetes to OpenStack in Sydney

1,106 views

Published on

This document is a presentation from OpenStack Summit Sydney. It describes how to easily install OpenStack on Kubernetes. It explains Kubernetes and OpenStack-Helm.

Published in: Software
  • Be the first to comment

From Kubernetes to OpenStack in Sydney

  1. 1. November 2017 Seungkyu Ahn / Taeil choi (Robert Choi) From Kubernetes to OpenStack
  2. 2. Index ▪ TACO Overview (SKT All Container OpenStack) ▪ Why Kubernetes? ▪ Software stack ▪ Demo (Installing OpenStack) ▪ Kubespray ▪ Kolla ▪ Helm ▪ OpenStack-Helm ▪ Deployment profiles ▪ Deploy OpenStack ▪ Challenges ▪ What’s missing ▪ TACO Milestone & Future Plan
  3. 3.  OpenStack Helm + Continuous Integration/Deployment  OpenStack Lifecycle Management on Kubernetes • Easy version upgrade • Minimize service impact on deployment (Rolling update) • Scale out and simply add a Compute server • Self-Healing (Automatic recovery in process down) TACO (SKT All Container OpenStack)
  4. 4. Why Kubernetes? ▪ Automatic binpacking (Managing container) ▪ Horizontal scaling ▪ Automated rollouts and rollbacks ▪ Self-healing ▪ Service discovery and load balancing ▪ Secret and configuration management
  5. 5. Software stack Chart Kubespray
  6. 6. Demo System deploy node k1-master01 k1-master02 k1-master03 k1-node01 k1-node02 k1-node03 k1-node04 Label : openstack-control-plane=enabled openvswitch=enabled Label : openstack-compute-node=enabled openvswitch=enabled k1-node05
  7. 7. Why Kubernetes? Demo
  8. 8. Installation order 1. Installing Kubernetes using kubespray 2. Creating ceph user secret and storageclass 3. Setting the label on nodes 4. Building OpenStack docker images using Kolla 5. Packaging OpenStack helm charts 6. Deploying OpenStack
  9. 9. Kubespray • Kubernetes incubator project • Ansible • Latest version support ✓ Kubernetes: v1.8.0 ✓ Calico: v2.5.0 or Flannel: v0.8.0 or Weave: 2.0.1 ✓ Helm: v2.6.1 ✓ EFK (Elastic Search, Fluentd, Kibana) : v5.4.0, 1.22, v5.4.0 • Added features in TACO (SKT All Container OpenStack) ✓ CI / CD ✓ Prometheus for monitoring
  10. 10. Kubespray • Should be changed files ✓ inventory/inventory.example ✓ inventory/group_vars/k8s-cluster.yml • Install Kubernetes ✓ ansible-playbook -u taco -b -i inventory/inventory.example cluster.yml • scale.yml : Adding nodes • upgrade-cluster.yaml : Upgrading kubernetes • reset.yaml : Uninstalling kubernetes cluster
  11. 11. Kubespray • Should be changed files ✓ inventory/inventory.example ✓ inventory/group_vars/k8s-cluster.yml • Install Kubernetes ✓ ansible-playbook -u taco -b -i inventory/inventory.example cluster.yml
  12. 12. Inventory example k1-master01 ansible_port=22 ansible_host=k1-master01 ip=192.168.30.13 k1-master02 ansible_port=22 ansible_host=k1-master02 ip=192.168.30.14 k1-master03 ansible_port=22 ansible_host=k1-master03 ip=192.168.30.15 k1-node01 ansible_port=22 ansible_host=k1-node01 ip=192.168.30.12 k1-node02 ansible_port=22 ansible_host=k1-node02 ip=192.168.30.17 k1-node03 ansible_port=22 ansible_host=k1-node03 ip=192.168.30.18 k1-node04 ansible_port=22 ansible_host=k1-node04 ip=192.168.30.21 [etcd] k1-master01 k1-master02 k1-master03 [kube-master] k1-master01 k1-master02 k1-master03 [kube-node] k1-node01 k1-node02 k1-node03 k1-node04 [k8s-cluster:children] kube-master kube-node
  13. 13. Kubespray • Should be changed files ✓ inventory/inventory.example ✓ inventory/group_vars/k8s-cluster.yml • Install Kubernetes ✓ ansible-playbook -u taco -b -i inventory/inventory.example cluster.yml
  14. 14. k8s-cluster.yml example kube_version: v1.8.0 kube_network_plugin: calico kube_service_addresses: 10.96.0.0/16 kube_pods_subnet: 172.16.0.0/16 etcd_deployment_type: docker kubelet_deployment_type: host etcd_memory_limit: 8192M dashboard_enabled: true efk_enabled: true helm_enabled: true
  15. 15. Kubespray • Should be changed files ✓ inventory/inventory.example ✓ inventory/group_vars/k8s-cluster.yml • Install Kubernetes ✓ ansible-playbook -u taco -b -i inventory/inventory.example cluster.yml
  16. 16. Storage - PV and PVC (w/ Ceph) • Secret files (openstack namespace) - user ✓ ceph-secret-user.yml • Storage class ✓ ceph-storageclass.yml • Secret files (kube-system namespace) - admin, user ✓ ceph-secret-admin.yml ✓ ceph-secret-user.yml
  17. 17. Kubernetes storage (w/ ceph) • Static Provosioning ✓ rbd manual creation PV manual creation : Setting rbd and storageclass PVC manual creation : Connect with PV (PV Name or PV Selector), Setting Storageclass (if not exist, using default storageclass) • Dynamic Provisioning ✓ Manual creation of PVC (Storageclass) : PV, rbd are automatically generated ✓ Automatic generation : Stateful (volumeClaimTemplates)
  18. 18. Storage - PV and PVC (w/ Ceph) • Secret files (openstack namespace) - user ✓ ceph-secret-user.yml • Storage class ✓ ceph-storageclass.yml • Secret files (kube-system namespace) - admin, user ✓ ceph-secret-admin.yml ✓ ceph-secret-user.yml
  19. 19. Secret file - ceph-secret-admin.yml apiVersion: v1 kind: Secret metadata: name: "ceph-secret-admin" namespace: "kube-system" type: "kubernetes.io/rbd" data: key: ”xxxxxxx==" grep key /etc/ceph/ceph.client.admin.keyring | awk '{printf "%s", $NF}' | base64
  20. 20. Storage - PV and PVC (w/ Ceph) • Secret files (openstack namespace) - user ✓ ceph-secret-user.yml • Storage class ✓ ceph-storageclass.yml • Secret files (kube-system namespace) - admin, user ✓ ceph-secret-admin.yml ✓ ceph-secret-user.yml
  21. 21. Secret file - ceph-secret-user.yml apiVersion: v1 kind: Secret metadata: name: "ceph-secret-user" namespace: "kube-system" type: "kubernetes.io/rbd" data: key: ”xxxxxx==" grep key /etc/ceph/ceph.client.kube.keyring | awk '{printf "%s", $NF}' | base64
  22. 22. • Secret files (openstack namespace) - user ✓ ceph-secret-user.yml • Storage class ✓ ceph-storageclass.yml • Secret files (kube-system namespace) - admin, user ✓ ceph-secret-admin.yml ✓ ceph-secret-user.yml Storage - PV and PVC (w/ Ceph)
  23. 23. Storage class file - ceph-storageclass.yml apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: "ceph" annotations: storageclass.beta.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/rbd parameters: monitors: "192.168.30.23:6789,192.168.30.24:6789,192.168.30.25:6789" adminId: "admin" adminSecretName: "ceph-secret-admin" adminSecretNamespace: "kube-system" pool: "kube" userId: "kube" userSecretName: "ceph-secret-user"
  24. 24. • Secret files (openstack namespace) - user ✓ ceph-secret-user.yml • Storage class ✓ ceph-storageclass.yml • Secret files (kube-system namespace) - admin, user ✓ ceph-secret-admin.yml ✓ ceph-secret-user.yml Storage - PV and PVC (w/ Ceph)
  25. 25. Secret file - ceph-secret-user.yml apiVersion: v1 kind: Secret metadata: name: "ceph-secret-user" namespace: ”openstack" type: "kubernetes.io/rbd" data: key: ”xxxxxx==" grep key /etc/ceph/ceph.client.kube.keyring | awk '{printf "%s", $NF}' | base64
  26. 26. Label kubectl label node k1-node01 openstack-control-plane=enabled kubectl label node k1-node01 openvswitch=enabled kubectl label node k1-node02 openstack-control-plane=enabled kubectl label node k1-node02 openvswitch=enabled kubectl label node k1-node03 openstack-control-plane=enabled kubectl label node k1-node03 openvswitch=enabled kubectl label node k1-node04 openstack-compute-node=enabled kubectl label node k1-node04 openvswitch=enabled
  27. 27. Kolla ● OpenStack project 로 OpenStack service 들의 docker image 를 생성 및 관리하는 Tool ● OpenStack 서비스들 뿐만 아니라 다양한 관련 application들의 docker image 제공
  28. 28. Kolla - Dockerfile example
  29. 29. Kolla Dockerfile.j2
  30. 30. Kolla build • kolla-build -b ubuntu -t source --template-override template-overrides.j2 keystone override template-override.j2
  31. 31. • Automation tool for managing Kubernetes applications. • Helm Charts helps you define, install, and upgrade Kubernetes application. (Server) (client) - Helm Architecture - Helm chart structure
  32. 32. Helm chart structure
  33. 33. Kubernetes manifest format • Manifest file for deploying minio pod kind: Deployment metadata: name: minio labels: app: minio spec: replicas: 1 template: metadata: labels: app: minio spec: affinity: nodeAffinity: … containers: - name: minio image: minio/minio:latest imagePullPolicy: Always args: - server - /storage
  34. 34. apiVersion: extensions/v1beta1 kind: Deployment metadata: name: {{ template "minio.fullname" . }} labels: app: {{ template "minio.fullname" . }} spec: {{- if eq .Values.mode "shared" }} replicas: {{ .Values.replicas }} {{- end }} template: metadata: name: {{ template "minio.fullname" . }} labels: app: {{ template "minio.fullname" . }} spec: volumes: - name: export {{- if .Values.persistence.enabled }} persistentVolumeClaim: claimName: {{ template "minio.fullname" . }} {{- else }} emptyDir: {} {{- end }} - name: minio-server-config configMap: name: {{ template "minio.fullname" . }}-config-cm - name: minio-user secret: secretName: {{ template "minio.fullname" . }}-user containers: - name: minio image: {{ .Values.image }}:{{ .Values.imageTag }} … • Rendering: actual values are assigned from separate values file • Rendered manifest is passed to k8s API (where is rendering done?) replicas: 1 image: "minio/minio:latest" imagePullPolicy: "Always“ … Values.yaml Helm Chart Template
  35. 35. 35 OpenStack-Helm • Collection of charts for managing most openstack services. • Since November 2016 by AT&T (https://github.com/openstack/openstack-helm)
  36. 36. 36 Openstakc-helm > Keystone chart structure Launches keystone pod Test pod Contains values Keystone config
  37. 37. 37 SKT’s pipeline > Wrapper Chart • Customize values for target environment. • Don’t touch original chart and the wrapper only has values to override. • Values merged -> SKT chart generated -> Pushed into internal repository.
  38. 38. Deployment Profiles • Necessary to deploy openstack cluster into various environments • Charts URLs + configuration overrides (Eg, network conf, repository URL and so on) • Open-source orchestration tools • Landscaper • Started on Nov 2016 by Eneco. • Each conf file is for single chart -> Many small configs • Pretty stable, but only provides basic functionality. • Armada • Started on Feb 2017 by AT&T. • One global big conf file for all charts • Not as stable as Landscaper yet, but has extra functionality. (Pre/post actions, undeploy, chart grouping, and so on.) • We’re trying to migrate from Landscaper to Armada.
  39. 39. Deployment Profiles (cont.) • Charts URL + env-specific configuration overrides
  40. 40. Deployment Profiles (cont.) • Profile: Templates + original values + wrapper values + env-specific values • To apply: “$ armada apply PROFILE_NAME” templates original values values by wrapper values for ‘dev’ values values for ‘stg’ values for ‘stg’ Keystone templates original values values by wrapper values for ‘dev’ values values for ‘stg’ values for ‘stg’ Glance … ‘dev’ profile SKT Wrapper Chart
  41. 41. Deployment Profiles (cont.) • Profile: Templates + original values + wrapper values + env-specific values • To apply: “$ armada apply PROFILE_NAME” templates original values values by wrapper values for ‘dev’ values values for ‘stg’ values for ‘stg’ Keystone templates original values values by wrapper values for ‘dev’ values values for ‘stg’ values for ‘stg’ Glance … ‘stg’ profile SKT Wrapper Chart
  42. 42. Challenges > Summary (#1) ▪ Too many artifacts to track or manage ▪ Tools (binary) ▪ Docker, Kubernetes, Helm, Landscaper or Armada, … ▪ Docker images & sources ▪ Base OS images & Kolla images ▪ Wrapper images ▪ Helm charts & source ▪ Openstack-helm charts ▪ Wrapper charts: Additional template, Override SKT-specific values ▪ Deployment profiles ▪ For various environments ( Eg, ‘dev’, ‘stg’, ‘prod’, … ) ▪ Versioning and promotion policies for the above artifacts ▪ Isolated environment for each build job ▪ Eg) daemonset conflicts for OVS or libvirt ▪ Etc
  43. 43. Challenges > Summary (#2) ▪ Too many artifacts to track or manage ▪ Tools (binary) ▪ Docker, Kubernetes, Helm, Landscaper or Armada, … ▪ Docker images & sources ▪ Base OS images & Kolla images ▪ Wrapper images ▪ Helm charts & source ▪ Openstack-helm charts ▪ Wrapper charts: Additional template, Override SKT-specific values ▪ Deployment profiles ▪ For various environments ( Eg, ‘dev’, ‘stg’, ‘prod’, … ) ▪ Versioning and promotion policies for the above artifacts ▪ Isolated environment for each build job ▪ Eg) daemonset conflicts for OVS or libvirt ▪ Etc
  44. 44. Challenge > Track upstream changes ▪ Problems: too many things to track ▪ Version upgrade of tools ▪ K8s, helm upgrade -> broken build! ▪ Openstack source, kolla source (trivial) ▪ Openstack-helm project (major one) ▪ Fast and actively moving target ▪ Hard to track upstream changes immediately by hand ▪ Periodic sync/merge -> too many change -> broken build (painful to fix it) ▪ Solution: Automation (on Jenkins) ▪ Fetch hourly -> Build SKT chart -> Test -> Merge if the test passes ▪ If test fails, create ticket and notify developers about the failure ▪ Jira plugin for create the ticket ▪ Slack plugin for the notification
  45. 45. Challenge > Versioning and Promotion ▪ Problems ▪ Should be able to identify relationship between related artifacts (Eg, kolla image <-> Helm chart) ▪ Solution ▪ Consistent versioning ▪ Dev: after build stage ▪ Stage: after daily integration test ▪ Release: on demand by hand Dev (hourly) Stage (daily) Release (manual) Artifact Kolla Image 0.1.0 yy.mm.dd 1.0.0 -> … -> 1.0.x Helm chart & profile 0.1.0 yy.mm.dd 1.0.0 -> … -> 1.0.x Source Code Branch Master stage ReleaseX Tag N/A yy.mm.dd 1.0.0 -> … -> 1.0.x
  46. 46. What’s missing ▪ CI for Kubernetes itself ▪ Track kubernetes version upgrade ▪ Apply new version ASAP with some validation tests ▪ Resiliency Test (like chaos-monkey) ▪ Make sure the openstack cluster tolerate node failure ▪ Randomly terminate resources such as pod, daemonset in the cluster at specified interval & duration ▪ TACO Client Tool ▪ CLI Tool with which users can use most functions of TACO easily. ▪ Deploy/undeploy/patch/upgrade openstack services ▪ Adding/removing openstack node (usually compute node)
  47. 47. TACO Milestone • Current Status • Currently beta release • Upstream-related work • Cooperating closely with members of OpenStack-Helm project (e.g., AT&T, Intel) • 3rd place in code contribution ranking in the OpenStack-Helm (as of 11/02/17) • OpenStack-Helm is now official project: Join us! • Future plan • Once Missing part is done -> Production-Ready Release! • Release Plan • 2018: Greenfield Production Deployment (SKT Internal Private Cloud) • 2018: Feasibility Test and PoC for Telco Infra (e.g., dataplane acceleration, security, etc) • 2019~ : Production Deployment for Telco Infra • TBD: Infra Service that provides both VM and Containers & Container-Based SW Delivery Platform
  48. 48. Q & A Question?

×