Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Handle your Kubernetes Upgrades


Published on

Suvrojeet Ghosh, Software Engineer at Ribbon, presented 'How to Handle your Kubernetes Upgrades' at the Kubernetes + Cloud Native meetup in Ottawa in March, 2019. He shared his experiences upgrading HA clusters from v1.0 to v1.13 via kubeadm in multiple hops. He pointed out certain problems and errors to be aware of as well as resources that can help.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

How to Handle your Kubernetes Upgrades

  1. 1. How to Handle your Upgrades Thursday, March 7, 2019
  2. 2. $ whoami - Suvro $ kubectl describe suvro Suvrojeet Kumar Ghosh by day at Ribbon Communications and by evening GitHub: @fOO223Fr Twitter: @_suvzz
  3. 3. $ ls -ltr • Architecture and General methodology • Problems/Errors faced • Update Flowchart • Resources used for help • Heads-up for things to look for when upgrading! • [ bonus slide ;) ] Automation ideas • Questions? @_suvzz
  4. 4. Architecture Master Master Worker Worker Worker VIP passiveactive GlusterFS for etcd BackU p @_suvzz
  5. 5. General methodology • Hop by Hop • example: v1.9.2 to v1.12.5 v1.9. 2 v1.10. 12 v1.11. 0 Reason for such a kind of methodology are two: 1. Usually the patch 0 of every version has major changes from the last lower version. 2. The last path version (incase of v1.10 is 12) is kind of the accumulation of the intermediate patches. So before moving next major version running though this has proved beneficial for many. Note: this is a strategy and not everyone need to follow it because it depends what features your cluster is using and what has changed. But by far most are benefitted following this strategy. @_suvzz
  6. 6. Update Flowchart Update kubeadm binary kubeadm config images list kubeadm config images pull kubeadm upgrade apply v1.XX.x kubeadm config migrate Update your CNI Update all binaries kubectl , kubeadm, kubelet Optional for newer version > v1.9 Version older than v1.9 download a temp kubeadm binary, else it might mess with the drop in file. 10-kubeadm.conf Version > v1.11.0, take advantage of these cool subcommands helps if you have a private docker repo then you can download ahead of upgrade and tag it appropriately. I have used this ever since v1.11.0 saves a lot of time in replacing deprecated flags or configuration changes. kubeadm upgrade plan Check and/or Update nodeName to current Master kubectl edit configmap -n kube- system kubeadm-config -o yaml @_suvzz
  7. 7. Flowchart continued.. (optional) Update kubectl in worker nodes kubectl drain $NODEs --ignore- daemonsets Update kubelet version in worker nodes kubeadm upgrade node config --kubelet- version v1.X.x Update the kubeadm config map Restart kubelet Kubectl uncordon $node @_suvzz
  8. 8. Problems • Backing off of containers! • Etcd: possible errors pid locked by last etcd and/or deprecated API endpoints. • Apiserver: usually when etcd unhealthy and/or deprecated flags. • Mirror pod hash race condition between etcd and apiserver which make upgrade to fail in the older versions. PR:61942 I have faced this problem in v1.9 and was fixed with retrying couple of times. • (Version < v1.11 )if kubelet fails to understand the node ip or if you are using private IP, use –-node-ip flag to explicitly mention the IP. • Explicitly mention --cgroup-driver to kubelet (in my case: cgroupfs but was detected as systemd) @_suvzz
  9. 9. Resources • Godoc: (I use this often to check on struct and value types) • CHANGELOGs: (Holy grail for updates) • Upgrade docs in cluster/kubeadm/ @_suvzz
  10. 10. Heads-up for things to look for when upgrading • Change the version on top right of and then manually find the document you need because if you search it will always point you to the latest version. Docs older than v1.9 are not available • Always check the Release notes/Changelog before performing any upgrade operation. Watch out for “[action required]” and/or “Before Upgrading” • Features gate changed from string “” to map {“”} PR:57962 in kubeadm-config. Applicable upgrading from v1.9 to v1.10 • (version v1.10 to v1.11) there has been change in kubeadm config file format MasterConfiguration (v1alpha2) is broken down (in v1alpha3)to InitConfiguration and ClusterConfiguration. Source • (version v1.10 to v1.11) Major changes in the drop-in file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf. Keep an eye here and put in your EXTRA KUBELET ARGS accordingly to the file it belongs. Source • (version v1.11 to v1.12) control plane images don’t require architecture type anymore. Source @_suvzz
  11. 11. Automation ideas • Ansible roles • Warmup role • Changes with every major version of k8s. All the release notes stuff goes in here. • Update role • Common module for all updates. This role runs after the warmup role • Bash helper scripts • script to drain nodes • Script to cordon and uncordon nodes @_suvzz
  12. 12. $ poweroff • The upgrade process is complicated but it worth taking the journey and moving to the latest stable version to enjoy all the cool Graduated stuffs! • Upgrade has taught me a lot about kubernetes. @_suvzz Source