Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kubernetes deployment on bare metal with container linux
Similar to Kubernetes deployment on bare metal with container linux (20)
Recently uploaded
Recently uploaded (20)
Kubernetes deployment on bare metal with container linux
- 1. Kubernetes Deployment on Bare Metal with Container Linux 資訊與通訊研究所 Mac Chiang (蔣是文)
- 2. Copyright 2017 ITRI 工業技術研究院 Agenda • Why bare metal? • Why Container Linux? • How to deployment? • Conclusion 2
- 3. Copyright 2017 ITRI 工業技術研究院 Why bare metal? • Hardware can’t support virtualization ▪ CPU Model: Dual Core AMD Opteron(tm) Processor 270 • Better performance ▪ Bare metal vs. VM 3
- 4. Copyright 2017 ITRI 工業技術研究院 Why Container Linux (CoreOS)? • Lightweight Linux • Container optimized OS • Security focused • Auto update • Integrated well with Kubernetes 4
- 5. Copyright 2017 ITRI 工業技術研究院 Agenda • Why bare metal? • Why Container Linux? • How to deployment? • Conclusion 5
- 6. Copyright 2017 ITRI 工業技術研究院 Deployment Approach • Manual Installation (Step by Step) ▪ https://coreos.com/kubernetes/docs/latest/getting-started.html • Matchbox + ignition (Recommended) ▪ https://github.com/coreos/matchbox/tree/master/examples/groups/k8s- install 6
- 7. Copyright 2017 ITRI 工業技術研究院 Environment Deployment Server Node1:Controller,Etcd Node2: Worker Node3: Worker 7
- 8. Copyright 2017 ITRI 工業技術研究院 CoreOS + Kubernetes Steps • Install CoreOS • Setup an etcd cluster • Generate the certificates for Kubernetes components • Deploy a controller (master) node • Deploy worker nodes • Configure kubectl to work with our cluster • Deploy the add-ons ▪ DNS ▪ Dashboard 8
- 9. Copyright 2017 ITRI 工業技術研究院 Install CoreOS • PXE and iPXE ▪ Booting with iPXE ▪ Booting with PXE ▪ Required RAM :1024M+ • Disk ▪ Installing to Disk Boot from PXE or iPXE Boot from ISO Install to Disk coreos-install -d /dev/sda -c cloud-config.yaml 9
- 10. Copyright 2017 ITRI 工業技術研究院 What’s etcd? • Distributed key, value store • Used for configuration and monitoring store • Used for Service discovery • JSON/REST API 10
- 11. Copyright 2017 ITRI 工業技術研究院 Deploy etcd Cluster Single-Node/Development Multi-Node/Production https://coreos.com/os/docs/latest/cluster-architectures.html 11
- 12. Copyright 2017 ITRI 工業技術研究院 What’s flannel? • A virtual network that gives a subnet to each host for use with container runtimes 12
- 13. Copyright 2017 ITRI 工業技術研究院 Deployment Options • MASTER_HOST ▪ Publicly routable IP of master node. a. Worker nodes must be able to reach the master node(s) via this address on port 443 ▪ Multiple master nodes a. Network load balancer b. DNS configure • ETCD_ENDPOINTS ▪ List of etcd machines (http://ip1:port,http://ip2:port,http://ip3:p ort) • POD_NETWORK=10.2.0.0/16 ▪ The flannel overlay network will provide routing to this network. • SERVICE_IP_RANGE=10.3.0.0/24 ▪ The CIDR network to use for service cluster VIPs (Virtual IPs) ▪ Handled by a local kube-proxy service to each host • K8S_SERVICE_IP=10.3.0.1 ▪ The VIP (Virtual IP) address of the Kubernetes API Service. • DNS_SERVICE_IP=10.3.0.10 ▪ The VIP (Virtual IP) address of the cluster DNS service. 13
- 14. Copyright 2017 ITRI 工業技術研究院 Generate Kubernetes TLS Assets • Root CA Public Key ▪ ca.pem • API Server Public & Private Keys ▪ apiserver.pem ▪ apiserver-key.pem • Worker Node Public & Private Keys ▪ ${WORKER_FQDN}-worker.pem ▪ ${WORKER_FQDN}-worker-key.pem • Cluster Admin Public & Private Keys ▪ admin.pem ▪ admin-key.pem https://coreos.com/kubernetes/docs/latest/openssl.html 14
- 15. Copyright 2017 ITRI 工業技術研究院 Deploy Kubernetes Master Node • Configure Service Components ▪ TLS Assets ▪ Network Configuration ▪ Docker Configuration ▪ Create the kubelet Unit ▪ Set Up the kube-* Pod a. kube-apiserver b. kube-proxy c. kube-controller-manager d. kube-scheduler • Start Services ▪ Load Changed Units ▪ Configure flannel Network ▪ Start kubelet ▪ Basic Health Checks 15
- 16. Copyright 2017 ITRI 工業技術研究院 Master TLS Assets • /etc/kubernetes/ssl/ca.pem • /etc/kubernetes/ssl/apiserver.pem • /etc/kubernetes/ssl/apiserver-key.pem 16
- 17. Copyright 2017 ITRI 工業技術研究院 Network & Docker Configuration /etc/flannel/options.env FLANNELD_ETCD_ENDPOINTS=${ETCD_ENDPOINTS} 17
- 18. Copyright 2017 ITRI 工業技術研究院 Kubelet Unit and Kube-* PODs /etc/systemd/system/ kubelet.service /usr/lib/coreos/kubelet-wrapper --pod-manifest-path=/etc/kubernetes/manifests Hyperkube /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-proxy.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml /etc/kubernetes/manifests/kube-scheduler.yaml An all-in-one binary for the Kubernetes server components 18
- 19. Copyright 2017 ITRI 工業技術研究院 Start Services • Load Changed Units • Configure flannel Network • Start kubelet • Basic Health Checks curl http://127.0.0.1:8080/version 19
- 20. Copyright 2017 ITRI 工業技術研究院 Deploy Kubernetes Worker Node • Configure Service Components ▪ TLS Assets ▪ Networking Configuration ▪ Docker Configuration ▪ Create the kubelet Unit ▪ Set Up the kube-proxy Pod ▪ Set Up kubeconfig • Start Services ▪ Load Changed Units ▪ Start kubelet, and flannel 20
- 21. Copyright 2017 ITRI 工業技術研究院 Worker TLS Assets • /etc/kubernetes/ssl/ca.pem • /etc/kubernetes/ssl/${WORKER_FQDN}-worker.pem • /etc/kubernetes/ssl/${WORKER_FQDN}-worker- key.pem 21
- 22. Copyright 2017 ITRI 工業技術研究院 Kubelet Unit and kube-proxy/kubeconfig /etc/systemd/system/ kubelet.service /usr/lib/coreos/kubelet-wrapper --api-servers=https://${MASTER_HOST} --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml --pod-manifest-path=/etc/kubernetes/manifests Hyperkube /etc/kubernetes/manifests/kube-proxy.yaml An all-in-one binary for the Kubernetes server components 22
- 23. Copyright 2017 ITRI 工業技術研究院 Start Services • Load Changed Units • Start kubelet, and flannel 23
- 24. Copyright 2017 ITRI 工業技術研究院 Configure kubectl • Download the kubectl Executable • Configure kubectl ▪ Master server host ▪ Root CA public key ▪ Cluster admin public & private Keys • Verify kubectl Configuration and Connection kubectl get nodes NAME LABELS STATUS X.X.X.X kubernetes.io/hostname=X.X.X.X Ready • Enabling shell autocompletion echo "source <(kubectl completion bash)" >> ~/.bashrc 24
- 25. Copyright 2017 ITRI 工業技術研究院 Deploy the Add-ons • DNS • Dashboard kubectl port-forward kubernetes-dashboard-xxxx 9090 --namespace=kube- system Then visit http://127.0.0.1:9090 in your browser. 25
- 26. Copyright 2017 ITRI 工業技術研究院 Kube Dashboard namespace=kube-system 26
- 27. Copyright 2017 ITRI 工業技術研究院 What’s MatchBox? • HTTP and gRPC service that renders signed Ignition configs, cloud-configs, network boot configs, and metadata to machines to create CoreOS clusters 27
- 28. Copyright 2017 ITRI 工業技術研究院 Machbox workflow https://github.com/coreos/matchbox/blob/master/Documentation/matchbox.md 28
- 29. Copyright 2017 ITRI 工業技術研究院 Matchbox Steps • Get CoreOS • Generate TLS assets • Prepare groups, profiles and ignition files • Setup dnsmasq and matchbox container • Start deployment • Configure kubectl to work with our cluster • Check all PODs and Services 29
- 30. Copyright 2017 ITRI 工業技術研究院 Get CoreOS ./scripts/get-coreos channel version examples/assets/ └── coreos └── 1298.6.0 ├── CoreOS_Image_Signing_Key.asc ├── coreos_production_image.bin.bz2 ├── coreos_production_image.bin.bz2.sig ├── coreos_production_pxe_image.cpio.gz ├── coreos_production_pxe_image.cpio.gz.sig ├── coreos_production_pxe.vmlinuz └── coreos_production_pxe.vmlinuz.sig https://github.com/coreos/matchbox/tree/master/scripts 30
- 31. Copyright 2017 ITRI 工業技術研究院 Generate TLS Assets ./scripts/tls/k8s-certgen -h Usage: k8s-certgen Options: -d DEST Destination for generated files (default: .examples/assets/tls) -s SERVER Reachable Server IP for kubeconfig (e.g. node1.example.com) -m MASTERS Controller Node Names/Addresses in SAN format (e.g. IP.1=10.3.0.1,DNS.1=node1.example.com) -w WORKERS Worker Node Names/Addresses in SAN format (e.g. DNS.1=node2.example.com,DNS.2=node3.example.com) -h Show help 31
- 32. Copyright 2017 ITRI 工業技術研究院 Prepare groups, profiles and ignition examples/ ├── assets │ ├── coreos │ │ ├── 1298.6.0 │ │ └── tls ├── groups │ ├── install.json │ ├── node1.json │ ├── node2.json │ └── node3.json ├── profiles │ ├── install-reboot.json │ ├── k8s-controller.json │ └── k8s-worker.json └──ignition ├── install-reboot.yaml ├── k8s-controller.yaml └── k8s-worker.yaml https://github.com/coreos/matchbox/tree/master/examples/groups/k8s-install 32
- 33. Copyright 2017 ITRI 工業技術研究院 Installation Flow install.json install- reboot.json install- reboot.yaml curl "{{.ignition_endpoint}}?{{.request.r aw_query}}&os=installed" -o ignition.json node1.json k8s- controller.json "selector": { "os": "installed", "mac": "00:26:2d:06:ff:bc" }, k8s- controller.yaml "coreos_channel": "stable", "coreos_version": “1298.6.0", 33
- 34. Copyright 2017 ITRI 工業技術研究院 Setup dnsmasq and matchbox • Dnsmasq docker run --name dnsmasq --cap-add=NET_ADMIN --network="host" -v $PWD/dnsmasq.conf:/etc/dnsmasq.conf:z quay.io/coreos/dnsmasq -d • Matchbox docker run -p 8080:8080 --rm -v $PWD/example:/var/lib/matchbox:Z quay.io/coreos/matchbox:latest -address=0.0.0.0:8080 -log-level=debug Notice: Don’t forget to open firewall port for matchbox(8080), dns, tftp and dhcp 34
- 35. Copyright 2017 ITRI 工業技術研究院 PXE boot time="2017-04-05T07:31:13Z" level=info msg="Starting matchbox HTTP server on 0.0.0.0:8080" time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /boot.ipxe" time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /ipxe?uuid=03000200-0400-0500-0006-000700080009&mac=00- 26-2d-07-00-78&domain=k8s.itri&hostname=WR1-43&serial=To%20Be%20Filled%20By%20O.E.M." time="2017-04-05T07:34:03Z" level=debug msg="Matched an iPXE config" labels=map[uuid:03000200-0400-0500-0006- 000700080009 mac:00:26:2d:07:00:78 domain:k8s.itri hostname:WR1-43 serial:To Be Filled By O.E.M.] profile=install-reboot time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /assets/coreos/current/coreos_production_pxe.vmlinuz" time="2017-04-05T07:34:04Z" level=info msg="HTTP GET /assets/coreos/current/coreos_production_pxe_image.cpio.gz" time="2017-04-05T07:36:29Z" level=info msg="HTTP GET /ignition?uuid=03000200-0400-0500-0006- 000700080009&mac=00-26-2d-07-00-78&os=installed" time="2017-04-05T07:36:29Z" level=debug msg="Matched an Ignition or Fuze template" group=node3 labels=map[uuid:03000200-0400-0500-0006-000700080009 mac:00:26:2d:07:00:78 os:installed] profile=k8s-controller matchbox logs Demo: https://youtu.be/z9eYOuWLc8k 35
- 36. Copyright 2017 ITRI 工業技術研究院 Configure kubectl • Use the generated kubeconfig directly KUBECONFIG=examples/assets/tls/kubeconfig • Overwrite kubeconfig cp examples/assets/tls/kubeconfig ~/.kube/config 36
- 37. Copyright 2017 ITRI 工業技術研究院 Check all PODs and Services [root@centos7 matchbox]# kubectl get po --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system heapster-v1.2.0-4088228293-7vwxd 2/2 Running 0 15h kube-system kube-apiserver-10.201.3.44 1/1 Running 0 15h kube-system kube-controller-manager-10.201.3.44 1/1 Running 0 15h kube-system kube-dns-782804071-j52dv 4/4 Running 0 15h kube-system kube-dns-autoscaler-2715466192-krz0p 1/1 Running 0 15h kube-system kube-proxy-10.201.3.42 1/1 Running 0 15h kube-system kube-proxy-10.201.3.43 1/1 Running 0 15h kube-system kube-proxy-10.201.3.44 1/1 Running 0 15h kube-system kube-scheduler-10.201.3.44 1/1 Running 0 15h kube-system kubernetes-dashboard-3543765157-xj185 1/1 Running 0 15h [root@centos7 matchbox]# kubectl get svc --all-namespaces NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes 10.3.0.1 <none> 443/TCP 15h kube-system heapster 10.3.0.95 <none> 80/TCP 15h kube-system kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 15h kube-system kubernetes-dashboard 10.3.0.66 <none> 80/TCP 15h 37
- 38. Copyright 2017 ITRI 工業技術研究院 Conclusion • Container Linux (CoreOS) is a good choice for bare metal & production • Manual installation vs. Matchbox+ignition • What’s next? ▪ Try it ▪ Join Kubernetes Taiwan User Group ▪ Kubernetes Training Courses and Playground a. https://www.katacoda.com/courses/kubernetes b. https://www.katacoda.com/courses/kubernetes/playground 38
- 39. Thank you! macchiang@itri.org.tw Kubernetes Taiwan User Group