Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rancher 2.x first step before deep dive

1,111 views

Published on

https://alterbooth.connpass.com/event/109272/
での発表資料です

Published in: Technology
  • Be the first to comment

Rancher 2.x first step before deep dive

  1. 1. Rancher 2.X First Step before Deep Dive LINE Corporation, Verda2 Yuki Nishiwaki
  2. 2. Who you are? Name: ● Yuki Nishiwaki Working Experience: ● Private Cloud Development (OpenStack) Jan 2015 -> Now ● Kubernetes as a Service Development (Rancher) July 2018 -> Now ● Function as a Service Development (Knative) Sep 2018 -> Now
  3. 3. Rancher and Me ● User and Developer ● Have contributed few patches to rancher 2.X related projects ○ rancher/types ■ https://github.com/rancher/types/pull/525 ○ rancher/machine ■ https://github.com/rancher/machine/pull/12 ○ rancher/norman ■ https://github.com/rancher/norman/pull/201 ■ https://github.com/rancher/norman/pull/202 ■ https://github.com/rancher/norman/pull/203 ○ rancher/rancher ■ https://github.com/rancher/rancher/pull/15909 ■ https://github.com/rancher/rancher/pull/15991 ■ https://github.com/rancher/rancher/pull/16044 Not so many contributions yet
  4. 4. Today’s scope ● We have only 20-30 min ● Rancher have bunch of features. It will take half of day if I talk all parts ● Try to explain whole picture ● Check following article for more detail ○ https://www.slideshare.net/linecorp/lets-unbox-rancher-20-v200 ○ https://github.com/ukinau/rancher-analyse
  5. 5. Rancher 2.X ? All Data for Rancher are stored in Kubernetes => Depend on Kubernetes to run Rancher 1 Manage Kubernetes Cluster by multiple Provider ● Google Container Engine ● Amazon EKS ● Microsoft Azure ● Any VM Provider like OpenStack (RKE) 2 RBAC Configuration Configure/Deploy Additional Tools on Cluster Addon Management 4 Resource Resource Resource Admin member RBAC Configuration 3 Sync RBAC configuration
  6. 6. Rancher 2.X
  7. 7. Rancher 2.X architecture API Controller Cluster Agent Node Agent Node Agent Node Agent Cluster Agent Node Agent Node Agent Node Agent Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster
  8. 8. Resource Creation process is same as Kubernetes Resource API Controller ClusterA Watch Kubernetes ClusterReconcile Get latest information from kube-apiserver Check if any difference Between desired and actual states Do something to make actual state desired Reconcile Loop Cluster Agent Node Agent Don’t do actual provisioning
  9. 9. Resource Creation process is same as Kubernetes Resource API Controller ClusterA Watch Kubernetes ClusterReconcile Get latest information from kube-apiserver Check if any difference Between desired and actual states Do something to make actual state desired Reconcile Loop Cluster Agent Node Agent
  10. 10. Rancher API is just kind of proxy for Kubernetes API ClusterA NodeA UserA GroupA ・・・ Create Cluster Store All data as a Kubernetes by using CRD CRD for Cluster CRD for Node CRD for User Call Kubernetes API
  11. 11. Custom Resource Definition(CRD) in Kubernetes? Kubernetes Native Resource Type Custom Resource Type CustomResourceDefinition ConfigMap Pod Nginx App A Nginx Config Cluster Node Cluster Node Cluster A Cluster B Node A Node B Kubernetes allow user to create custom resource type in addition to natively supported resource.
  12. 12. > kubectl get crd clusters.management.cattle.io -o yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: creationTimestamp: 2018-10-26T13:49:37Z generation: 1 name: clusters.management.cattle.io resourceVersion: "1278" selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/clusters.management.cattle.io uid: fa628204-d925-11e8-b840-fa163e305e2c spec: group: management.cattle.io names: kind: Cluster listKind: ClusterList plural: clusters singular: cluster scope: Cluster version: v3 > kubectl get cluster NAME AGE local 1d Example of CRD for Rancher Resource: Cluster CRD for Cluster Cluster Resource
  13. 13. Question: Do we really needed to have API server? Create Create Why not Create Cluster Resource directly ? Resource information tend to be large and some of the attributes are only needed internally and don’t have to be exposed to user. Thus, Rancher mutate resource before try to create resource in k8s to add some internal information and also mutate resource before try to return resource to user in order to drop some attributes. Cluster A Cluster A Cluster A some attributes some attributes Add
  14. 14. Engineer for Rancher believe their approach is common ● Develop API Server Framework for Kubernetes API(CRD) ○ https://github.com/rancher/norman ● Support ○ Generate API function based on CRD schema ○ Determine path based on CRD schema ○ Override function to do some additional work like overriding attributes, drop attributes…
  15. 15. Resource Creation process is same as Kubernetes Resource API Controller ClusterA Watch Kubernetes ClusterReconcile Get latest information from kube-apiserver Check if any difference Between desired and actual states Do something to make actual state desired Reconcile Loop Cluster Agent Node Agent
  16. 16. Rancher Controllers Management Controllers Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster User Controllers User Controllers User Controllers Cluster Controller Node Controller Catalog Controller ・・・・ Alert Controller RABC Controller Endpoint Controller ・・・・
  17. 17. ● Run 1 Controller in a Rancher Server ● Do something across Kubernetes Clusters deployed ○ Catalog for Rancher User to deploy helm chart onto Kubernetes Clusters ○ Authentication for Rancher ○ Node Create/Update/Remove ○ Cluster Create/Update/Remove ○ Start User Controller for New Cluster Management Controllers Management Controllers Kubernetes Cluster Kubernetes Cluster
  18. 18. User Controllers ● Run 1 Controller for each Kubernetes Cluster deployed ● Do something for assigned Kubernetes Cluster ○ Sync data between Rancher Configuration and Kubernetes Cluster deployed ○ Propagate event happening in Kubernetes Cluster deployed to Rancher ○ Watch all workload(Pod...) in Kubernetes Cluster deployed and Do something User Controllers Kubernetes Cluster Kubernetes Cluster
  19. 19. After deployed, How to check/update Node/Cluster state? kube-apiserver rancher-server management controller user controller kubelet Pod A Pod B Target Resources Any New Pod? Anything Change in existing Pod? Pod A and B is alive! rancher cluster agent Target ResourcesTarget Resources Container (kubelet) File (certificates) Any change in Container, Files? Work as a TCP Proxy Please check my internal state as you want rancher node agent api-server
  20. 20. After deployed, How to check/update Node/Cluster state? kube-apiserver rancher-server management controller user controller kubelet Pod A Pod B Target Resources Any New Pod? Anything Change in existing Pod? Pod A and B is alive! rancher cluster agent Target ResourcesTarget Resources Container (kubelet) File (certificates) Any change in Container, Files? Work as a TCP Proxy Please check my internal state as you want rancher node agent api-server
  21. 21. Agent TCP Proxy is used when you call k8s API
  22. 22. Please look at endpoint closely https://<rancher-server-endpoint>/k8s/cluster/local All requests against k8s cluster are sent to Rancher first and Rancher proxy it into correct cluster
  23. 23. How Rancher proxy Kubernetes Cluster rancher cluster agent Kubernetes Cluster rancher cluster agent User can not call Kubernetes API While cluster-agent is down If rancher cluster agent got down for some reason
  24. 24. Rancher 2.X Updates
  25. 25. Rancher 2.0 -> 2.1: Support multiple Rancher Server Kubernetes Cluster Kubernetes Cluster LB Rancher Server play very very important role, if it is down, following features are not available 1. Create/Update/Delete Node, Cluster 2. All management function for Node, Cluster 3. Proxy Kubernetes API request to Cluster
  26. 26. Rancher 2.0 -> 2.1: Support multiple Rancher Server Kubernetes Cluster Kubernetes Cluster LB Rancher 2.0 is not allowed operator to run multiple rancher server because of websocket session handling logic.
  27. 27. Rancher 2.0 -> 2.1: Support multiple Rancher Server Kubernetes Cluster Kubernetes Cluster LB Leader From 2.1, Rancher’s websocket session handling logic got improved and now they allow us to run multiple rancher server for high availability and scalability.
  28. 28. Rancher 2.0 -> 2.X: Move User Controllers to... User Controllers User Controllers User Controllers Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster CPU/Memory Usage increase every time create cluster ・・・・ ・ ・ Management Controllers
  29. 29. Rancher 2.0 -> 2.X: Move User Controllers to... Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster User Controllers User Controllers User Controllers Management Controllers CPU/Memory increasing ratio by new cluster got improved than before because new user controller for new cluster will be running on new cluster itself

×