Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kubernetes and Cloud Native Update Q4 2018


Published on

This year’s final set of Kubernetes and Cloud Native meetups just took place. They kicked off in Kitchener-Waterloo on November 29th, and continued in Montreal December 3rd, Ottawa December 4th, Toronto December 5th, and Quebec December 6th. In preparation for the upcoming KubeCon and CloudNativeCon in Seattle, a wide range of open source solutions were discussed and, as always, beer and pizza provided. Ayrat Khayretdinov began each meetup with an update of Kubernetes and the Cloud Native landscape.

Published in: Technology
  • Be the first to comment

Kubernetes and Cloud Native Update Q4 2018

  1. 1. Kubernetes & Cloud Native Toronto Bienvenue ! Welcome!
  2. 2. Thank you to our sponsors!
  3. 3. Joignez-vous au Slack K8s Canada
  4. 4. Seattle! Dec 11-13 Join #kubecon-seattle2018
  5. 5. Aidez-nous ! ● À Montréal, Toronto, Ottawa, Québec, Kitchener-Waterloo ● Soumettez une présentation ● Commanditez ! Rejoignez-nous sur ● Aidez nous à organizer un meetup
  6. 6. Page 8 Intro Archy Solutions Architect & CNCF Ambassador Carol Trang Community Manager
  7. 7. Kubernetes Certification
  8. 8. @cloudops_
  9. 9. Ateliers pratiques ! Montréal et en ligne Deepen your knowledge of containers and microservices and their ecosystems. ● Docker and Kubernetes ● CI/CD ● IaC ● Advanced Docker and Kubernetes ● Machine Learning ● OpenShift ● Kubernetes on Google Cloud ● Kubernetes on Azure ● Kubernetes on AWS
  10. 10. K8s 5000 ft. view
  11. 11. Page 13 Kubernetes - K-10 • • • •
  12. 12. Page Kubernetes K-100
  13. 13. Page ● Kubernetes Addons ● CNI (Container Network Interface) (stable) ● CRI (Container Runtime Interface plugins) (alpha) ● CSI (Container Storage Interface plugins) (alpha) ● Scheduler webhook & multiple (beta) ● Device plugins (e.g GPUs, NICs, FPGAs, InfiniBand)(alpha) ● External Cloud Provider Integrations (beta) ● API Server authn / authz webhooks (stable) Extending Kubernetes Platform K-200
  14. 14. Page ● Initializers & Admission webhook (beta) ● Istio sidecar auto injection via mutating webhook admissions ● API Aggregation (beta) ● kubectl plugins (alpha) ● Example: kubectl ssh, kubectl switch, kubectl ip, kubectl uptime ● CustomResourceDefinitions (beta) ● Operators Framework (Rook, Vault, Prometheus, Kafka) Extending Apps in Kubernetes K-300
  15. 15. K8s 1.12 & 1.13
  16. 16. Page ● The third release in 2018!!! September 28th ● Release link: ● The Kubernetes 1.13, 4th release December 4th!!! Kubernetes 1.12
  17. 17. Cluster Bootstrap
  18. 18. @cloudops_Page Kubernetes The Hard Way 20 Kelsey Hightower Developer advocate
  19. 19. @cloudops_Page Kubernetes The Hard Way 21 1. Provisioning Compute Resources 2. Provisioning the CA and Generating TLS Certificates 3. Generating Kubernetes Configuration Files for Authentication 4. Generating the Data Encryption Config and Key 5. Bootstrapping the etcd Cluster 6. Bootstrapping the Kubernetes Control Plane 7. Bootstrapping the Kubernetes Worker Nodes 8. Configuring kubectl for Remote Access 9. Provisioning Pod Network Routes 10. Deploying the DNS Cluster Add-on
  20. 20. KOPS Kubeadm
  21. 21. Page Kubeadm, Kops and other Deployment tools can now benefit from: ● TLS Bootstrapping (Stable) ● kubelet generates a private key and a CSR for submission to a cluster-level certificate signing process. ● TLS Server Certificate Rotation (Beta) ● In addition to self-signed certificates. Users can now generate a key locally and use it to issue a CSR to the cluster API server for a Certificate Authority certificate, which will be updated when it expires. What’s new in 1.12
  22. 22. Page ● Kubeadm (Stable) !!! ● Stable command-line UX (GA) ● Implementation (GA) ● Configuration file schema (beta) What’s new in 1.13
  23. 23. Page Example kubeadm-config.yaml apiVersion: kind: ClusterConfiguration kubernetesVersion: stable apiServer: certSANs: - "LOAD_BALANCER_DNS" controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" etcd: external: endpoints: - https://ETCD_0_IP:2379 - https://ETCD_1_IP:2379 - https://ETCD_2_IP:2379 caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
  24. 24. Page ● Kubeadm (Stable) !!! ● Stable command-line UX (GA) ● Implementation (GA) ● Configuration file schema (beta) ● Upgrades between minor versions (GA) What’s new in 1.13
  25. 25. Page ● Kubeadm (Stable) !!! ● Stable command-line UX (GA) ● Implementation (GA) ● Configuration file schema (beta) ● Upgrades between minor versions (GA) ● Secure bootstrap Etcd What’s new in 1.13
  26. 26. Page ● Kubeadm (Stable) !!! ● Stable command-line UX (GA) ● Implementation (GA) ● Configuration file schema (beta) ● Upgrades between minor versions (GA) ● Secure bootstrap Etcd ● HA (alpha) kubeadm init kubeadm join --experimental-control-plane What’s new in 1.13
  27. 27. Page HA Kubeadm Topologies ● 3 Master + 3 etcd (Collocated)
  28. 28. Page HA Kubeadm Topologies ● 3 Master + 3 etcd External
  29. 29. Scheduling
  30. 30. Page Current state of scheduling ● Basic scheduling ● DaemonSets ● Nodes Selectors (e.g. Scheduling on nodes with GPU) ● Advanced Scheduling ● Node Affinity Priority ● Custom schedulers ● Taints/tolerations (e.g scenario for Specialized Hardware, Hardware failing (but not failed) ● Disruption budget (Cluster upgrades with stateful workloads) ● Pod Priority and Pre-emption (e.g. Run debuggers during overload) (allows assign priority to specific pods)
  31. 31. Page What’s new in 1.12 SIG Scheduling updates ● Quota by priority - beta ● Allows to set different namespaces to have different priorities, and assign quotas to those namespaces accordingly. This enhances the existing priority and preemption feature that was delivered in Kubernetes 1.11.
  32. 32. Page What’s new in 1.13 SIG Scheduling updates ● Scheduler can be configured to score a subset of the cluster nodes ● Kubernetes scheduler can be configured to only consider a percentage of the nodes, as long as it can find enough feasible nodes in that set. This improves the scheduler’s performance in large clusters.
  33. 33. Container Runtime Interface (CRI)
  34. 34. Page Container Runtime Interface (CRI) 1.7 - GA 36 AVOID LOCK-IN Goal of CRI: ● Remove docker kubelet code of out Kubernetes ● Simplify integration of K8s with other runtimes CRI runtimes ● cri-docker ● rktlet ● cri-o (based on OCI) ● cri-containerd (alpha) ● virtlet (alpha) ● frakti (alpha)
  35. 35. Page What’s new in 1.12 SIG Scheduling updates: ● RuntimeClass - alpha (cluster-scoped runtime properties) ●The runtimeClass is a new field on the PodSpec that enables users to designate the specific runtime they want to use ● E.g. it will allow users to run Docker and Gvisor containers in same Kubernetes cluster and specify specific parameters related to that runtime.
  36. 36. Kubernetes configuration management
  37. 37. Page Sig Nodes update: ● Dynamic audit configuration (alpha) ● Kubectl diff command (beta) What’s new in 1.13
  38. 38. Networking
  39. 39. Page 41 Container Network Interface (CNI) 41 CNI is a specification proposed by CoreOS and adopted by Kubernetes. CNI is currnetly part of CNCF Goal of CNI: ● To make network layer easy pluggable ● CNM is not good option for K8s ● Avoid code duplication Third-party CNI plugins: ● Flannel ● Weave ● Calico ● Contiv and many more
  40. 40. Page Pod-to-Pod Communication (Continues) 42 CloudProvider Networking (kubenet): ● GCE ● AWS (50 host limit) Overlay type: ● Flannel ● Weave Layer 3 via BGP: ● Calico ● Kube-router (new) Mixed ● Canal=Calico+Flannel SDN ● Romana, OpenContrail ● Cisco, Openshift-SDN ● OVS
  41. 41. @cloudops_ Network Policy
  42. 42. Page State of Network Policy in Kubernetes Network Policy is (stable) Kubernetes 1.7 release and above Features: ● Ingress (stable) policies can be defined ● Cross-namespace policies ● Egress (beta)
  43. 43. Page Focus of SIG-Networking was improve to Network Policy features ● Egress - Stable ● Enables administrators to define how network traffic leaves a Pod, this rules added in addition to Ingress Network Policy rules. ● ipBlock - Stable ● ipBlock functionality allows for defining CIDR ranges in NetworkPolicy definitions. What’s new in 1.12
  44. 44. Page Example of egress and ipBlock kind: NetworkPolicy apiVersion: metadata: name: default-block namespace: netpol-test spec: podSelector: matchLabels: role: db egress: - to: - ipBlock: cidr: except: - policyTypes: - Egress
  45. 45. Page Focus of SIG-Networking ● CoreDNS - GA and default What’s new in 1.13
  46. 46. Storage
  47. 47. Page ● K8s has Kubernetes Volume Plugins, however it is challenging adding support for new “in-tree” volume plugins ● CSI makes Kubernetes volume layer truly extensible (Beta) Current state of Storage
  48. 48. Page Sig-Storage contributed some following enhancements: ● Topology-aware dynamic provisioning - Beta ● Topology aware provisioning makes it possible for Kubernetes to more intelligently provision resources. Prevents from situation where a pod can’t start because the storage resources it needs are in a different zone. What’s new in 1.12
  49. 49. Page Sig-Storage contributed some following enhancements: ● Container Storage Interface (CSI) - GA ● Raw block device using persistent volume source (Beta) ● Topology-aware dynamic provisioning (Stable) What’s new in 1.13
  50. 50. Auto Scaling Feature
  51. 51. Page Autoscaling in Kubernetes 55 ● Horizontal Pod Autoscaling (HPA) Based on CPU Based on Memory Based on Custom Metrics ● Vertical Pods Autoscaling (VPA) - alpha ● Cluster Autoscaling
  52. 52. HPA
  53. 53. Page Horizontal Pod Autoscaling (HPA) 57 Kubernetes automatically scales the number of pods in ● Deployment Metrics for autoscaling ● observed CPU utilization ● observed Memory utilization ● application-provided metrics aka Custom Metrics Pod 1 Pod 2 Pod .. Pod N RC / Deployment Autoscaler
  54. 54. Page HPA based on CPU with Metrics Server
  55. 55. Page Maintain a decent load ● If pods are heavily loaded then starting new pods may bring average load down.
  56. 56. Page Maintain a decent load ● If pods are heavily loaded then starting new pods may bring average load down.
  57. 57. Page Maintain a decent load ● If pods are heavily loaded then starting new pods may bring average load down.
  58. 58. Page Maintain a decent load ● If pods are heavily loaded then starting new pods may bring average load down. ● If pods are barely loaded then stopping pods will free some resources and the deployment should still be ok..
  59. 59. Page Maintain a decent load ● If pods are heavily loaded then starting new pods may bring average load down. ● If pods are barely loaded then stopping pods will free some resources and the deployment should still be ok..
  60. 60. Page Maintain a decent load ● If pods are heavily loaded then starting new pods may bring average load down. ● If pods are barely loaded then stopping pods will free some resources and the deployment should still be ok.. ● Specify the target for the load and try to be as close as possible to it.
  61. 61. Example of HPA with custom metrics using Prometheus
  62. 62. Page HPA v2 (Beta) based on Custom Metric using Prometheus apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: my-hpa spec: scaleTargetRef: apiVersion: apps/v1beta1 kind: Deployment name: <object name> minReplicas: 1 maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 - type: Pods pods: metricName:<your pod custom metric> targetAverageValue: 1k - type: Object object: metricName: <any custom metric> target: apiVersion: extensions/v1beta1 kind: <resource type> name: <resource name> targetValue: 10k
  63. 63. VPA
  64. 64. Page Vertical Pod Autoscaling (VPA) How VPA works: ● Resource: CPU/Memory ● “Increasing CPU/Memory resources when necessary” ● Less complicated to design for resource increase ● Harder to autoscale 68
  65. 65. Page VPA Architecture 69
  66. 66. Page VPA Limitations ● alpha, so need testing and tease out edge-cases ● in-place updates (requires support from container runtime) ● usage spikes—how to deal with it best? 70
  67. 67. Page (Proposal) VPA Architecture with in-place update 71
  68. 68. Page Sig-Autoscaling made significant improvements in HPA and VPA ● HPA (Horizontal Pod Autoscaler) ● Scaling via custom metrics (metrics-server) - beta ● Improving scaling algorithm to reach size faster - beta The algorithm used to determine how many pods should be active has been adjusted to improve the time-to-completion ● VPA (Vertical Pod Autoscaler) - beta What’s new in 1.12
  69. 69. Cloud Providers
  70. 70. Page Cloud Providers
  71. 71. Page ● Runtime options: ● Container-Optimized OS with containerd (cos_containerd) - beta ● Gvisor ● Container-native Load Balancing ● Serverless Add-on (knative) ● Managed Istio Kubernetes 1.12 (GCP)
  72. 72. Page ● Support for Azure Virtual Machine Scale Sets (VMSS) ● Cluster autoscaler support (Stable) ● Azure availability zone support (alpha) ● In future AKS will come with VMSS support Kubernetes 1.12 (Azure)
  73. 73. Page Kubernetes 1.13 (AWS) ● AWS ALB ingress controller (alpha) ● EBS CSI driver (alpha)
  74. 74. Page
  75. 75. Page
  76. 76. CNCF Update
  77. 77. @cloudops_
  78. 78. @cloudops_ Keynotes - CNCF Project Update
  79. 79. @cloudops_
  80. 80. Cloud Native Computing Foundation84
  81. 81. Cloud Native Computing Foundation85
  82. 82. Falco A runtime security tool developed by Sysdig, designed to detect anomalous activity and intrusions in Kubernetes
  83. 83. ● Abnormal Behavior Detection for Linux based Containers, Hosts, and Orchestration Platforms ● Commonly referred to “Runtime Security” ● Filter language can easily detect events such as: ○ Shells/processes spawned in a container ○ Unexpected outbound connections ○ Processes listening on unexpected ports ○ Files/binaries changed after container start ○ Container isolation compromised ● Automated action can be taken when abnormal events are detected Falco
  84. 84. Why do you need Falco? ● Image scanning is “point in time” security of choices made by developers ● Need the have ability to detect breakdowns in isolation when containers are running ● Falco can detect comprised: ○ Container isolation (vulnerabilities in container runtimes/Linux kernel) ○ Applications (exploited applications) ○ Orchestration Systems (Exposed dashboards, API ports) ● Enforces best practices & compliance requirements (PCI, SOC, GDPR)
  85. 85. How Falco Works?
  86. 86. Falco Ecosystem Integrations with CNCF projects: - Kubernetes, rkt, containerd, fluentd Other integrations: - Sysdig, Mesos, Marathon Default Rule Set: - Ships with 25 rules around container best practices. Example: -
  87. 87. Example Falco Use Case
  88. 88. Runtime Security Tools Space Proprietary A number of vendors provide runtime security as part of a broader container security product. These products bundle capabilities from multiple security areas - such as image scanning, access control, and firewalling - to create a more extensive security product. - Sysdig Secure: The Falco rules engine is used along with proprietary software to create a SaaS based security product. - Aqua Security - Twistlock Open Source Falco is one component of a complete security tool set for Cloud Native platforms. Other complementary open source projects include Anchore, Clair, Inspec, Cilium, Notary, TUF, SPIFFE, Vault, etc. Each project covers a different area of infrastructure, software build, or runtime security. - Falco
  89. 89. Incubation
  90. 90. Rook: Sandbox -> Incubation CN Orchestrator for distributed storage systems
  91. 91. ● Cloud-Native Storage Orchestrator ● Automates deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management ● Framework for many storage providers and solutions What is rook ?
  92. 92. Rook Design etcdetcd Kubernetes API kubectl etcd Rook Operator Rook Agent Flexvolume Driver Kubelet Rook vol. plugin Attach & Mount Operations Management & Health APINew Object: Volume AttachmentNew Objects: Storage Clusters Storage Pools Object Store File Store Ref:
  93. 93. Rook Design with Ceph Container Container Container Container Volume Claim Volume Claim Rook-agent Rook-agent Rook-agent Rook-agent Rook-agentOperator
  94. 94. ● v0.7 released Feb 21, v0.8 released July 18 ○ 545 commits total ● Instituted formalized project governance policies, added a new maintainer ● Rook Framework for Storage Providers ○ Makes Rook a general cloud-native storage orchestrator ○ Supports multiple new storage solutions with reusable specs, logic, policies ○ CockroachDB and Minio orchestration released in v0.8 ○ NFS, Cassandra, Nexenta, Alluxio ongoing ● Ceph support graduated to Beta maturity ● Automatic horizontal scaling by the Ceph operator ● Improved security model and support for OpenShift ● Numerous other features and improvements 98 Progress Since Sandbox Entry
  95. 95. Adopters: Production Usage 99 There are additional adopters of Rook, especially those with on-premise deployments, that are not ready to share the details of their usage publicly at this time.
  96. 96. Centre of Excellence in Next Generation Networks 100 ● 20 bare-metal nodes providing 100TB, with more being added ● Heterogeneous mix of nodes with high disk density as well as compute-focused nodes ● Several databases, web applications, and a self-hosted file sharing solution “Rook is giving us a big head start in deploying cloud-native Ceph...having an operator that can help deploy and manage Ceph in a cloud-native environment is an ideal us the ability to leverage both the storage and the extra compute capabilities of the storage-dense nodes” Raymond Maika, Cloud Infrastructure Engineer at CENGN
  97. 97. Harbor: Sandbox -> Incubation A trusted container registry that stores, signs, and scans docker images.
  98. 98. @cloudops_ An open source trusted cloud native registry project. HARBOR ™
  99. 99. @cloudops_ What makes a trusted cloud native registry? − Registry features include ■ Docker and Helm Registry ■ Multi-tenant content signing and validation ■ Security and vulnerability analysis ■ Role based access control and LDAP/AD support ■ Image deletion & garbage collection ■ Image replication between instances ■ Internationalization (currently English and Chinese) − Operational experience ■ Deployed in containers ■ Extends, manages, and integrates proven open source components
  100. 100. @cloudops_ Architecture API Routing Core Service (API/Auth/GUI) Image Registry Trusted Content Vulnerability ScanningJob Service Admin Service Harbor components 3rd party components SQL DatabaseKey/Value Storage Harbor integrates multiple open source components to provide a trusted registry. Persistence components Local or Remote Storage (block, file, object) Users (GUI/API) Container Schedulers/Runtimes Consumers LDAP/Active Directory Supporting services HarborPackaging
  101. 101. @cloudops_ Kubernetes Deployment
  102. 102. @cloudops_ Web interface and vulnerability scanning
  103. 103. Graduation
  104. 104. Envoy: Incubation -> Graduation A modern edge and service proxy
  105. 105. Envoy ● A C++ based L4/L7 proxy ● Low memory footprint ● Battle-tested @ Lyft ○ 100+ services ○ 10,000+ VMs ○ 2M req/s Features: ● API driven config updates → no reloads ● Zone-aware load balancing w/ failover ● Traffic routing and splitting ● Health checks, circuit breakers, timeouts, retry budgets, fault injection, … ● HTTP/2 & gRPC ● Transparent proxying ● Designed for observability
  106. 106. Solutions build with Envoy
  107. 107. What’s Next ?
  108. 108. How to learn more about CNCF projects?
  109. 109. Cloud Native Computing Foundation 11 3 2018-19 KubeCon + CloudNativeCon • China – Shanghai: November 14-15, 2018 – General session CFP closed! – Intro and Deep Dive Sessions CFP • North America – Seattle: December 11 - 13, 2018 – CFP open until August 12, 2018 – Intro and Deep Dive Sessions CFP • Europe – Barcelona: May 21 - 23, 2019
  110. 110. Cloud Native Computing Foundation 11 4 2018-19 KubeCon + CloudNativeCon
  111. 111. CNCF Landscape (card mode)
  112. 112. CNCF Landscape
  113. 113. CNCF Trail Map