Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Demystifying Kubernetes Statefulsets


Published on

A deep-dive on Kubernetes StatefulSets and how to deploy them for various scenarios. A distributed E-Commerce app built on the YERN (YugaByte, Express, ReactJS, NodeJS) stack and running on Kubernetes is used as an example.

Published in: Software
  • Be the first to comment

Demystifying Kubernetes Statefulsets

  1. 1. 1© 2018 All rights reserved. Demystifying Kubernetes StatefulSets Karthik Ranganathan Co-Founder & CTO, YugaByte March 2018
  2. 2. 2© 2018 All rights reserved. About YugaByte Kannan Muthukkaruppan, CEO Nutanix ♦ Facebook ♦ Oracle Karthik Ranganathan, CTO Nutanix ♦ Facebook ♦ Microsoft Mikhail Bautin, Software Architect Clear Story Data ♦ Facebook ♦ D.E.Shaw  Founded Feb 2016  25 employees  Ex-Facebook, Oracle, Nutanix, Google & LinkedIn engineers  Apache HBase committers  Built NoSQL platform at Facebook for: message inbox message search time series spam detection
  3. 3. 3© 2018 All rights reserved. TRANSACTIONAL PLANET-SCALEHIGH PERFORMANCE Distributed ACID Transactions Document-Based, Strongly Consistent Low Latency, Tunable Reads High Throughput CLOUD-NATIVE OPEN SOURCE Apache 2.0 Popular APIs Extended Apache Cassandra, Redis and PostgreSQL (coming soon) Built For The Container Era Self-Healing, Fault-Tolerant Auto Sharding & Rebalancing Global Data Distribution YugaByte DB
  4. 4. 4© 2018 All rights reserved. Inside The Hood - 3 Node Cluster DocDB Storage Engine Purpose-built for ever-growing data, extended from RocksDB yb-master1 yb-master3 yb-master2 YB-Master Manage shard metadata & coordinate cluster-wide ops node1 node3 node2 Global Transaction Manager Tracks ACID txns across multi-row ops, incl. clock skew mgmt. Raft Consensus Replication Highly resilient, used for both data replication & leader election tablet 1’ tablet 1’ yb-tserver1 yb-tserver2 yb-tserver3 tablet 1’ tablet2-leader tablet3-leader tablet1-leaderYB-TServer Stores/serves data in/from tablets (shards) tablet1-follower tablet1-follower tablet3-follower tablet2-follower tablet3-follower tablet2-follower … … …
  5. 5. 5© 2018 All rights reserved. Starting a Cluster Stable network IDs Persistent volumesOrdered operations STATEFUL 1. Start all yb-masters first (usually 3) 2. Start as many yb-tservers as needed
  6. 6. 6© 2018 All rights reserved. Mapping to Kubernetes Controller APIs DEPLOYMENT Handles updates (rolling/recreation) on ReplicaSets Most commonly used REPLICASET Run N identical Pods Can only use ephemeral storage (pod loss = data loss) DAEMONSET Max 1 Pod per node For node-level functions (e.g. monitoring) JOB Ensure N successful completions For batch jobs such as crons NOT FOR STATEFUL APPS
  7. 7. 7© 2018 All rights reserved. StatefulSets API 1. Ordered operations with ordinal index – Startup, scale-up, scale-down, rolling upgrades, termination 2. Stable, unique network ID/name across restarts – Re-spawning a pod will not make the cluster treat it as a new member 3. Stable, persistent storage (linked to ordinal index/name) – Attach same persistent disk to a pod even if it gets rescheduled to new node 4. Mandatory headless service (no single IP) for integrations – No load balancer, smart clients aware of all pods and connect to any Alpha v1.3 Jul 2016 Beta v1.5 Dec 2016 Stable v1.9 Dec 2017 Purpose-built for stateful apps
  8. 8. 8© 2018 All rights reserved. YugaByte DB Deployed as StatefulSets node2node1 node4node3 yb-master StatefulSet yugabytedb yb-master-1 pod yugabytedb yb-master-0 pod yugabytedb yb-master-2 pod yb-tserver StatefulSet tablet 1’ yugabytedb yb-tserver-1 podtablet 1’ yugabytedb yb-tserver-0 pod tablet 1’ yugabytedb yb-tserver-3 podtablet 1’ yugabytedb yb-tserver-2 pod … Local/Remote Persistent Volume Local/Remote Persistent Volume Local/Remote Persistent Volume Local/Remote Persistent Volume yb-masters Headless Service yb-tservers Headless Service App ClientsAdmin Clients
  9. 9. 9© 2018 All rights reserved. YB-Master yb-masters Headless Service yb-master StatefulSet Access the yb-masters service as $name.$namespace.svc.cluster.local 3 pods ui & rpc ports volume mountpath matches fs_data_dirs Headless
  10. 10. 10© 2018 All rights reserved. YB-TServer yb-tservers Headless Service yb-tserver StatefulSet Access the yb-masters service as $name.$namespace.svc.cluster.local 3 pods to begin with, scale as needed ui, rpc, cassandra & redis ports volume mountpath matches fs_data_dirs Headless
  11. 11. 11© 2018 All rights reserved. Other Examples clusters-kubernetes-statefulsets.html
  12. 12. 12© 2018 All rights reserved. Ensuring High Performance LOCAL STORAGE (Beta in latest v1.10) (Stable) REMOTE STORAGE Lower latency, Higher throughput Recommended for workloads that do their own replication Pre-provision outside of K8s Use SSDs for latency-sensitive apps Higher latency, Lower throughput Recommended for workloads do not perform any replication on their own Provision dynamically in K8s Use alongside local storage for cost-efficient tiering
  13. 13. 13© 2018 All rights reserved. Configuring Data Resilience POD ANTI-AFFINITY MULTI-ZONE/REGIONAL/MULTI-REGION POD SCHEDULING Pods of the same type should not be scheduled on the same node Keeps impact of node failures to absolute minimum (Beta in latest v1.10) Multi-Zone - Tolerate zone failures for k8s slave nodes Regional – Tolerate zone failures for both k8s slave and master nodes [GKE only] Multi-Region – Requires federation of k8s clusters
  14. 14. 14© 2018 All rights reserved. A Real-World Example Yugastore – E-Commerce app on the YERN stack Deployed on
  15. 15. 15© 2018 All rights reserved. Kubernetes Deployment Architecture yb-master StatefulSet yugabytedb yb-master-1 pod yugabytedb yb-master-0 pod yugabytedb yb-master-2 pod yb-tserver StatefulSet tablet 1’ yugabytedb yb-tserver-1 podtablet 1’ yugabytedb yb-tserver-0 pod tablet 1’ yugabytedb yb-tserver-2 pod yb-masters Headless Service yb-master-ui LoadBalancer Service yb-tservers Headless Service yugastore Deployment tablet 1’ yugastore yugastore-1 podtablet 1’ yugastore yugastore-0 pod yugastore LoadBalancer Service End User Admin User
  16. 16. 16© 2018 All rights reserved. DEMO
  17. 17. 17© 2018 All rights reserved. Extending StatefulSets with Operators Based on Custom Controllers that have direct access to lower level K8S API Excellent fit for stateful apps requiring human operational knowledge to correctly scale, reconfigure and upgrade while simultaneously ensuring high performance and data resilience Complementary to Helm for packaging CPU usage in the yb-tserver StatefulSet Scale yb-tserver by 1 pod CPU > 80% for 1min and max_threshold not exceeded
  18. 18. 18© 2018 All rights reserved. Summary 1. StatefulSets are getting increasingly more powerful 2. Ensuring high performance and data resilience requires careful planning 3. Operators make day-to-day operations a breeze 4. Community ready to help if you want to get started
  19. 19. 19© 2018 All rights reserved. YugaByte DB is Apache 2.0 Watch/Star us on GitHub!
  20. 20. 20© 2018 All rights reserved. Try on your laptop
  21. 21. 21© 2018 All rights reserved. @YugaByte Thank you