SlideShare a Scribd company logo
1 of 33
Download to read offline
Designing Stateful Apps for
Cloud and Kubernetes
Evan Chan — Nov 10, 2020
State is hard!
THE hardest problem in distributed systems
@mathiasverraes
“There are only two hard problems in distributed
systems: 2. Exactly-once delivery 1. Guaranteed
order of messages 2. Exactly-once delivery”
What kind of state?
• Structured
• Semi-structured (logs, JSON, etc.)
• Graphs and networks
• Unstructured
• Config and passwords
• ML models and parameters
Characteristics of State
• Mutable vs Immutable
• Persistence (Temporary? Permanent? How permanent?)
• Availability
• Latency to retrieve and mutate
• Consistency
Stateless Kubernetes —
Just punt the state to the DB!
Kubernetes Stateless
“Stateless App”
RDS
PostGres
Op1 Op2 Op3 Op4 Kubernetes
S3
Kinesis
MongoDB
Container
Stateless - Where is my State
Requests/
Events
Memory
App
ReadOnly Disk Images
Temp local disk
Cloud Storage
— not persistent
“BUT WAIT… I thought stateless will solve all
my problems??”
Observations about Stateless
• A pattern that works for many scenarios
• All state pushed to other services - $$$
• Latency - stateless means every state change involves network
• Recovery - all local state must be recovered
• Many cloud data services are cloud specific (eg Dynamo, Kinesis) - multi-
cloud or moving clouds is huge amount of work
• Keeping state consistent across the cluster can be tricky
Container
Stateless vs Serverless
Requests/
Events
Memory
App
ReadOnly Disk Images
Temp local disk
Function
Temp mem/disk
across invocations
Mem use within
invocation only
Container
Using Local State with Cloud Storage
Requests/
Events
Memory
App
ReadOnly Disk Images
Temp local disk
Cloud Storage (S3?)
— not persistent
Local
State
Local
State
Logs: Reasoning about State
“Stateless App”
Op1 Op2 Op3 Op4 Kubernetes
DB
DB2Event
Checkpoint
Logs: Reasoning about State
• A log of events and mutations are kept
• Checkpoints in the log represent snapshots of state
• Consistent state of system that can be recovered to
• Replaying the log allows predictable reconstruction of state and changes
• The foundation of all modern databases and data systems
Container
Example: FiloDB
Requests/
Events
Memory
App
ReadOnly Disk Images
Temp local disk
Cassandra
Failure: recover state
from Kafka
Column
Cache
Lucene
Kafka
Millions of
samples/sec
Thousands of
chunk writes/
sec
Logs == Reactive/Streams?
“App/Data Processing”Events
Checkpoint
Output Data
Output Commands/
State Changes
Stateful Kubernetes —
Persistent Volumes
Container
Kubernetes Persistent Volumes
Requests/
Events
Memory
App
ReadOnly Disk Images
Cloud Storage
— not persistent
PV
• Persistent
• Survives pod restarts
Kubernetes Persistent Volumes
• Standard POSIX file semantics - just a mounted volume
• Yaml config of volume type, size, replication factor, desired speed, etc.
• Local Persistent Volumes - basically a HDD/SSD
• Networked, Replicated, not shared (single pod attachment only)
• AWS EBS, GCE PD, Azure Disk, Ceph, ScaleIO
• Can be very close to local disk in performance
• Replicated, Shared Network Storage (Multi pod attach)
• AWS EFS, CephFS, GlusterFS, NFS
Sample PV provisioning .yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: slow
provisioner: kubernetes.io/aws-ebs
parameters:
type: io1
iopsPerGB: "10"
fsType: ext4
• Decide on storage characteristics at deploy time!
Kubernetes StatefulSets: State Affinity
Pod 1
Memory
App
ReadOnly Disk Images
PV 1
Pod 2
Memory
App
ReadOnly Disk Images
PV 2
Pod 3
Memory
App
ReadOnly Disk Images
PV 3
Why Stateful Kubernetes?
• Run stateful services and databases yourself - to save $$
• You need local state persisted, or have a large amount of state
• Caching - lower latency
• ML models, iterative data transformations
• Want faster recovery for local state
• Need to work with local files (eg Lucene)
• Design for PVs, 1 abstraction - use on any cloud
Replicated DBs on Kubernetes
Leader Pod
Memory
PostGres L
ReadOnly Disk Images
PV 1
Follower Pod
Memory
PostGres F
ReadOnly Disk Images
PV 2
Kubernetes PVs vs S3
• Persistent, fast Local State
• S3/Remote Storage only for backups
• Kafka/persistent logging eliminated
or reduced
Pod 1
App
ReadOnly Disk Images
PV 1
Memory
Container
Kafka
App
ReadOnly Disk Images
Local DIsk
S3
Local
State
Memory
Local
State
PV
• Persistent logging (Kafka) and
cloud storage both essential
The Power of Replicated File Storage
• Don’t reinvent
distributed
coordination and
replication in every
data system/
database.
• Reuse a solid data
replication system.
Using Replicated Storage as a Building Block
App
RocksDB Lucene
Replicated PV
ML Model
Pod 1
Replicated Local State Using Kubernetes
App Shard 1
RocksDB Lucene
Replicated PV 1
SQLite
• Shard your app, each shard gets replicated storage
• Consistent snapshotting and state for diff parts of your app
Pod 2
App Shard 2
RocksDB Lucene
Replicated PV 2
SQLite
Reactive Event Streaming + Stateful K8s!
Pod 1
App Shard 1
Lucene
Replicated PV 1
SQLite
Pod 2
App Shard 2
RocksDB Lucene
Replicated PV 2
SQLiteRocksDB
Akka Cluster Sharding
Akka Event Sourcing etc.
Shared K8s PV for Machine Learning
• Shared networked
Persistent Volume (FSx
for Lustre)
• Training job writes
files to FSx
• Kubernetes pods
serves models from
FSx
https://aws.amazon.com/blogs/storage/using-high-performance-persistent-storage-for-machine-learning-workloads-on-kubernetes/
PVs vs Cloud Data Services
Cloud Data Services Persistent Volumes
Replication and distribution handled by service/
database
Data in volume is replicated (if replicated PV used).
App needs to shard and handle coordination.
Each database/service has its own APIs Standard POSIX volume
Additional network latency of cloud services
Varies, but options for latency and performance
close to local drives
Each data service has its own consistency and failure-
handling characteristics
All data shared on the same PV has same
consistency & failure
Where State can Live
Type of State Cloud Service Local/Persistent Volume
Structured/SQL
MySQL, PostGres, RedShift,
etc. etc.
SQLite, H2, etc.
Key/Value Cassandra, Redis RocksDB, LMDB, MapDB, etc.
Semi-structured MongoDB, etc. etc.
Unstructured (binary, ML models, etc)
S3 Files on PV
Config K8s ConfigMap K8s ConfigMap
In Conclusion
• Super important:
• Where is your state
• What are its characteristics
• Think about state recovery and failure handling during design phase
• Replicated storage (PVs) is a very useful paradigm for data systems
Thank You
• Evan Chan
• @evanfchan (Twitter)
• @platypus.arts (Instagram)
• https://velvia.github.io/about

More Related Content

What's hot

What's hot (19)

Using ТРСС to study Firebird performance
Using ТРСС to study Firebird performanceUsing ТРСС to study Firebird performance
Using ТРСС to study Firebird performance
 
Containerizing Distributed Pipes
Containerizing Distributed PipesContainerizing Distributed Pipes
Containerizing Distributed Pipes
 
Delivering big content at NBC News with RavenDB
Delivering big content at NBC News with RavenDBDelivering big content at NBC News with RavenDB
Delivering big content at NBC News with RavenDB
 
What's new in Confluent 3.2 and Apache Kafka 0.10.2
What's new in Confluent 3.2 and Apache Kafka 0.10.2 What's new in Confluent 3.2 and Apache Kafka 0.10.2
What's new in Confluent 3.2 and Apache Kafka 0.10.2
 
Building large scale, job processing systems with Scala Akka Actor framework
Building large scale, job processing systems with Scala Akka Actor frameworkBuilding large scale, job processing systems with Scala Akka Actor framework
Building large scale, job processing systems with Scala Akka Actor framework
 
DC/OS: The definitive platform for modern apps
DC/OS: The definitive platform for modern appsDC/OS: The definitive platform for modern apps
DC/OS: The definitive platform for modern apps
 
Introduction to Apache Mesos
Introduction to Apache MesosIntroduction to Apache Mesos
Introduction to Apache Mesos
 
Python & Cassandra - Best Friends
Python & Cassandra - Best FriendsPython & Cassandra - Best Friends
Python & Cassandra - Best Friends
 
Python performance profiling
Python performance profilingPython performance profiling
Python performance profiling
 
Lifting the Blinds: Monitoring Windows Server 2012
Lifting the Blinds: Monitoring Windows Server 2012Lifting the Blinds: Monitoring Windows Server 2012
Lifting the Blinds: Monitoring Windows Server 2012
 
Mobius: C# Language Binding For Spark
Mobius: C# Language Binding For SparkMobius: C# Language Binding For Spark
Mobius: C# Language Binding For Spark
 
Data Stores @ Netflix
Data Stores @ NetflixData Stores @ Netflix
Data Stores @ Netflix
 
Cassandra Introduction & Features
Cassandra Introduction & FeaturesCassandra Introduction & Features
Cassandra Introduction & Features
 
Cassandra Core Concepts - Cassandra Day Toronto
Cassandra Core Concepts - Cassandra Day TorontoCassandra Core Concepts - Cassandra Day Toronto
Cassandra Core Concepts - Cassandra Day Toronto
 
Cassandra Day Denver 2014: Setting up a DataStax Enterprise Instance on Micro...
Cassandra Day Denver 2014: Setting up a DataStax Enterprise Instance on Micro...Cassandra Day Denver 2014: Setting up a DataStax Enterprise Instance on Micro...
Cassandra Day Denver 2014: Setting up a DataStax Enterprise Instance on Micro...
 
Diagnosing Problems in Production - Cassandra
Diagnosing Problems in Production - CassandraDiagnosing Problems in Production - Cassandra
Diagnosing Problems in Production - Cassandra
 
Scaling Database Modernisation with MongoDB - Infosys
Scaling Database Modernisation with MongoDB - InfosysScaling Database Modernisation with MongoDB - Infosys
Scaling Database Modernisation with MongoDB - Infosys
 
ENT101 Embracing the Cloud - AWS re: Invent 2012
ENT101 Embracing the Cloud - AWS re: Invent 2012ENT101 Embracing the Cloud - AWS re: Invent 2012
ENT101 Embracing the Cloud - AWS re: Invent 2012
 
Immutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App DeploymentImmutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App Deployment
 

Similar to Designing Stateful Apps for Cloud and Kubernetes

Containerization - The DevOps Revolution
Containerization - The DevOps RevolutionContainerization - The DevOps Revolution
Containerization - The DevOps Revolution
Yulian Slobodyan
 
Understand oracle real application cluster
Understand oracle real application clusterUnderstand oracle real application cluster
Understand oracle real application cluster
Satishbabu Gunukula
 
Robust Containers by Eric Brewer
Robust Containers by Eric BrewerRobust Containers by Eric Brewer
Robust Containers by Eric Brewer
Docker, Inc.
 
Erlang and the Cloud: A Fractal Approach to Throughput
Erlang and the Cloud: A Fractal Approach to ThroughputErlang and the Cloud: A Fractal Approach to Throughput
Erlang and the Cloud: A Fractal Approach to Throughput
Wooga
 

Similar to Designing Stateful Apps for Cloud and Kubernetes (20)

State of the Container Ecosystem
State of the Container EcosystemState of the Container Ecosystem
State of the Container Ecosystem
 
Containerization - The DevOps Revolution
Containerization - The DevOps RevolutionContainerization - The DevOps Revolution
Containerization - The DevOps Revolution
 
Deliver Big Data, Database and AI/ML as-a-Service anywhere
Deliver Big Data, Database and AI/ML as-a-Service anywhereDeliver Big Data, Database and AI/ML as-a-Service anywhere
Deliver Big Data, Database and AI/ML as-a-Service anywhere
 
Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications
 
Understand oracle real application cluster
Understand oracle real application clusterUnderstand oracle real application cluster
Understand oracle real application cluster
 
Scaling a MeteorJS SaaS app on AWS
Scaling a MeteorJS SaaS app on AWSScaling a MeteorJS SaaS app on AWS
Scaling a MeteorJS SaaS app on AWS
 
Kubernetes Basics - ICP Workshop Batch II
Kubernetes Basics - ICP Workshop Batch IIKubernetes Basics - ICP Workshop Batch II
Kubernetes Basics - ICP Workshop Batch II
 
Containers 101 Meetup - VMs vs Containers
Containers 101 Meetup - VMs vs ContainersContainers 101 Meetup - VMs vs Containers
Containers 101 Meetup - VMs vs Containers
 
CosmosDB for DBAs & Developers
CosmosDB for DBAs & DevelopersCosmosDB for DBAs & Developers
CosmosDB for DBAs & Developers
 
Robust Containers by Eric Brewer
Robust Containers by Eric BrewerRobust Containers by Eric Brewer
Robust Containers by Eric Brewer
 
Cnam azure 2015 storage
Cnam azure 2015  storageCnam azure 2015  storage
Cnam azure 2015 storage
 
To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…
 
MySQL Options in OpenStack
MySQL Options in OpenStackMySQL Options in OpenStack
MySQL Options in OpenStack
 
Using Kubernetes to deliver a “serverless” service
Using Kubernetes to deliver a “serverless” serviceUsing Kubernetes to deliver a “serverless” service
Using Kubernetes to deliver a “serverless” service
 
FOSS4G In The Cloud: Using Open Source to build Cloud based Spatial Infrastru...
FOSS4G In The Cloud: Using Open Source to build Cloud based Spatial Infrastru...FOSS4G In The Cloud: Using Open Source to build Cloud based Spatial Infrastru...
FOSS4G In The Cloud: Using Open Source to build Cloud based Spatial Infrastru...
 
Select Stars: A SQL DBA's Introduction to Azure Cosmos DB (SQL Saturday Orego...
Select Stars: A SQL DBA's Introduction to Azure Cosmos DB (SQL Saturday Orego...Select Stars: A SQL DBA's Introduction to Azure Cosmos DB (SQL Saturday Orego...
Select Stars: A SQL DBA's Introduction to Azure Cosmos DB (SQL Saturday Orego...
 
Lessons learned from writing over 300,000 lines of infrastructure code
Lessons learned from writing over 300,000 lines of infrastructure codeLessons learned from writing over 300,000 lines of infrastructure code
Lessons learned from writing over 300,000 lines of infrastructure code
 
Erlang as a cloud citizen, a fractal approach to throughput
Erlang as a cloud citizen, a fractal approach to throughputErlang as a cloud citizen, a fractal approach to throughput
Erlang as a cloud citizen, a fractal approach to throughput
 
Erlang as a Cloud Citizen
Erlang as a Cloud CitizenErlang as a Cloud Citizen
Erlang as a Cloud Citizen
 
Erlang and the Cloud: A Fractal Approach to Throughput
Erlang and the Cloud: A Fractal Approach to ThroughputErlang and the Cloud: A Fractal Approach to Throughput
Erlang and the Cloud: A Fractal Approach to Throughput
 

More from Evan Chan

Spark Summit 2014: Spark Job Server Talk
Spark Summit 2014:  Spark Job Server TalkSpark Summit 2014:  Spark Job Server Talk
Spark Summit 2014: Spark Job Server Talk
Evan Chan
 

More from Evan Chan (16)

Porting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to RustPorting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to Rust
 
Histograms at scale - Monitorama 2019
Histograms at scale - Monitorama 2019Histograms at scale - Monitorama 2019
Histograms at scale - Monitorama 2019
 
FiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
FiloDB: Reactive, Real-Time, In-Memory Time Series at ScaleFiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
FiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
 
Building a High-Performance Database with Scala, Akka, and Spark
Building a High-Performance Database with Scala, Akka, and SparkBuilding a High-Performance Database with Scala, Akka, and Spark
Building a High-Performance Database with Scala, Akka, and Spark
 
700 Updatable Queries Per Second: Spark as a Real-Time Web Service
700 Updatable Queries Per Second: Spark as a Real-Time Web Service700 Updatable Queries Per Second: Spark as a Real-Time Web Service
700 Updatable Queries Per Second: Spark as a Real-Time Web Service
 
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
Building Scalable Data Pipelines - 2016 DataPalooza SeattleBuilding Scalable Data Pipelines - 2016 DataPalooza Seattle
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
 
FiloDB - Breakthrough OLAP Performance with Cassandra and Spark
FiloDB - Breakthrough OLAP Performance with Cassandra and SparkFiloDB - Breakthrough OLAP Performance with Cassandra and Spark
FiloDB - Breakthrough OLAP Performance with Cassandra and Spark
 
Breakthrough OLAP performance with Cassandra and Spark
Breakthrough OLAP performance with Cassandra and SparkBreakthrough OLAP performance with Cassandra and Spark
Breakthrough OLAP performance with Cassandra and Spark
 
Productionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job ServerProductionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job Server
 
Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015
 
MIT lecture - Socrata Open Data Architecture
MIT lecture - Socrata Open Data ArchitectureMIT lecture - Socrata Open Data Architecture
MIT lecture - Socrata Open Data Architecture
 
OLAP with Cassandra and Spark
OLAP with Cassandra and SparkOLAP with Cassandra and Spark
OLAP with Cassandra and Spark
 
Spark Summit 2014: Spark Job Server Talk
Spark Summit 2014:  Spark Job Server TalkSpark Summit 2014:  Spark Job Server Talk
Spark Summit 2014: Spark Job Server Talk
 
Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)
Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)
Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)
 
Cassandra Day 2014: Interactive Analytics with Cassandra and Spark
Cassandra Day 2014: Interactive Analytics with Cassandra and SparkCassandra Day 2014: Interactive Analytics with Cassandra and Spark
Cassandra Day 2014: Interactive Analytics with Cassandra and Spark
 
Real-time Analytics with Cassandra, Spark, and Shark
Real-time Analytics with Cassandra, Spark, and SharkReal-time Analytics with Cassandra, Spark, and Shark
Real-time Analytics with Cassandra, Spark, and Shark
 

Recently uploaded

一比一原版(NEU毕业证书)东北大学毕业证成绩单原件一模一样
一比一原版(NEU毕业证书)东北大学毕业证成绩单原件一模一样一比一原版(NEU毕业证书)东北大学毕业证成绩单原件一模一样
一比一原版(NEU毕业证书)东北大学毕业证成绩单原件一模一样
A
 
01-vogelsanger-stanag-4178-ed-2-the-new-nato-standard-for-nitrocellulose-test...
01-vogelsanger-stanag-4178-ed-2-the-new-nato-standard-for-nitrocellulose-test...01-vogelsanger-stanag-4178-ed-2-the-new-nato-standard-for-nitrocellulose-test...
01-vogelsanger-stanag-4178-ed-2-the-new-nato-standard-for-nitrocellulose-test...
AshwaniAnuragi1
 
☎️Looking for Abortion Pills? Contact +27791653574.. 💊💊Available in Gaborone ...
☎️Looking for Abortion Pills? Contact +27791653574.. 💊💊Available in Gaborone ...☎️Looking for Abortion Pills? Contact +27791653574.. 💊💊Available in Gaborone ...
☎️Looking for Abortion Pills? Contact +27791653574.. 💊💊Available in Gaborone ...
mikehavy0
 

Recently uploaded (20)

NO1 Best Powerful Vashikaran Specialist Baba Vashikaran Specialist For Love V...
NO1 Best Powerful Vashikaran Specialist Baba Vashikaran Specialist For Love V...NO1 Best Powerful Vashikaran Specialist Baba Vashikaran Specialist For Love V...
NO1 Best Powerful Vashikaran Specialist Baba Vashikaran Specialist For Love V...
 
Dynamo Scripts for Task IDs and Space Naming.pptx
Dynamo Scripts for Task IDs and Space Naming.pptxDynamo Scripts for Task IDs and Space Naming.pptx
Dynamo Scripts for Task IDs and Space Naming.pptx
 
Databricks Generative AI Fundamentals .pdf
Databricks Generative AI Fundamentals  .pdfDatabricks Generative AI Fundamentals  .pdf
Databricks Generative AI Fundamentals .pdf
 
一比一原版(NEU毕业证书)东北大学毕业证成绩单原件一模一样
一比一原版(NEU毕业证书)东北大学毕业证成绩单原件一模一样一比一原版(NEU毕业证书)东北大学毕业证成绩单原件一模一样
一比一原版(NEU毕业证书)东北大学毕业证成绩单原件一模一样
 
Augmented Reality (AR) with Augin Software.pptx
Augmented Reality (AR) with Augin Software.pptxAugmented Reality (AR) with Augin Software.pptx
Augmented Reality (AR) with Augin Software.pptx
 
Fuzzy logic method-based stress detector with blood pressure and body tempera...
Fuzzy logic method-based stress detector with blood pressure and body tempera...Fuzzy logic method-based stress detector with blood pressure and body tempera...
Fuzzy logic method-based stress detector with blood pressure and body tempera...
 
Call for Papers - Journal of Electrical Systems (JES), E-ISSN: 1112-5209, ind...
Call for Papers - Journal of Electrical Systems (JES), E-ISSN: 1112-5209, ind...Call for Papers - Journal of Electrical Systems (JES), E-ISSN: 1112-5209, ind...
Call for Papers - Journal of Electrical Systems (JES), E-ISSN: 1112-5209, ind...
 
Filters for Electromagnetic Compatibility Applications
Filters for Electromagnetic Compatibility ApplicationsFilters for Electromagnetic Compatibility Applications
Filters for Electromagnetic Compatibility Applications
 
01-vogelsanger-stanag-4178-ed-2-the-new-nato-standard-for-nitrocellulose-test...
01-vogelsanger-stanag-4178-ed-2-the-new-nato-standard-for-nitrocellulose-test...01-vogelsanger-stanag-4178-ed-2-the-new-nato-standard-for-nitrocellulose-test...
01-vogelsanger-stanag-4178-ed-2-the-new-nato-standard-for-nitrocellulose-test...
 
☎️Looking for Abortion Pills? Contact +27791653574.. 💊💊Available in Gaborone ...
☎️Looking for Abortion Pills? Contact +27791653574.. 💊💊Available in Gaborone ...☎️Looking for Abortion Pills? Contact +27791653574.. 💊💊Available in Gaborone ...
☎️Looking for Abortion Pills? Contact +27791653574.. 💊💊Available in Gaborone ...
 
Path loss model, OKUMURA Model, Hata Model
Path loss model, OKUMURA Model, Hata ModelPath loss model, OKUMURA Model, Hata Model
Path loss model, OKUMURA Model, Hata Model
 
Developing a smart system for infant incubators using the internet of things ...
Developing a smart system for infant incubators using the internet of things ...Developing a smart system for infant incubators using the internet of things ...
Developing a smart system for infant incubators using the internet of things ...
 
Signal Processing and Linear System Analysis
Signal Processing and Linear System AnalysisSignal Processing and Linear System Analysis
Signal Processing and Linear System Analysis
 
Adsorption (mass transfer operations 2) ppt
Adsorption (mass transfer operations 2) pptAdsorption (mass transfer operations 2) ppt
Adsorption (mass transfer operations 2) ppt
 
Involute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdf
Involute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdfInvolute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdf
Involute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdf
 
Autodesk Construction Cloud (Autodesk Build).pptx
Autodesk Construction Cloud (Autodesk Build).pptxAutodesk Construction Cloud (Autodesk Build).pptx
Autodesk Construction Cloud (Autodesk Build).pptx
 
History of Indian Railways - the story of Growth & Modernization
History of Indian Railways - the story of Growth & ModernizationHistory of Indian Railways - the story of Growth & Modernization
History of Indian Railways - the story of Growth & Modernization
 
handbook on reinforce concrete and detailing
handbook on reinforce concrete and detailinghandbook on reinforce concrete and detailing
handbook on reinforce concrete and detailing
 
UNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptxUNIT 4 PTRP final Convergence in probability.pptx
UNIT 4 PTRP final Convergence in probability.pptx
 
analog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptxanalog-vs-digital-communication (concept of analog and digital).pptx
analog-vs-digital-communication (concept of analog and digital).pptx
 

Designing Stateful Apps for Cloud and Kubernetes

  • 1. Designing Stateful Apps for Cloud and Kubernetes Evan Chan — Nov 10, 2020
  • 2. State is hard! THE hardest problem in distributed systems
  • 3. @mathiasverraes “There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery”
  • 4. What kind of state? • Structured • Semi-structured (logs, JSON, etc.) • Graphs and networks • Unstructured • Config and passwords • ML models and parameters
  • 5. Characteristics of State • Mutable vs Immutable • Persistence (Temporary? Permanent? How permanent?) • Availability • Latency to retrieve and mutate • Consistency
  • 6. Stateless Kubernetes — Just punt the state to the DB!
  • 7. Kubernetes Stateless “Stateless App” RDS PostGres Op1 Op2 Op3 Op4 Kubernetes S3 Kinesis MongoDB
  • 8. Container Stateless - Where is my State Requests/ Events Memory App ReadOnly Disk Images Temp local disk Cloud Storage — not persistent
  • 9. “BUT WAIT… I thought stateless will solve all my problems??”
  • 10. Observations about Stateless • A pattern that works for many scenarios • All state pushed to other services - $$$ • Latency - stateless means every state change involves network • Recovery - all local state must be recovered • Many cloud data services are cloud specific (eg Dynamo, Kinesis) - multi- cloud or moving clouds is huge amount of work • Keeping state consistent across the cluster can be tricky
  • 11. Container Stateless vs Serverless Requests/ Events Memory App ReadOnly Disk Images Temp local disk Function Temp mem/disk across invocations Mem use within invocation only
  • 12. Container Using Local State with Cloud Storage Requests/ Events Memory App ReadOnly Disk Images Temp local disk Cloud Storage (S3?) — not persistent Local State Local State
  • 13. Logs: Reasoning about State “Stateless App” Op1 Op2 Op3 Op4 Kubernetes DB DB2Event Checkpoint
  • 14. Logs: Reasoning about State • A log of events and mutations are kept • Checkpoints in the log represent snapshots of state • Consistent state of system that can be recovered to • Replaying the log allows predictable reconstruction of state and changes • The foundation of all modern databases and data systems
  • 15. Container Example: FiloDB Requests/ Events Memory App ReadOnly Disk Images Temp local disk Cassandra Failure: recover state from Kafka Column Cache Lucene Kafka Millions of samples/sec Thousands of chunk writes/ sec
  • 16. Logs == Reactive/Streams? “App/Data Processing”Events Checkpoint Output Data Output Commands/ State Changes
  • 18. Container Kubernetes Persistent Volumes Requests/ Events Memory App ReadOnly Disk Images Cloud Storage — not persistent PV • Persistent • Survives pod restarts
  • 19. Kubernetes Persistent Volumes • Standard POSIX file semantics - just a mounted volume • Yaml config of volume type, size, replication factor, desired speed, etc. • Local Persistent Volumes - basically a HDD/SSD • Networked, Replicated, not shared (single pod attachment only) • AWS EBS, GCE PD, Azure Disk, Ceph, ScaleIO • Can be very close to local disk in performance • Replicated, Shared Network Storage (Multi pod attach) • AWS EFS, CephFS, GlusterFS, NFS
  • 20. Sample PV provisioning .yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: slow provisioner: kubernetes.io/aws-ebs parameters: type: io1 iopsPerGB: "10" fsType: ext4 • Decide on storage characteristics at deploy time!
  • 21. Kubernetes StatefulSets: State Affinity Pod 1 Memory App ReadOnly Disk Images PV 1 Pod 2 Memory App ReadOnly Disk Images PV 2 Pod 3 Memory App ReadOnly Disk Images PV 3
  • 22. Why Stateful Kubernetes? • Run stateful services and databases yourself - to save $$ • You need local state persisted, or have a large amount of state • Caching - lower latency • ML models, iterative data transformations • Want faster recovery for local state • Need to work with local files (eg Lucene) • Design for PVs, 1 abstraction - use on any cloud
  • 23. Replicated DBs on Kubernetes Leader Pod Memory PostGres L ReadOnly Disk Images PV 1 Follower Pod Memory PostGres F ReadOnly Disk Images PV 2
  • 24. Kubernetes PVs vs S3 • Persistent, fast Local State • S3/Remote Storage only for backups • Kafka/persistent logging eliminated or reduced Pod 1 App ReadOnly Disk Images PV 1 Memory Container Kafka App ReadOnly Disk Images Local DIsk S3 Local State Memory Local State PV • Persistent logging (Kafka) and cloud storage both essential
  • 25. The Power of Replicated File Storage • Don’t reinvent distributed coordination and replication in every data system/ database. • Reuse a solid data replication system.
  • 26. Using Replicated Storage as a Building Block App RocksDB Lucene Replicated PV ML Model
  • 27. Pod 1 Replicated Local State Using Kubernetes App Shard 1 RocksDB Lucene Replicated PV 1 SQLite • Shard your app, each shard gets replicated storage • Consistent snapshotting and state for diff parts of your app Pod 2 App Shard 2 RocksDB Lucene Replicated PV 2 SQLite
  • 28. Reactive Event Streaming + Stateful K8s! Pod 1 App Shard 1 Lucene Replicated PV 1 SQLite Pod 2 App Shard 2 RocksDB Lucene Replicated PV 2 SQLiteRocksDB Akka Cluster Sharding Akka Event Sourcing etc.
  • 29. Shared K8s PV for Machine Learning • Shared networked Persistent Volume (FSx for Lustre) • Training job writes files to FSx • Kubernetes pods serves models from FSx https://aws.amazon.com/blogs/storage/using-high-performance-persistent-storage-for-machine-learning-workloads-on-kubernetes/
  • 30. PVs vs Cloud Data Services Cloud Data Services Persistent Volumes Replication and distribution handled by service/ database Data in volume is replicated (if replicated PV used). App needs to shard and handle coordination. Each database/service has its own APIs Standard POSIX volume Additional network latency of cloud services Varies, but options for latency and performance close to local drives Each data service has its own consistency and failure- handling characteristics All data shared on the same PV has same consistency & failure
  • 31. Where State can Live Type of State Cloud Service Local/Persistent Volume Structured/SQL MySQL, PostGres, RedShift, etc. etc. SQLite, H2, etc. Key/Value Cassandra, Redis RocksDB, LMDB, MapDB, etc. Semi-structured MongoDB, etc. etc. Unstructured (binary, ML models, etc) S3 Files on PV Config K8s ConfigMap K8s ConfigMap
  • 32. In Conclusion • Super important: • Where is your state • What are its characteristics • Think about state recovery and failure handling during design phase • Replicated storage (PVs) is a very useful paradigm for data systems
  • 33. Thank You • Evan Chan • @evanfchan (Twitter) • @platypus.arts (Instagram) • https://velvia.github.io/about