Desert Code Camp 2016.1 - Stateful Distributed Systems
•
1 like•223 views
Have you ever built a service where, if you could just maintain a bit of state, the world would be a much simpler place? This presentation will take you on a journey through the wonderful world of stateful services in a distributed environment. It'll introduce you to some basic concepts that will require re-orienting your mind to think about statefulness vs statelessness, some common use cases for stateful services, the pitfalls to avoid, and my own experience building a highly available, high throughput stateful workflow engine. And, as a bonus, learn how you are already using stateful services in production today (and might not know about it) along with easy ways to get started in case you aren't.
Recommended
More Related Content
Similar to Desert Code Camp 2016.1 - Stateful Distributed Systems
Similar to Desert Code Camp 2016.1 - Stateful Distributed Systems (20)
Recently uploaded
Recently uploaded (20)
Desert Code Camp 2016.1 - Stateful Distributed Systems
- 2. Amazon is hiring! To learn more about our Dev Centers: http://bit.ly/phxdevcenters To learn more about current opportunities, email: phx@amazon.com Our mission is to provide the most innovative, scalable, and reliable systems in the world.
- 3. @JoeRawlings
- 5. Outline
- 7. Outline What’s that smell? What’s the meaning of this?
- 8. Outline What’s that smell? What’s the meaning of this? Pics or it didn’t happen.
- 9. Outline What’s that smell? What’s the meaning of this? Pics or it didn’t happen. Swallowing the red pill.
- 10. Outline What’s that smell? What’s the meaning of this? Pics or it didn’t happen. Swallowing the red pill. Challenge accepted!
- 21. What’s the meaning of this?
- 23. Stateful Services You’re already doing it!
- 24. Required to do something useful Stateful Services You’re already doing it!
- 25. Required to do something useful Stateful Services (most of the time) You’re already doing it!
- 26. Required to do something useful Stateful Services (most of the time) You’re already doing it! stateful processing systems share similar concerns
- 27. Required to do something useful Stateful Services (most of the time) You’re already doing it! stateful processing systems share similar concerns (This is where the fun is)
- 30. data intensive systems Stateful Processing Data locality
- 31. data intensive systems Stateful Processing Strong Consistency Data locality
- 32. data intensive systems Stateful Processing Strong Consistency Data locality High performance
- 34. Stateless Services does not care what has happened
- 35. Stateless Services does not care what has happened does not care what has changed
- 37. Needs to care what has happened Stateful Processing
- 38. Needs to care what has happened Needs To care what has changed Stateful Processing
- 39. Needs to care what has happened Needs To care what has changed Divorce Process lifecycle from data lifecycle Stateful Processing
- 52. Pics Or It Didn’t Happen
- 60. Gossip Protocols
- 62. Gossip Protocols Usage dissemination - event / background
- 63. Gossip Protocols Usage dissemination - event / background anti-entropy - data repair
- 64. Gossip Protocols Usage dissemination - event / background anti-entropy - data repair aggregation - compute across the network
- 65. Gossip Protocols Usage dissemination - event / background anti-entropy - data repair aggregation - compute across the network SWIM
- 66. Gossip Protocols Usage dissemination - event / background anti-entropy - data repair aggregation - compute across the network SWIM scalable weakly-consistent infection-style process group membership protocol - membership list (non-faulty processes)
- 67. SWIM Protocol
- 69. SWIM Protocol failure detection 1. ping random process from member list
- 70. SWIM Protocol failure detection 1. ping random process from member list A. Receives Ack, Great! end
- 71. SWIM Protocol failure detection 1. ping random process from member list A. Receives Ack, Great! end B. does not receive ack, goto 2
- 72. SWIM Protocol failure detection 1. ping random process from member list A. Receives Ack, Great! end B. does not receive ack, goto 2 2. send ping request to k processes from member list
- 73. SWIM Protocol failure detection 1. ping random process from member list A. Receives Ack, Great! end B. does not receive ack, goto 2 2. send ping request to k processes from member list 3. k processes try to ping “failed” process
- 74. SWIM Protocol failure detection 1. ping random process from member list A. Receives Ack, Great! end B. does not receive ack, goto 2 2. send ping request to k processes from member list 3. k processes try to ping “failed” process A. K processes respond back to original process about status
- 75. SWIM Protocol failure detection 1. ping random process from member list A. Receives Ack, Great! end B. does not receive ack, goto 2 2. send ping request to k processes from member list 3. k processes try to ping “failed” process A. K processes respond back to original process about status dissemination
- 76. SWIM Protocol failure detection 1. ping random process from member list A. Receives Ack, Great! end B. does not receive ack, goto 2 2. send ping request to k processes from member list 3. k processes try to ping “failed” process A. K processes respond back to original process about status dissemination multicast failure (vol. leave) updates
- 77. SWIM Protocol failure detection 1. ping random process from member list A. Receives Ack, Great! end B. does not receive ack, goto 2 2. send ping request to k processes from member list 3. k processes try to ping “failed” process A. K processes respond back to original process about status dissemination multicast failure (vol. leave) updates multicast new members
- 83. Paxos
- 86. Paxos Proposers, Acceptors, Learners Proposers: Ask Acceptors to Approve Proposals
- 87. Paxos Proposers, Acceptors, Learners Proposers: Ask Acceptors to Approve Proposals Ask Acceptors to accept a proposal & Version
- 88. Paxos Proposers, Acceptors, Learners Proposers: Acceptors: Ask Acceptors to Approve Proposals Ask Acceptors to accept a proposal & Version
- 89. Paxos Proposers, Acceptors, Learners Proposers: Acceptors: Ask Acceptors to Approve Proposals Ask Acceptors to accept a proposal & Version Do not have to approve or accept
- 90. Paxos Proposers, Acceptors, Learners Proposers: Acceptors: Ask Acceptors to Approve Proposals Ask Acceptors to accept a proposal & Version Do not have to approve or accept Only Approve/Accept what has been proposed
- 91. Paxos Proposers, Acceptors, Learners Proposers: Acceptors: Ask Acceptors to Approve Proposals Ask Acceptors to accept a proposal & Version Do not have to approve or accept Only Approve/Accept what has been proposed LEARNERS:
- 92. Paxos Proposers, Acceptors, Learners Proposers: Acceptors: Ask Acceptors to Approve Proposals Ask Acceptors to accept a proposal & Version Do not have to approve or accept Only Approve/Accept what has been proposed LEARNERS: Proposed Value is chosen when a majority of Acceptors have accepted the value
- 95. Paxos Proposers, Acceptors, Learners Phase 1: 1) Proposers propose a value
- 96. Paxos Proposers, Acceptors, Learners Phase 1: 1) Proposers propose a value 2) Acceptors approve value
- 97. Paxos Proposers, Acceptors, Learners Phase 1: 1) Proposers propose a value 2) Acceptors approve value 1) not approve or accept smaller values
- 98. Paxos Proposers, Acceptors, Learners Phase 1: 1) Proposers propose a value 2) Acceptors approve value 1) not approve or accept smaller values 2) send back highest (accepted) version
- 99. Paxos Proposers, Acceptors, Learners Phase 1: 1) Proposers propose a value 2) Acceptors approve value 1) not approve or accept smaller values 2) send back highest (accepted) version Phase 2:
- 100. Paxos Proposers, Acceptors, Learners Phase 1: 1) Proposers propose a value 2) Acceptors approve value 1) not approve or accept smaller values 2) send back highest (accepted) version Phase 2: 1) proposer Receives approval from majority
- 101. Paxos Proposers, Acceptors, Learners Phase 1: 1) Proposers propose a value 2) Acceptors approve value 1) not approve or accept smaller values 2) send back highest (accepted) version Phase 2: 1) proposer Receives approval from majority 2) sends accept! to acceptors with highest version
- 102. Paxos Proposers, Acceptors, Learners Phase 1: 1) Proposers propose a value 2) Acceptors approve value 1) not approve or accept smaller values 2) send back highest (accepted) version Phase 2: 1) proposer Receives approval from majority 2) sends accept! to acceptors with highest version Phase 3:
- 103. Paxos Proposers, Acceptors, Learners Phase 1: 1) Proposers propose a value 2) Acceptors approve value 1) not approve or accept smaller values 2) send back highest (accepted) version Phase 2: 1) proposer Receives approval from majority 2) sends accept! to acceptors with highest version Phase 3: 1) acceptors notify all learners
- 106. consistent hashing Gossip protocols consensus Stateful Processing
- 110. Consistent Hashing
- 111. Consistent Hashing Distributed Hash Tables
- 112. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1)
- 113. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1) Great for lookups given a key
- 114. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1) Great for lookups given a key Issue
- 115. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1) Great for lookups given a key Issue Horizontal scaling
- 116. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1) Great for lookups given a key Issue Horizontal scaling Fault Tolerance
- 117. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1) Great for lookups given a key Issue Horizontal scaling Fault Tolerance hash(object) (mod n) n = number of slots
- 118. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1) Great for lookups given a key Issue Horizontal scaling Fault Tolerance hash(object) (mod n) Remapping keys to more slots n = number of slots
- 119. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1) Great for Key / Value lookups Solution K = keys, n = number of slots
- 120. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1) Great for Key / Value lookups Solution consistent hashing K = keys, n = number of slots
- 121. Consistent Hashing Distributed Hash Tables Convenient, Fast O(1) Great for Key / Value lookups Solution consistent hashing During resize, k / n keys are remapped K = keys, n = number of slots
- 122. Consistent Hashing
- 123. Challenge Accepted!
- 126. Challenges result composition work distribution Stateful Processing
- 127. Challenges result composition work distribution code deployments Stateful Processing
- 128. Challenges result composition work distribution code deployments unbounded data structures Stateful Processing
- 129. Challenges result composition work distribution code deployments unbounded data structures memory management Stateful Processing
- 130. Challenges result composition work distribution code deployments unbounded data structures memory management persistence strategies Stateful Processing
- 131. Challenges result composition work distribution code deployments unbounded data structures memory management persistence strategies concurrency Stateful Processing
- 133. Key Takeaways
- 134. Key Takeaways Stateful and Stateless systems co-exist
- 135. Key Takeaways Stateful and Stateless systems co-exist process lifecycle vs data lifecyCle
- 136. Key Takeaways Stateful and Stateless systems co-exist process lifecycle vs data lifecyCle Stateful services have their place
- 137. Key Takeaways Stateful and Stateless systems co-exist process lifecycle vs data lifecyCle Stateful services have their place Interesting opportunities
- 138. Key Takeaways Stateful and Stateless systems co-exist process lifecycle vs data lifecyCle Stateful services have their place Interesting opportunities But not all is rosy
- 139. Dynamo: Amazon’s Highly Available Key-value Store http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf Papers We Love The Chubby lock service for loosely-coupled distributed systems research.google.com/archive/chubby-osdi06.pdf Paxos made simple http://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf Time, Clocks, and the Ordering of Events in a Distributed System http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf The Google File System http://research.google.com/archive/gfs-sosp2003.pdf
- 140. Resources Caitie McCaffrey’s talks - https://goo.gl/8MSdRz Apache Foundation - http://apache.org/ graphite - https://graphiteapp.org/ elastic - https://www.elastic.co/
- 141. Thanks! Q&A Amazon is hiring! To learn more about our Dev Centers: http://bit.ly/phxdevcenters To learn more about current opportunities, email: phx@amazon.com