AWS and Amazon.com operate some of the world's largest distributed systems infrastructure and applications. In our past 18 years of operating this infrastructure, we have come to realize that building such large distributed systems to meet the durability, reliability, scalability, and performance needs of AWS requires us to build our services using a few common distributed systems primitives. Examples of these primitives include a reliable method to build consensus in a distributed system, reliable and scalable key-value store, infrastructure for a transactional logging system, scalable database query layers using both NoSQL and SQL APIs, and a system for scalable and elastic compute infrastructure. In this session, we discuss some of the solutions that we employ in building these primitives and our lessons in operating these systems. We also cover the history of some of these primitives; DHTs, transactional logging, materialized views and various other deep distributed systems concepts; how their design evolved over time; and how we continue to scale them to AWS.

1. November 13, 2014 | Las Vegas, NV
Al Vermeulen and Swami Sivasubramanian

3. Trend #1: The race between Computing power
and expectations
Computing systems keep getting more capableBut… expectations are going up even faster

5. Trend #2: Every application is a distributed app
The number of computers is going up fastMany applications are distributedDistributed systems is not a niche field anymore

7. Hardware and software trends
Specialized (expensive) hardware
Built in redundancy
Simple Software
Commodity hardware
Smarter software

8. With commodity hardware and scale -server failures are inevitable! But by using smarter software,
we can build more robust systems
Trend #3: Commodity Hardware and Smarter Software

10. Online
Online
Online
Online
Online

11. Online
Online
Online
Online
Online

12. Online
Online
Online
Online Online

13. Cloud –Elasticity is the new normalThis results in fleets being dynamic

14. Our World

15. Challenges

16. Addressing Distributed Computing challengesprimitives

17. Core Distributed Systems Primitives
Group Membership
Discovery
Metadata Store
Failure Detection
Workflows

18. Group Membership

19. Group Membership–
Amazon RDS Multi-AZ
Amazon ElastiCache Group –List of caches in a Memcachegroup

20. An example… Replica BReplica C
Writes from client AReplica AReplica DNew member in the groupShould I continue to serve reads? Should I start a new quorum? Replica EReplica F
Reads and Writes from client B
Classic Split Brain Issue in Replicated systems leading to lost writes!

21. Group Membership Fundamentals
Addinga new member to the group
Removinga member from the group
Discoveringwhen the group membership changes
Discoveringroles within the group

22. Discovery

23. Discovery

24. Discovery –Configuration File

25. Discovery -DNS

26. Discovery –DNS (cons)

27. Discovery –Gossip Protocol

28. Cons)
Discovery –Gossip Protocol (Con)

29. Discovery –Metadata store/consensusAmazon DynamoDB

30. Metadata Store

31. Metadata Store

32. Metadata Store -what are good characteristics?
Simplicity
Availability
Scalability
Amazon DynamoDB –top choice for metadata storage in Amazon

33. Metadata Store –Lessons Learned

34. Failure Detection

35. Failure Detection -Challenges

36. Failure Detection -Techniques

37. Failure detection: Lessons Learned

38. Workflows

39. Workflow –What is it?
To execute a series of actions asynchronously

40. What is a workflow?

41. What is not a workflow? synchronousasynchronous

42. Workflow –A simple script

43. Workflow –Recommended Approach
Activity 1
Activity 2
Activity 3
Activity 4

44. Workflow –Lessons Learned
Idempotent metadataAmazon Simple Workflow Service

45. What is the underlying problem?
Group Membership
Discovery
Metadata Store
Failure Detection
Workflows

46. Consensus

47. Paxos and consensus
single point
of failure
Paxos at
the bottom
broken

48. Consensus –Lessons Learned

49. consensus..

50. Paxos at Amazon

51. Group Membership
Discovery
Metadata Store
Failure Detection
Workflows
Lock
Management
Amazon Kinesis
Amazon
DynamoDB
Streams
???

52. Summary

53. http://bit.ly/awsevals