Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NoSQL like there is No Tomorrow


Published on

The AWS NoSQL team shares the design philosophy behind DynamoDB and lessons learned in building for massive scale.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

NoSQL like there is No Tomorrow

  1. 1. NoSQL like there is NoTomorrow Khawaja Engineering Lead for NoSQL, AWS @swami_79 @ksshams
  2. 2. NASA JPL has visited every planet in the @ksshams solar system ... except Pluto
  3. 3. 25 Gbps!
  4. 4. @ksshams
  5. 5. let’s start with a trilogy … @swami_79 @ksshams
  6. 6. episode 1 once upon a time... (in 2000) @swami_79 @ksshams
  7. 7. a half mile away... (Seattle) @swami_79 @ksshams
  8. 8. - a rapidly growing Internet based retail business relied on relational databases @swami_79 @ksshams
  9. 9. we had 1000s of independent services @swami_79 @ksshams
  10. 10. each service managed its own state in Relational Databases @swami_79 @ksshams
  11. 11. Relational Databases are pretty seductive @swami_79 @ksshams
  12. 12. first of all... SQL!! @swami_79 @ksshams
  13. 13. so it is easier to query.. @swami_79 @ksshams
  14. 14. easier to learn @swami_79 @ksshams
  15. 15. They are as versatile as a swiss army knife key-value access complex queries analytics transactions @swami_79 @ksshams
  16. 16. Relational Databases are very similar to Swiss Army Knives @swami_79 @ksshams
  17. 17. sometimes.. swiss army knifes.. can be more than what you bargained for @swami_79 @ksshams
  18. 18. partitioning easy re-partitioning HARD.. @swami_79 @ksshams
  19. 19. so we bought bigger boxes... @swami_79 @ksshams
  20. 20. benchmark new hardware migrate to new hardware Q4 was hard-work at Amazon repartition databases pray ... @swami_79 @ksshams
  21. 21. Relational Databases have availability challenges.. @swami_79 @ksshams
  22. 22. episode 2 then.. (in 2005) @swami_79 @ksshams
  23. 23. amazon dynamo predecessor to dynamoDB replicated DHT with consistent hashing optimistic replication “sloppy quorum” anti-entropy mechanism object versioning specialist tool : •limited querying capabilities •simpler consistency @swami_79 @ksshams
  24. 24. dynamo had many benefits • higher availability • we traded it off for consistency • incremental scalability • no more repartitioning • no need to architect apps for peak • just add boxes • simpler querying model ==>> predictable performance @swami_79 @ksshams
  25. 25. but dynamo was not perfect... lacked strong consistency @swami_79 @ksshams
  26. 26. but dynamo was not perfect... scaling was easier, but... @swami_79 @ksshams
  27. 27. but dynamo was not perfect... steep learning curve @swami_79 @ksshams
  28. 28. but dynamo was not perfect... dynamo was a library ==>> not a service... @swami_79 @ksshams
  29. 29. episode 3 then.. (in 2012) @swami_79 @ksshams
  30. 30. Managed NoSQL Database ADMIN DynamoDB Built for Scale Fast & Predictable Performance @swami_79 @ksshams
  31. 31. DynamoDB “Even though we have years of experience with large, complex NoSQL architectures, we are happy to be finally out of the business of managing it ourselves.” - Don MacAskill, CEO @swami_79 @ksshams
  32. 32. DynamoDB Goals and Philosophies durability and availability scale is our problem easy to use scale in rps consistent and low latencies @swami_79 @ksshams
  33. 33. durability is key… @swami_79 @ksshams
  34. 34. availability is key… @swami_79 @ksshams
  35. 35. scale is our problem, not yours.. @swami_79 @ksshams
  36. 36. Fault Tolerant Design Infrastructure Fails - deal with it! Planning for failures is not easy How do you ensure your recovery strategies work correctly? @swami_79 @ksshams
  37. 37. Byzantine General Problem @swami_79 @ksshams
  38. 38. A simple 2-way replication system of a traditional database… Writes @swami_79 @ksshams
  39. 39. S is dead, need to trigger new replica P is dead, need to promote myself P S @swami_79 @ksshams
  40. 40. Improved Replication: Quorum Writes Quorum: Successful write on a majority @swami_79 @ksshams
  41. 41. Easy? Writes from client X Reads and Writes from client Y Classic Split Brain Issue in Replicated systems leading to lost writes! @swami_79 @ksshams
  42. 42. Building correct distributed systems is not straight forward.. Handle replica failures Handle partial failures of replicas Handle concurrent failures Ensure there isn’t a parallel quorum @swami_79 @ksshams
  43. 43. Trends in the World Of Databases @swami_79 @ksshams
  44. 44. A decade ago, it was all about the DBAs @swami_79 @ksshams
  45. 45. Last 5 years have been about self service. @swami_79 @ksshams
  46. 46. Today is about managed services. @swami_79 @ksshams
  47. 47. Plan for Success … Plan for Scale @swami_79 @ksshams
  48. 48. @swami_79 @ksshams
  49. 49. @swami_79 @ksshams