Your SlideShare is downloading. ×
NoSQL like there is No Tomorrow
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

NoSQL like there is No Tomorrow


Published on

The AWS NoSQL team shares the design philosophy behind DynamoDB and lessons learned in building for massive scale.

The AWS NoSQL team shares the design philosophy behind DynamoDB and lessons learned in building for massive scale.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • This story will be a little biased because Swami and I are from the NoSQL generation and we really don’t like relational databases
  • more people already know it, etc
  • Swami
    They seriously don’t scale well
    and if you’re not careful and try to use all the features at once, you can get cut in the process
  • it was like playing minesweeper
  • RDBMs focus way too much on consistency - and compromise availability
    And seriously - are they really all that consistent? ever had slaves fall behind? or how about a recent bug we had where all slaves got updates, but the master didn’t take the update
    Like Swiss army knives - there is waaaay too much functionality in the knives. Let’s move it to the app (easier to debug and change).
    Developers often use transactions when they shouldn’t - very annoying - it is like bringing a swiss army knife through a TSA checkpoint. Just foolish
  • Khawaja
  • Swami:
  • K:
  • required action from developers
    Q4 scaling: While not as scary as phase 1, still was work
    Need benchmarking, patching, adding hardware, rebalancing clusters..
  • developers still had to learn to set it up
    if you are building a recommendation system, you want to be learning k-clustering and collaborative filtering algorithms rather than focusing on lamport’s papers
  • Swami:

    like s3 - and it required developers to be admins.

    this is the really key point - our developers wanted a service, not a product. If you look at mongodb, that is what they are doing today.
  • K
  • </K>
  • 2 dc writes
    on disk
    continuous replication
    we should brag about availability
  • multi-region service!
    Regional service
    spans multiple availability zones
    All data is continuously replicated to multiple AZ’s
  • plan for success, plan for scalability
  • Transcript

    • 1. NoSQL like there is NoTomorrow Khawaja Engineering Lead for NoSQL, AWS @swami_79 @ksshams
    • 2. NASA JPL has visited every planet in the @ksshams solar system ... except Pluto
    • 3. 25 Gbps!
    • 4. @ksshams
    • 5. let’s start with a trilogy … @swami_79 @ksshams
    • 6. episode 1 once upon a time... (in 2000) @swami_79 @ksshams
    • 7. a half mile away... (Seattle) @swami_79 @ksshams
    • 8. - a rapidly growing Internet based retail business relied on relational databases @swami_79 @ksshams
    • 9. we had 1000s of independent services @swami_79 @ksshams
    • 10. each service managed its own state in Relational Databases @swami_79 @ksshams
    • 11. Relational Databases are pretty seductive @swami_79 @ksshams
    • 12. first of all... SQL!! @swami_79 @ksshams
    • 13. so it is easier to query.. @swami_79 @ksshams
    • 14. easier to learn @swami_79 @ksshams
    • 15. They are as versatile as a swiss army knife key-value access complex queries analytics transactions @swami_79 @ksshams
    • 16. Relational Databases are very similar to Swiss Army Knives @swami_79 @ksshams
    • 17. sometimes.. swiss army knifes.. can be more than what you bargained for @swami_79 @ksshams
    • 18. partitioning easy re-partitioning HARD.. @swami_79 @ksshams
    • 19. so we bought bigger boxes... @swami_79 @ksshams
    • 20. benchmark new hardware migrate to new hardware Q4 was hard-work at Amazon repartition databases pray ... @swami_79 @ksshams
    • 21. Relational Databases have availability challenges.. @swami_79 @ksshams
    • 22. episode 2 then.. (in 2005) @swami_79 @ksshams
    • 23. amazon dynamo predecessor to dynamoDB replicated DHT with consistent hashing optimistic replication “sloppy quorum” anti-entropy mechanism object versioning specialist tool : •limited querying capabilities •simpler consistency @swami_79 @ksshams
    • 24. dynamo had many benefits • higher availability • we traded it off for consistency • incremental scalability • no more repartitioning • no need to architect apps for peak • just add boxes • simpler querying model ==>> predictable performance @swami_79 @ksshams
    • 25. but dynamo was not perfect... lacked strong consistency @swami_79 @ksshams
    • 26. but dynamo was not perfect... scaling was easier, but... @swami_79 @ksshams
    • 27. but dynamo was not perfect... steep learning curve @swami_79 @ksshams
    • 28. but dynamo was not perfect... dynamo was a library ==>> not a service... @swami_79 @ksshams
    • 29. episode 3 then.. (in 2012) @swami_79 @ksshams
    • 30. Managed NoSQL Database ADMIN DynamoDB Built for Scale Fast & Predictable Performance @swami_79 @ksshams
    • 31. DynamoDB “Even though we have years of experience with large, complex NoSQL architectures, we are happy to be finally out of the business of managing it ourselves.” - Don MacAskill, CEO @swami_79 @ksshams
    • 32. DynamoDB Goals and Philosophies durability and availability scale is our problem easy to use scale in rps consistent and low latencies @swami_79 @ksshams
    • 33. durability is key… @swami_79 @ksshams
    • 34. availability is key… @swami_79 @ksshams
    • 35. scale is our problem, not yours.. @swami_79 @ksshams
    • 36. Fault Tolerant Design Infrastructure Fails - deal with it! Planning for failures is not easy How do you ensure your recovery strategies work correctly? @swami_79 @ksshams
    • 37. Byzantine General Problem @swami_79 @ksshams
    • 38. A simple 2-way replication system of a traditional database… Writes @swami_79 @ksshams
    • 39. S is dead, need to trigger new replica P is dead, need to promote myself P S @swami_79 @ksshams
    • 40. Improved Replication: Quorum Writes Quorum: Successful write on a majority @swami_79 @ksshams
    • 41. Easy? Writes from client X Reads and Writes from client Y Classic Split Brain Issue in Replicated systems leading to lost writes! @swami_79 @ksshams
    • 42. Building correct distributed systems is not straight forward.. Handle replica failures Handle partial failures of replicas Handle concurrent failures Ensure there isn’t a parallel quorum @swami_79 @ksshams
    • 43. Trends in the World Of Databases @swami_79 @ksshams
    • 44. A decade ago, it was all about the DBAs @swami_79 @ksshams
    • 45. Last 5 years have been about self service. @swami_79 @ksshams
    • 46. Today is about managed services. @swami_79 @ksshams
    • 47. Plan for Success … Plan for Scale @swami_79 @ksshams
    • 48. @swami_79 @ksshams
    • 49. @swami_79 @ksshams