Under the Covers of DynamoDB

6,552 views

Published on

In this session you'll learn about the decisions that went into designing and building DynamoDB, and how it allows you to stay focused on your application while enjoying single digit latencies at any scale. We'll dive deep on how to model data, maintain maximum throughput, and drive analytics against your data, while profiling  real world use cases, tips and tricks from customers running on DynamoDB today.

Published in: Technology
1 Comment
10 Likes
Statistics
Notes
No Downloads
Views
Total views
6,552
On SlideShare
0
From Embeds
0
Number of Embeds
29
Actions
Shares
0
Downloads
177
Comments
1
Likes
10
Embeds 0
No embeds

No notes for slide

Under the Covers of DynamoDB

  1. 1. Under the Covers of DynamoDB Matt Wood Principal Data Scientist @mza
  2. 2. Hello.
  3. 3. Overview 1. Getting started 2. Data modeling 3. Partitioning 4. Replication & Analytics 5. Customer story: Localytics
  4. 4. 1Getting started
  5. 5. DynamoDB is a managedNoSQL database service.Store and retrieve any amount of data. Serve any level of request traffic.
  6. 6. Without the operational burden.
  7. 7. Consistent, predictable performance. Single digit millisecond latency. Backed on solid-state drives.
  8. 8. Flexible data model.Key/attribute pairs. No schema required. Easy to create. Easy to adjust.
  9. 9. Seamless scalability.No table size limits. Unlimited storage. No downtime.
  10. 10. Durable. Consistent, disk only writes.Replication across data centers and availability zones.
  11. 11. Without the operational burden.
  12. 12. Focus on your app.
  13. 13. Two decisions + three clicks = ready for use
  14. 14. Level of throughput Primary keysTwo decisions + three clicks = ready for use
  15. 15. Level of throughput Primary keysTwo decisions + three clicks = ready for use
  16. 16. Provisioned throughput.Reserve IOPS for reads and writes. Scale up for down at any time.
  17. 17. Pay per capacity unit.Priced per hour of provisioned throughput.
  18. 18. Write throughput.Size of item x writes per second $0.0065 for 10 write units
  19. 19. Consistent writes. Atomic increment and decrement.Optimistic concurrency control: conditional writes.
  20. 20. Transactions. Item level transactions only.Puts, updates and deletes are ACID.
  21. 21. Strong or eventual consistency Read throughput.
  22. 22. Strong or eventual consistency Read throughput.Provisioned units = size of item x reads per second $0.0065 per hour for 50 units
  23. 23. Strong or eventual consistency Read throughput.Provisioned units = size of item x reads per second 2 $0.0065 per hour for 100 units
  24. 24. Strong or eventual consistency Read throughput. Same latency expectations. Mix and match at ‘read time’.
  25. 25. Provisioned throughput ismanaged by DynamoDB.
  26. 26. Data is partitioned andmanaged by DynamoDB.
  27. 27. Indexed data storage. $0.25 per GB per month. Tiered bandwidth pricing:aws.amazon.com/dynamodb/pricing
  28. 28. Reserved capacity.Up to 53% for 1 year reservation.Up to 76% for 3 year reservation.
  29. 29. Authentication. Session based to minimize latency.Uses the Amazon Security Token Service. Handled by AWS SDKs. Integrates with IAM.
  30. 30. Monitoring. CloudWatch metrics:latency, consumed read and write throughput, errors and throttling.
  31. 31. Libraries, mappers and mocks. ColdFusion, Django, Erlang, Java, .Net, Node.js, Perl, PHP, Python, Ruby http://j.mp/dynamodb-libs
  32. 32. 2Data modeling
  33. 33. date = 2012-05-16-id = 100 09-00-10 total = 25.00 date = 2012-05-15-id = 101 15-00-11 total = 35.00 date = 2012-05-16-id = 101 12-00-10 total = 100.00
  34. 34. Table date = 2012-05-16-id = 100 09-00-10 total = 25.00 date = 2012-05-15-id = 101 15-00-11 total = 35.00 date = 2012-05-16-id = 101 12-00-10 total = 100.00
  35. 35. date = 2012-05-16-id = 100 09-00-10 total = 25.00 Item date = 2012-05-15-id = 101 15-00-11 total = 35.00 date = 2012-05-16-id = 101 12-00-10 total = 100.00
  36. 36. date = 2012-05-16-id = 100 09-00-10 total = 25.00 Attribute date = 2012-05-15-id = 101 15-00-11 total = 35.00 date = 2012-05-16-id = 101 12-00-10 total = 100.00
  37. 37. Where is the schema?Tables do not require a formal schema. Items are an arbitrarily sized hash.
  38. 38. Indexing.Items are indexed by primary and secondary keys. Primary keys can be composite. Secondary keys are local to the table.
  39. 39. ID Date Total
  40. 40. Hash key ID Date Total
  41. 41. Hash key Range key ID Date Total Composite primary key
  42. 42. Hash key Range key Secondary range key ID Date Total
  43. 43. Programming DynamoDB. Small but perfectly formed API.
  44. 44. CreateTable PutItemUpdateTable GetItemDeleteTable UpdateItemDescribeTable DeleteItemListTables BatchGetItemQuery BatchWriteItemScan
  45. 45. CreateTable PutItemUpdateTable GetItemDeleteTable UpdateItemDescribeTable DeleteItemListTables BatchGetItemQuery BatchWriteItemScan
  46. 46. CreateTable PutItemUpdateTable GetItemDeleteTable UpdateItemDescribeTable DeleteItemListTables BatchGetItemQuery BatchWriteItemScan
  47. 47. Conditional updates.PutItem, UpdateItem, DeleteItem can take optional conditions for operation.UpdateItem performs atomic increments.
  48. 48. One API call, multiple items BatchGet returns multiple items by key.BatchWrite performs up to 25 put or delete operations. Throughput is measured by IO, not API calls.
  49. 49. CreateTable PutItemUpdateTable GetItemDeleteTable UpdateItemDescribeTable DeleteItemListTables BatchGetItemQuery BatchWriteItemScan
  50. 50. Query vs Scan Query returns items by key.Scan reads the whole table sequentially.
  51. 51. Query patterns Retrieve all items by hash key. Range key conditions:==, <, >, >=, <=, begins with, between. Counts. Top and bottom n values. Paged responses.
  52. 52. EXAMPLE 1:Mapping relationships.
  53. 53. Players user_id = location = joined = mza Cambridge 2011-07-04 user_id = location = joined = jeffbarr Seattle 2012-01-20 user_id = location = joined = werner Worldwide 2011-05-15
  54. 54. Players user_id = location = joined = mza Cambridge 2011-07-04 user_id = location = joined = jeffbarr Seattle 2012-01-20 user_id = location = joined = werner Worldwide 2011-05-15Scores user_id = game = score = mza angry-birds 11,000 user_id = game = score = mza tetris 1,223,000 user_id = location = score = werner bejewelled 55,000
  55. 55. Players user_id = location = joined = mza Cambridge 2011-07-04 user_id = location = joined = jeffbarr Seattle 2012-01-20 user_id = location = joined = werner Worldwide 2011-05-15Scores Leader boards user_id = game = score = game = score = user_id = mza angry-birds 11,000 angry-birds 11,000 mza user_id = game = score = game = score = user_id = mza tetris 1,223,000 tetris 1,223,000 mza user_id = location = score = game = score = user_id = werner bejewelled 55,000 tetris 9,000,000 jeffbarr
  56. 56. Players user_id = location = joined = mza Cambridge 2011-07-04 user_id = location = joined = Query for scores jeffbarr Seattle 2012-01-20 by user user_id = location = joined = werner Worldwide 2011-05-15Scores Leader boards user_id = game = score = game = score = user_id = mza angry-birds 11,000 angry-birds 11,000 mza user_id = game = score = game = score = user_id = mza tetris 1,223,000 tetris 1,223,000 mza user_id = location = score = game = score = user_id = werner bejewelled 55,000 tetris 9,000,000 jeffbarr
  57. 57. Players user_id = location = joined = mza Cambridge 2011-07-04 user_id = location = joined = High scores by game jeffbarr Seattle 2012-01-20 user_id = location = joined = werner Worldwide 2011-05-15Scores Leader boards user_id = game = score = game = score = user_id = mza angry-birds 11,000 angry-birds 11,000 mza user_id = game = score = game = score = user_id = mza tetris 1,223,000 tetris 1,223,000 mza user_id = location = score = game = score = user_id = werner bejewelled 55,000 tetris 9,000,000 jeffbarr
  58. 58. EXAMPLE 2:Storing large items.
  59. 59. Unlimited storage.Unlimited attributes per item. Unlimited items per table. Maximum of 64k per item.
  60. 60. Split across items. message =message_id = 1 part = 1 <first 64k> message =message_id = 1 part = 2 <second 64k> joined =message_id = 1 part = 3 <third 64k>
  61. 61. Store a pointer to S3. message =message_id = 1 http://s3.amazonaws.com... message =message_id = 2 http://s3.amazonaws.com... message =message_id = 3 http://s3.amazonaws.com...
  62. 62. EXAMPLE 3:Time series data
  63. 63. Hot and cold tables.April event_id = timestamp = key = 1000 2013-04-16-09-59-01 value event_id = timestamp = key = 1001 2013-04-16-09-59-02 value event_id = timestamp = key = 1002 2013-04-16-09-59-02 valueMarch event_id = timestamp = key = 1000 2013-03-01-09-59-01 value event_id = timestamp = key = 1001 2013-03-01-09-59-02 value event_id = timestamp = key =
  64. 64. December January February March April
  65. 65. Archive data. Move old data to S3: lower cost. Still available for analytics.Run queries across hot and cold data with Elastic MapReduce.
  66. 66. 3Partitioning
  67. 67. Uniform workload. Data stored across multiple partitions. Data is primarily distributed by primary key.Provisioned throughput is divided evenly across partitions.
  68. 68. To achieve and maintain full provisioned throughput, spreadworkload evenly across hash keys.
  69. 69. Non-Uniform workload.Might be throttled, even at high levels of throughput.
  70. 70. BEST PRACTICE 1:Distinct values for hash keys. Hash key elements should have a high number of distinct values.
  71. 71. Lots of users with unique user_id.Workload well distributed across hash key.user_id = first_name = last_name = mza Matt Wooduser_id = first_name = last_name = jeffbarr Jeff Barruser_id = first_name = last_name = werner Werner Vogelsuser_id = first_name = last_name = simone Simone Brunozzi ... ... ...
  72. 72. BEST PRACTICE 2:Avoid limited hash key values. Hash key elements should have a high number of distinct values.
  73. 73. Small number of status codes.Unevenly, non-uniform workload.status = date = 200 2012-04-01-00-00-01status = date = 404 2012-04-01-00-00-01 status date = 404 2012-04-01-00-00-01status = date = 404 2012-04-01-00-00-01
  74. 74. BEST PRACTICE 3:Model for even distribution.Access by hash key value should be evenly distributed across the dataset.
  75. 75. Large number of devices.Small number which are much more popular than others. Workload unevenly distributed. mobile_id = access_date = 100 2012-04-01-00-00-01 mobile_id = access_date = 100 2012-04-01-00-00-02 mobile_id = access_date = 100 2012-04-01-00-00-03 mobile_id = access_date = 100 2012-04-01-00-00-04 ... ...
  76. 76. Sample access pattern.Workload randomized by hash key. mobile_id = access_date = 100.1 2012-04-01-00-00-01 mobile_id = access_date = 100.2 2012-04-01-00-00-02 mobile_id = access_date = 100.3 2012-04-01-00-00-03 mobile_id = access_date = 100.4 2012-04-01-00-00-04 ... ...
  77. 77. 4Replication & Analytics
  78. 78. Seamless scale.Scalable methods for data processing.Scalable methods for backup/restore.
  79. 79. Amazon Elastic MapReduce. Managed Hadoop service for data-intensive workflows. aws.amazon.com/emr
  80. 80. create external table items_db (id string, votes bigint, views bigint) stored by org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler tblproperties ("dynamodb.table.name" = "items", "dynamodb.column.mapping" = "id:id,votes:votes,views:views");
  81. 81. select id, likes, viewsfrom items_dborder by views desc;
  82. 82. 5
  83. 83. DynamoDB @ Localytics Mohit Dilawari Director of Engineering @mdilawari
  84. 84. About Localytics• Mobile App Analytics Service• 750+ Million Devices and over 20,000 Apps• Customers Include: …and many more. 84
  85. 85. About the Development Team• Small team of four managing entire AWS infrastructure - 100 EC2 Instances• Experts in BigData• Leveraging Amazons service has been the key to our success• Large scale users of:• SQS• S3• ELB• RDS• Route53• Elastic Cache• EMR …and of course DynamoDB 85
  86. 86. Why DynamoDB?Set it and Forget it 86
  87. 87. Our use-case: Dedup Data• Each datapoint includes a globally unique ID• Mobile traffic over 2G/3G will upload periodic duplicate data• We accept data up to a 28 day window 87
  88. 88. First Design for Dedup tableUnique ID: aaaaaaaaaaaaaaaaaaaaaaaaa333333333333333 Table Name = dedup_table ID aaaaaaaaaaaaaaaaaaaaaaaaa111111111111111 aaaaaaaaaaaaaaaaaaaaaaaaa222222222222222aaaaaaaaaaaaaaaaaaaaaaaaa333333333333333 "Test and Set" in a single operation 88
  89. 89. Optimization One - Data Aging• Partition by Month• Create new table day before the month• Need to keep two months of data 89
  90. 90. Optimization One - Data AgingUnique ID: bbbbbbbbbbbbbbbbbbbbbbbbb333333333333333 Check Previous month: Table Name = March2013_dedup ID aaaaaaaaaaaaaaaaaaaaaaaaa111111111111111 Not Here! aaaaaaaaaaaaaaaaaaaaaaaaa222222222222222 90
  91. 91. Optimization One - Data AgingUnique ID: bbbbbbbbbbbbbbbbbbbbbbbbb333333333333333 Test and Set in current month: Table Name = April2013_dedup ID bbbbbbbbbbbbbbbbbbbbbbbbb111111111111111 bbbbbbbbbbbbbbbbbbbbbbbbb222222222222222 bbbbbbbbbbbbbbbbbbbbbbbbb333333333333333 Inserted 91
  92. 92. Optimization Two• Reduce the index size - Reduces costs• Each item has a 100 byte overhead which is substantial• Combine multiple IDs together to one record• Split each ID into two halveso First half is the key. Second Half is added to the set 92
  93. 93. Optimization Two - Use SetsUnique ID: ccccccccccccccccccccccccccc999999999999999 ccccccccccccccccccccccccccc 999999999999999Prefix Valuesaaaaaaaaaaaaaaaaaaaaaaaaa [111111111111111, 222222222222222, 333333333333333]bbbbbbbbbbbbbbbbbbbbbbbbb [444444444444444, 555555555555555, 666666666666666]ccccccccccccccccccccccccccc [777777777777777, 888888888888888, ] 93
  94. 94. Optimization Three - Combine Months • Go back to a single tablePrefix March2013 April2013aaaaaaaaaa... [111111111111111, 22222222222... [1212121212121212, 3434343434....bbbbbbbbbb... [444444444444444, 555555555.... [4545454545454545, 6767676767.....ccccccccccc... [777777777777777, 888888888... [8989898989898989, 1313131313.... One Operation 1. Delete February2013 Field 2. Check ID in March2013 • Test and Set into April 2013 94
  95. 95. Recap Compare Plans for 20 Billion IDs per monthPlan Storage Read Write Costs Total Savings Costs CostsNaive (after a $8400 0 $4000 $12400year)Data Age $900 $350 $4000 $5250 57%Using Sets $150 $350 $4000 $4500 64%Multiple Months $150 0 $4000 $4150 67% 95
  96. 96. Thank You@mdilawari 96
  97. 97. Summary 1. Getting started 2. Data modeling 3. Partitioning 4. Replication & Analytics 5. Customer story: Localytics
  98. 98. Free tier.
  99. 99. aws.amazon.com/dynamodb
  100. 100. Thank you!matthew@amazon.com@mza

×