Your SlideShare is downloading. ×
  • Like
AWS Webcast - Data Modeling and Best Practices for Scaling your Application with Amazon DynamoDB
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

AWS Webcast - Data Modeling and Best Practices for Scaling your Application with Amazon DynamoDB

  • 1,240 views
Published

Amazon DynamoDB is a fully managed, highly scalable distributed database service. In this technical talk, we will deep dive on how to: …

Amazon DynamoDB is a fully managed, highly scalable distributed database service. In this technical talk, we will deep dive on how to:
• Use DynamoDB to build high-scale applications like social gaming, chat, and voting.
• Model these applications using DynamoDB, including how to use building blocks such as conditional writes, consistent reads, and batch operations to build the higher-level functionality such as multi-item atomic writes and join queries.
• Incorporate best practices such as index projections, item sharding, and parallel scan for maximum scalability

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,240
On SlideShare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
14
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Data Modeling and Best Practices in DynamoDB Peter-Mark Verwoerd Solutions Architect
  • 2. Traditional Database Architecture Client Tier App/Web Tier RDBMS
  • 3. One Database for All Workloads • • • • key-value access complex queries transactions analytics Client Tier App/Web Tier RDBMS
  • 4. Cloud Data Tier Architecture Client Tier App/Web Tier Data Tier Search Cache Blob Store NoSQL RDBMS Data Warehouse
  • 5. Workload Driven Data Store Selection rich search key/value simple query hot reads transaction processing logging Data Tier Search Cache Blob Store NoSQL RDBMS Data Warehouse analytics
  • 6. AWS Services for the Data Tier rich search key/value simple query hot reads transaction processing logging Data Tier Amazon CloudSearch Amazon ElastiCache Amazon DynamoDB Amazon RDS Amazon S3 Amazon Redshift analytics
  • 7. Plan • Basics – Basic Game State – Save Games – Social Gaming • Advanced – Social Gaming – Voting – Global Leaderboard
  • 8. Plan • Basics – Basic Game State (conditional writes) – Save Games (hash + range) – Social Gaming (secondary indexes) • Advanced – Social Gaming (transactions) – Voting (write sharding) – Global Leaderboard (scatter-gather query)
  • 9. Basic Game State conditional writes Basics
  • 10. Tic Tac Toe Basics
  • 11. Tic Tac Toe Alice Bob Your App DynamoDB Basics
  • 12. Tic Tac Toe Table Game Table Id O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  • 13. Tic Tac Toe Table Game Table Id O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  • 14. Tic Tac Toe Table Id State IsTie [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics O abecd Item Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  • 15. Tic Tac Toe Table Id Players O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace [ Alice, Bob ] Alice STARTED Data … Alice … … Attribute Basics Winner
  • 16. Tic Tac Toe Table Primary Key Id O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  • 17. Tic Tac Toe Table Id Players O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace [ Alice, Bob ] Alice STARTED Set Basics String Winner Data … Alice … … Number Binary
  • 18. Tic Tac Toe Table { "Data" : [ [ "X", null, "O" ], [ null, "O", null], [ "O", null, "X" ] ] } Id O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  • 19. State Transitions with Conditional Writes Alice Bob DynamoDB Basics
  • 20. State Transitions with Conditional Writes Alice Bob UpdateItem: Top-Right = O Turn = Bob DynamoDB Basics
  • 21. State Transitions with Conditional Writes Alice Bob UpdateItem: Top-Left = X Turn = Alice DynamoDB Basics
  • 22. State Transitions with Conditional Writes Alice Bob (1) DynamoDB Basics Bob (2) Bob (3)
  • 23. State Transitions with Conditional Writes Alice Bob (1) DynamoDB Basics Bob (2) Bob (3)
  • 24. State Transitions with Conditional Writes Alice Bob (1) DynamoDB Basics Bob (2) Bob (3)
  • 25. State Transitions with Conditional Writes Bob (1) Bob (2) State : STARTED, Turn : Bob, Top-Right : O DynamoDB Basics Bob (3)
  • 26. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X Bob (2) Bob (3) Update: Turn : Alice Low-Right : X State : STARTED, Turn : Bob, Top-Right : O DynamoDB Basics Update: Turn : Alice Mid : X
  • 27. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X, Mid: X, Low-Right: X Basics Bob (2) Update: Turn : Alice Mid : X Bob (3) Update: Turn : Alice Low-Right : X DynamoDB
  • 28. Conditional Writes • • Basics Apply an update only if values are as expected Otherwise reject the write
  • 29. Conditional Writes UpdateItem Id=abecd Game Item { Id : abecd, Players : [ Alice, Bob ], State : STARTED, Turn : Bob, Top-Right: O } Basics Updates: { Turn : Alice, Top-Left: X } Expected: { Turn : Bob, Top-Left : null, State : STARTED }
  • 30. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Bob, Top-Right : O Basics Bob (2) DynamoDB Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Bob (3) Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null
  • 31. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Bob, Top-Right : O Basics Bob (2) DynamoDB Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Bob (3) Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null
  • 32. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X Basics Bob (2) DynamoDB Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Bob (3) Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null
  • 33. Save Games hash + range Save Games
  • 34. Save Games Save Games
  • 35. Primary Key Schemas Hash Key Schema Primary Key Id Players O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  • 36. Primary Key Schemas Hash and Range Key Schema Primary Key Id Turn Players Turn State abecd 0 [ Alice, Bob ] Alice STARTED … abecd 1 [ Alice, Bob ] Bob STARTED … abecd 2 [ Alice, Bob ] Alice STARTED … abecd 3 [ Alice, Bob ] Bob STARTED … abecd 4 [ Alice, Bob ] Alice DONE dbace 0 [ Alice, Bob ] Bob STARTED dbace 1 [ Alice, Bob ] Alice STARTED Save Games IsTie Winner Alice Data … …
  • 37. Primary Key Schemas Primary Key Id Turn Players Turn State abecd 0 [ Alice, Bob ] Alice STARTED … abecd 1 [ Alice, Bob ] Bob STARTED … abecd 2 [ Alice, Bob ] Alice STARTED … abecd 3 [ Alice, Bob ] Bob STARTED … abecd 4 [ Alice, Bob ] Alice DONE dbace 0 [ Alice, Bob ] Bob STARTED dbace 1 [ Alice, Bob ] Alice STARTED Save Games IsTie Winner Alice Data … …
  • 38. Primary Key Schemas • Hash-only – Key/value lookups only • Hash and Range – Given a hash key value, query for items by range key – Items are sorted by range key within each hash key Save Games
  • 39. Primary Key Schemas Query WHERE Id=abecd, ORDER BY Turn DESC, LIMIT 2 Primary Key Id Turn Players Turn State abecd 0 [ Alice, Bob ] Alice STARTED … abecd 1 [ Alice, Bob ] Bob STARTED … abecd 2 [ Alice, Bob ] Alice STARTED … abecd 3 [ Alice, Bob ] Bob STARTED … abecd 4 [ Alice, Bob ] Alice DONE dbace 0 [ Alice, Bob ] Bob STARTED dbace 1 [ Alice, Bob ] Alice STARTED Save Games IsTie Winner Alice Data … …
  • 40. Social Gaming local secondary indexes Social Gaming
  • 41. Social Gaming • • • • Host games Invite friends to play Find friends’ games to play See history of games Social Gaming
  • 42. Social Gaming Hash: UserId Range: GameId Attributes: OpponentId, Date, (rest of game state) HostedGame Table UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 Social Gaming
  • 43. Social Gaming • • • • Host games Invite friends to play Find friends’ games to play See history of games Social Gaming
  • 44. Social Gaming: find recent games UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 Query UserId=Alice Social Gaming
  • 45. Query cost • • Provisioned Throughput: Work / sec allowed on your table Capacity Units: Amount of provisioned throughput consumed by an operation Social Gaming
  • 46. Query cost UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 (397 more games for Alice) Social Gaming (1 item = 600 bytes)
  • 47. Query cost UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 (1 item = 600 bytes) (397 more games for Alice) (Items evaluated by Query) (KB per Read Capacity Unit) 400 X 600 / 1024 / 4 (bytes per item) Social Gaming (KB per byte) = 60 Read Capacity Units
  • 48. Local Secondary Indexes • An alternate range key on a table HostedGame Table LocalSecondaryIndex on Date UserId GameId Date UserId Date GameId Carol e23f5a 2013-10-08 Carol 2013-10-08 e23f5a Alice d4e2dc 2013-10-01 Alice 2013-09-27 e9cba3 Alice e9cba3 2013-09-27 Alice 2013-10-01 d4e2dc Alice f6a3bd 2013-10-01 Alice 2013-10-01 f6a3bd Social Gaming
  • 49. Query cost on Local Secondary Indexes UserId … Carol 2013-10-08 e23f5a … Alice (397 older games) 2013-09-27 e9cba3 … Alice 2013-10-01 d4e2dc … Alice Social Gaming GameId Alice Query for the 10 most recent games Date 2013-10-01 f6a3bd …
  • 50. Query cost on Local Secondary Indexes UserId GameId … Carol 2013-10-08 e23f5a … Alice (397 older games) Alice 2013-09-27 e9cba3 … Alice 2013-10-01 d4e2dc … Alice Query for the 10 most recent games Date 2013-10-01 f6a3bd … (Items evaluated by Query) (KB per Read Capacity Unit) 10 X 600 / 1024 / 4 (bytes per item) Social Gaming (KB per byte) = 2 Read Capacity Units
  • 51. Example Local Secondary Indexes • Find 10 recent matches between Alice and Bob Social Gaming
  • 52. Example Local Secondary Indexes • Find 10 recent matches between Alice and Bob – Hash: UserId – Range: OpponentId + Date Query WHERE UserId=Alice AND OpponentAndDate STARTS_WITH “Bob-” LIMIT 10 DESC Social Gaming
  • 53. More example Local Secondary Indexes • Find a host’s matches without an opponent Social Gaming
  • 54. More example Local Secondary Indexes • Find a host’s matches without an opponent – Hash: UserId – Range: UnmatchedDate (sparse index) Query WHERE UserId=Alice LIMIT 10 DESC Social Gaming
  • 55. Local Secondary Index Projections • Choose what attributes are copied into the index – ALL, SPECIFIC, KEYS • • • Substantially cheaper to Query only projection Project the attributes that your use case requires Can make writes cheaper too Social Gaming
  • 56. Write cost for Local Secondary Index • Insert new item – 1 additional write • Setting index range key to / from null – 1 additional write • Updating a projected attribute – 1 additional write • Updating a non-projected attribute – 0 additional writes • Updating the index range key – 2 additional writes Social Gaming
  • 57. Read cost for Query of non-projected attributes • • Regular Query cost + Single-item Get cost for each evaluated item Social Gaming
  • 58. Example Local Secondary Index Projections • Query Alice’s 10 most recent Games UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 Social Gaming
  • 59. Example Local Secondary Index Projections • Query Alice’s 10 most recent Games – Opponent, Winner, (UserId, GameId, Date) – Projected item size from 600 bytes to 40 bytes • Write cost: – 1 Write Capacity Unit for insert, opponent joining, and completion – 0 Write Capacity Units for other state transitions Social Gaming
  • 60. Social Gaming transactions Social Gaming
  • 61. Social Gaming: Friends • • • Query who you are friends with Ask to be friends with someone Acknowledge (or decline) friend request Social Gaming
  • 62. Social Gaming: Friends Hash: UserId Range: FriendId Attributes: Status, Date, etc Friends Table UserId FriendId Status Date … Alice Bob FRIENDS 2013-08-20 … Bob Alice FRIENDS 2013-08-20 … Bob Chuck INCOMING 2013-10-08 … Chuck Bob SENT 2013-10-08 … Social Gaming
  • 63. Becoming Friends: Multi-item Atomic Writes A friend request! UserId Status Bob FRIENDS Bob Alice FRIENDS Bob Chuck INCOMING Chuck Social Gaming FriendId Alice Bob Bob SENT
  • 64. Becoming Friends: Multi-item Atomic Writes UserId Social Gaming Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record FriendId Alice Bob Alice FRIENDS Bob Chuck INCOMING Chuck Bob SENT
  • 65. Becoming Friends: Multi-item Atomic Writes UpdateItem Status=FRIENDS Social Gaming FriendId Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record UserId Alice Bob Alice FRIENDS Bob Chuck FRIENDS Chuck Bob SENT
  • 66. Becoming Friends: Multi-item Atomic Writes UpdateItem Status=FRIENDS Social Gaming FriendId Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record UserId Alice Bob Alice FRIENDS Bob Chuck FRIENDS Chuck Bob FRIENDS
  • 67. When things go wrong Social Gaming
  • 68. Becoming Friends: When things go wrong A friend request! UserId Social Gaming Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record FriendId Alice Bob Alice FRIENDS Bob Chuck INCOMING Chuck Bob SENT
  • 69. Becoming Friends: When things go wrong UpdateItem Status=FRIENDS Social Gaming FriendId Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record UserId Alice Bob Alice FRIENDS Bob Chuck FRIENDS Chuck Bob SENT
  • 70. Becoming Friends: When things go wrong UpdateItem Status=FRIENDS Social Gaming FriendId Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record UserId Alice Bob Alice FRIENDS Bob Chuck FRIENDS Chuck Bob SENT
  • 71. Multi-item transaction in DynamoDB • • • Scan for “stuck” transactions Use the Client Transactions Library on the AWS SDK for Java Roll your own scheme Social Gaming
  • 72. Replayable state machines INCOMING ACCEPTING SENT FRIENDS Bob/ Chuck SENDING Chuck/ Bob Social Gaming FRIENDS
  • 73. Client Transactions Library Bob Transactions Table Social Gaming Transaction Client Transaction Images Table Friends Table
  • 74. Client Transactions Usage • • • • • Low contention only Don’t mix Tx Client writes with normal writes No Query support Expensive, slower But, easy to use Social Gaming
  • 75. Specialized Transactions Transactions Table A friend request! Id Status V1 V2 Bob 1. 2. 3. 4. FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  • 76. Specialized Transactions Transactions Table Id V1 V2 BatchGetItem Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  • 77. Specialized Transactions Transactions Table Id V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 PutItem, Expect not exists Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  • 78. Specialized Transactions Transactions Table Id V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 UpdateItem, Expect V=Vprev Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob FRIENDS 3
  • 79. Specialized Transactions Transactions Table Id DeleteItem, Expect V1=V1prev, V2=V2prev, Bob 1. 2. 3. 4. Status V1 V2 FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob FRIENDS 3
  • 80. When things go wrong Social Gaming
  • 81. Specialized Transactions Transactions Table Id V1 V2 BatchGetItem Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  • 82. Specialized Transactions Transactions Table Id V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 PutItem, Expect not exists Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  • 83. Specialized Transactions Transactions Table Id V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 UpdateItem, Expect V=Vprev Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  • 84. Social Gaming
  • 85. Specialized Transactions Transactions Table Id Status V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 Scan Sweeper FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob 1. Scan for stuck Tx 2. Apply writes 3. Delete from Tx table Bob SENT 2
  • 86. Specialized Transactions Transactions Table Id Status V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 UpdateItem, Expect V=Vprev Sweeper FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob 1. Scan for stuck Tx 2. Apply writes 3. Delete from Tx table Bob FRIENDS 3
  • 87. Specialized Transactions Transactions Table Id DeleteItem, Expect V1=V1prev, V2=V2prev, Sweeper Status V1 V2 FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob 1. Scan for stuck Tx 2. Apply writes 3. Delete from Tx table Bob FRIENDS 3
  • 88. Transaction advice • Lock items before modifying – Including items that don’t exist yet • • • Don’t stomp on future writes (use versions) Sweep for stuck transactions Avoid deadlock Social Gaming
  • 89. High-Throughput Voting write sharding Voting
  • 90. Voting Voter Candidate A Votes: 20 Candidate B Votes: 30 Votes Table Voting
  • 91. Voting Voter UpdateItem ADD 1 to “Candidate A” (aka Atomic Increment) Candidate A Votes: 21 Candidate B Votes: 30 Votes Table Voting
  • 92. Scaling on DynamoDB You Votes Table Voting Need to scale for the election
  • 93. Scaling on DynamoDB You Provision 1200 Write Capacity Units Votes Table Voting
  • 94. Scaling on DynamoDB You Provision 1200 Write Capacity Units Partition 1 Partition 2 600 Write Capacity Units (each) Votes Table Voting
  • 95. Scaling on DynamoDB You Provision 1200 Write Capacity Units Partition 1 Partition 2 (no sharing) Votes Table Voting
  • 96. Scaling on DynamoDB You Provision 200,000 Write Capacity Units Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Votes Table Voting Partition N (600 WCU)
  • 97. Scaling bottlenecks Voters Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Candidate B Candidate A Votes Table Voting Partition N (600 WCU)
  • 98. Scaling bottlenecks Voters Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Candidate B Candidate A Votes Table Voting Partition N (600 WCU)
  • 99. Best Practice: Uniform Workloads “To achieve the full amount of request throughput you have provisioned for a table, keep your workload spread evenly across the hash key values.” – DynamoDB Developer Guide Voting
  • 100. Scaling on DynamoDB Voter Candidate A_1 Candidate A_4 Candidate A_7 Candidate B_5 Candidate B_1 Candidate A_5 Candidate A_2 Candidate B_8 Candidate B_4 Candidate B_3 Candidate B_7 Candidate A_3 Candidate A_6 Candidate A_8 Voting Votes Table Candidate B_2 Candidate B_6
  • 101. Scaling on DynamoDB Voter UpdateItem: “CandidateA_” + rand(0, 10) ADD 1 to Votes Candidate A_1 Candidate A_4 Candidate A_7 Candidate B_5 Candidate B_1 Candidate A_5 Candidate A_2 Candidate B_8 Candidate B_4 Candidate B_3 Candidate B_7 Candidate A_3 Candidate A_6 Candidate A_8 Voting Votes Table Candidate B_2 Candidate B_6
  • 102. Scaling on DynamoDB Periodic Process Voter 2. Store 1. Sum Candidate A_1 Candidate A_4 Candidate A_7 Candidate B_5 Candidate B_1 Candidate A_5 Candidate A_2 Candidate B_8 Candidate B_4 Candidate A Total: 2.5M Candidate B_3 Candidate B_7 Candidate A_3 Candidate A_6 Candidate A_8 Voting Votes Table Candidate B_2 Candidate B_6
  • 103. Scaling Reads Voters Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Candidate A_Total Votes: 2.5M Partition N (600 WCU) Candidate B_Total Votes: 2.1M Votes Table Voting
  • 104. Scaling Reads Voters Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Candidate A_Total Votes: 2.5M Partition N (600 WCU) Candidate B_Total Votes: 2.1M Votes Table Voting
  • 105. Global Leaderboard scatter-gather Leaderboards
  • 106. Game-Wide Leaderboard • Find the top 10 scores game-wide HighScore User 1000 Alice 850 Dave 580 Erin 470 Bob 30 Chuck Leaderboards
  • 107. Game-Wide Leaderboard • Find the top 10 scores game-wide HighScore 1000 Table Schemas must begin with a Hash Key User Alice 850 Dave 580 Erin 470 Bob 30 Chuck Leaderboards
  • 108. Game-Wide Leaderboard • Find the top 10 scores game-wide User Chuck Cannot be Queried the way we want HighScore 20 Alice 1000 Bob 470 Dave 850 Erin 580 Leaderboards
  • 109. Game-Wide Leaderboard • Use a constant Hash key? Constant HighScore-User 1 0001000-Alice 1 0000850-Dave 1 0000580-Erin 1 0000470-Bob 1 0000030-Chuck Leaderboards Zero-pad strings for sort stability
  • 110. Game-Wide Leaderboard • Use a constant Hash key? Constant 1 0001000-Alice 1 Extremely non-uniform workload HighScore-User 0000850-Dave 1 0000580-Erin 1 0000470-Bob 1 0000030-Chuck Leaderboards
  • 111. Scatter-Gather Leading Range Key HighScores Table Shard Shard HighScore-User 2 0000600-Frank 5 0000999-Oscar 2 0000581-Trent 5 0000700-Craig 5 0000030-Chuck 0001000-Alice 1 0000980-Eve HighScore-User 1 HighScore-User 2 Shard 0000850-Dave 1 0000580-Erin Shard HighScore-User 3 0000900-Dan 3 0000850-Wendy 3 0000080-Trent Shard HighScore-User 4 0000500-Merlin 4 0000350-Carole 4 0000280-Paul Leaderboards
  • 112. Scatter-Gather Leading Range Key 1. Periodically Query each Shard DESC, LIMIT N HighScores Table Shard Shard HighScore-User 2 0000600-Frank 5 0000999-Oscar 2 0000581-Trent 5 0000700-Craig 5 0000030-Chuck 0001000-Alice 1 0000980-Eve HighScore-User 1 HighScore-User 2 Shard 0000850-Dave 1 0000580-Erin Shard HighScore-User 3 0000900-Dan 3 0000850-Wendy 3 0000080-Trent Shard HighScore-User 4 0000500-Merlin 4 0000350-Carole 4 0000280-Paul Leaderboards
  • 113. Scatter-Gather Leading Range Key HighScore 1000 Alice 999 2. Keep only the top N, Store somewhere User Oscar HighScores Table Shard Shard HighScore-User 2 0000600-Frank 5 0000999-Oscar 2 0000581-Trent 5 0000700-Craig 5 0000030-Chuck 0001000-Alice 1 0000980-Eve HighScore-User 1 HighScore-User 2 Shard 0000850-Dave 1 0000580-Erin Shard HighScore-User 3 0000900-Dan 3 0000850-Wendy 3 0000080-Trent Shard HighScore-User 4 0000500-Merlin 4 0000350-Carole 4 0000280-Paul Leaderboards
  • 114. Scatter-Gather Leading Range Key User Alice 5 Carole HighScores Table 1 Oscar Store the Shard id by User for high score updates Shard 4 Shard Shard HighScore-User 2 0000600-Frank 5 0000999-Oscar 2 0000581-Trent 5 0000700-Craig 5 0000030-Chuck 0001000-Alice 1 0000980-Eve HighScore-User 1 HighScore-User 2 Shard 0000850-Dave 1 0000580-Erin Shard HighScore-User 3 0000900-Dan 3 0000850-Wendy 3 0000080-Trent Shard HighScore-User 4 0000500-Merlin 4 0000350-Carole 4 0000280-Paul Leaderboards
  • 115. Recap • Basics – Basic Game State (conditional writes) – Save Games (hash + range) – Social Gaming (secondary indexes) • Advanced – Social Gaming (transactions) – Replication (cross-region, cross-data-store) – Global Leaderboard (scatter-gather query)
  • 116. Thanks! aws.amazon.com/dynamodb/resources