Data Modeling and Best Practices in DynamoDB
Peter-Mark Verwoerd
Solutions Architect
Traditional Database Architecture
Client Tier

App/Web Tier

RDBMS
One Database for All Workloads
•
•
•
•

key-value access
complex queries
transactions
analytics

Client Tier

App/Web Tier...
Cloud Data Tier Architecture
Client Tier

App/Web Tier

Data Tier
Search

Cache

Blob Store

NoSQL

RDBMS

Data Warehouse
Workload Driven Data Store Selection
rich search
key/value
simple query

hot reads
transaction
processing

logging

Data T...
AWS Services for the Data Tier
rich search
key/value
simple query

hot reads
transaction
processing

logging

Data Tier
Am...
Plan
•

Basics
– Basic Game State
– Save Games
– Social Gaming

•

Advanced
– Social Gaming
– Voting
– Global Leaderboard
Plan
•

Basics
– Basic Game State (conditional writes)
– Save Games (hash + range)
– Social Gaming (secondary indexes)

•
...
Basic Game State
conditional writes

Basics
Tic Tac Toe

Basics
Tic Tac Toe

Alice

Bob

Your App

DynamoDB

Basics
Tic Tac Toe Table

Game Table

Id

O

State

IsTie

abecd

[ Alice, Bob ]

Alice

DONE

1

fbdcc

[ Alice, Bob ]

Alice

D...
Tic Tac Toe Table

Game Table

Id

O

State

IsTie

abecd

[ Alice, Bob ]

Alice

DONE

1

fbdcc

[ Alice, Bob ]

Alice

D...
Tic Tac Toe Table

Id

State

IsTie

[ Alice, Bob ]

Alice

DONE

1

fbdcc

[ Alice, Bob ]

Alice

DONE

dbace

Basics

O
...
Tic Tac Toe Table

Id

Players

O

State

IsTie

abecd

[ Alice, Bob ]

Alice

DONE

1

fbdcc

[ Alice, Bob ]

Alice

DONE...
Tic Tac Toe Table

Primary Key

Id

O

State

IsTie

abecd

[ Alice, Bob ]

Alice

DONE

1

fbdcc

[ Alice, Bob ]

Alice

...
Tic Tac Toe Table

Id

Players

O

State

IsTie

abecd

[ Alice, Bob ]

Alice

DONE

1

fbdcc

[ Alice, Bob ]

Alice

DONE...
Tic Tac Toe Table
{
"Data" : [ [ "X", null, "O" ],
[ null, "O", null],
[ "O", null, "X" ]
]
}
Id

O

State

IsTie

abecd

...
State Transitions with Conditional Writes

Alice

Bob

DynamoDB

Basics
State Transitions with Conditional Writes

Alice

Bob

UpdateItem:
Top-Right = O
Turn = Bob

DynamoDB

Basics
State Transitions with Conditional Writes

Alice

Bob

UpdateItem:
Top-Left = X
Turn = Alice

DynamoDB

Basics
State Transitions with Conditional Writes

Alice

Bob (1)

DynamoDB

Basics

Bob (2)

Bob (3)
State Transitions with Conditional Writes

Alice

Bob (1)

DynamoDB

Basics

Bob (2)

Bob (3)
State Transitions with Conditional Writes

Alice

Bob (1)

DynamoDB

Basics

Bob (2)

Bob (3)
State Transitions with Conditional Writes

Bob (1)

Bob (2)

State : STARTED,
Turn : Bob,
Top-Right : O
DynamoDB

Basics

...
State Transitions with Conditional Writes

Bob (1)

Update:
Turn : Alice
Top-Left : X

Bob (2)

Bob (3)

Update:
Turn : Al...
State Transitions with Conditional Writes

Bob (1)

Update:
Turn : Alice
Top-Left : X

State : STARTED,
Turn : Alice,
Top-...
Conditional Writes
•
•

Basics

Apply an update only if values are as expected
Otherwise reject the write
Conditional Writes
UpdateItem Id=abecd
Game Item

{
Id : abecd,
Players : [ Alice, Bob ],
State : STARTED,
Turn : Bob,
Top...
State Transitions with Conditional Writes

Bob (1)

Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null

...
State Transitions with Conditional Writes

Bob (1)

Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null

...
State Transitions with Conditional Writes

Bob (1)

Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null

...
Save Games
hash + range

Save Games
Save Games

Save Games
Primary Key Schemas
Hash Key Schema
Primary Key

Id

Players

O

State

IsTie

abecd

[ Alice, Bob ]

Alice

DONE

1

fbdc...
Primary Key Schemas
Hash and Range Key Schema
Primary Key

Id

Turn

Players

Turn

State

abecd

0

[ Alice, Bob ]

Alice...
Primary Key Schemas

Primary Key

Id

Turn

Players

Turn

State

abecd

0

[ Alice, Bob ]

Alice

STARTED

…

abecd

1

[...
Primary Key Schemas
•

Hash-only
– Key/value lookups only

•

Hash and Range
– Given a hash key value, query for items by ...
Primary Key Schemas
Query WHERE Id=abecd, ORDER BY Turn DESC, LIMIT 2
Primary Key

Id

Turn

Players

Turn

State

abecd

...
Social Gaming
local secondary indexes

Social Gaming
Social Gaming
•
•
•
•

Host games
Invite friends to play
Find friends’ games to play
See history of games

Social Gaming
Social Gaming
Hash: UserId
Range: GameId
Attributes: OpponentId, Date, (rest of game state)
HostedGame
Table

UserId

Game...
Social Gaming
•
•
•
•

Host games
Invite friends to play
Find friends’ games to play
See history of games

Social Gaming
Social Gaming: find recent games

UserId

GameId

Date

OpponentId

…

Carol

e23f5a

2013-10-08

Charlie

…

Alice

d4e2d...
Query cost
•
•

Provisioned Throughput: Work / sec allowed on your table
Capacity Units: Amount of provisioned throughput ...
Query cost
UserId

GameId

Date

OpponentId

…

Carol

e23f5a

2013-10-08

Charlie

…

Alice

d4e2dc

2013-10-01

Bob

…

...
Query cost
UserId

GameId

Date

OpponentId

…

Carol

e23f5a

2013-10-08

Charlie

…

Alice

d4e2dc

2013-10-01

Bob

…

...
Local Secondary Indexes
•

An alternate range key on a table

HostedGame Table

LocalSecondaryIndex on Date

UserId

GameI...
Query cost on Local Secondary Indexes
UserId

…

Carol

2013-10-08

e23f5a

…

Alice

(397 older games)
2013-09-27

e9cba3...
Query cost on Local Secondary Indexes
UserId

GameId

…

Carol

2013-10-08

e23f5a

…

Alice

(397 older games)

Alice

20...
Example Local Secondary Indexes
•

Find 10 recent matches between Alice and Bob

Social Gaming
Example Local Secondary Indexes
•

Find 10 recent matches between Alice and Bob
– Hash: UserId
– Range: OpponentId + Date
...
More example Local Secondary Indexes
•

Find a host’s matches without an opponent

Social Gaming
More example Local Secondary Indexes
•

Find a host’s matches without an opponent
– Hash: UserId
– Range: UnmatchedDate

(...
Local Secondary Index Projections
•

Choose what attributes are copied into the index
– ALL, SPECIFIC, KEYS

•
•
•

Substa...
Write cost for Local Secondary Index
•

Insert new item
– 1 additional write

•

Setting index range key to / from null
– ...
Read cost for Query of non-projected attributes
•
•

Regular Query cost
+
Single-item Get cost for each evaluated item

So...
Example Local Secondary Index Projections
•

Query Alice’s 10 most recent Games

UserId

GameId

Date

OpponentId

…

Caro...
Example Local Secondary Index Projections
•

Query Alice’s 10 most recent Games
– Opponent, Winner, (UserId, GameId, Date)...
Social Gaming
transactions

Social Gaming
Social Gaming: Friends
•
•
•

Query who you are friends with
Ask to be friends with someone
Acknowledge (or decline) frien...
Social Gaming: Friends
Hash: UserId
Range: FriendId
Attributes: Status, Date, etc
Friends
Table

UserId

FriendId

Status
...
Becoming Friends: Multi-item Atomic Writes
A friend request!

UserId

Status

Bob

FRIENDS

Bob

Alice

FRIENDS

Bob

Chuc...
Becoming Friends: Multi-item Atomic Writes

UserId

Social Gaming

Status

Bob

FRIENDS

Bob
1. Update Bob/Chuck record
2....
Becoming Friends: Multi-item Atomic Writes

UpdateItem
Status=FRIENDS

Social Gaming

FriendId

Status

Bob

FRIENDS

Bob
...
Becoming Friends: Multi-item Atomic Writes

UpdateItem
Status=FRIENDS

Social Gaming

FriendId

Status

Bob

FRIENDS

Bob
...
When things go wrong

Social Gaming
Becoming Friends: When things go wrong
A friend request!

UserId

Social Gaming

Status

Bob

FRIENDS

Bob
1. Update Bob/C...
Becoming Friends: When things go wrong

UpdateItem
Status=FRIENDS

Social Gaming

FriendId

Status

Bob

FRIENDS

Bob
1. U...
Becoming Friends: When things go wrong

UpdateItem
Status=FRIENDS

Social Gaming

FriendId

Status

Bob

FRIENDS

Bob
1. U...
Multi-item transaction in DynamoDB
•
•
•

Scan for “stuck” transactions
Use the Client Transactions Library on the AWS SDK...
Replayable state machines

INCOMING

ACCEPTING

SENT

FRIENDS

Bob/
Chuck

SENDING
Chuck/
Bob

Social Gaming

FRIENDS
Client Transactions Library

Bob

Transactions
Table

Social Gaming

Transaction
Client

Transaction
Images
Table

Friends...
Client Transactions Usage
•
•
•
•
•

Low contention only
Don’t mix Tx Client writes with normal writes
No Query support
Ex...
Specialized Transactions
Transactions Table
A friend request!

Id

Status

V1

V2

Bob

1.
2.
3.
4.

FriendId

Status

V

...
Specialized Transactions
Transactions Table
Id

V1

V2

BatchGetItem

Bob

1.
2.
3.
4.

Status

FriendId

Status

V

Alice...
Specialized Transactions
Transactions Table
Id

V1

V2

Bob-Chuck

Bob: FRIENDS
Chuck: FRIENDS

2

2

PutItem,
Expect not ...
Specialized Transactions
Transactions Table
Id

V1

V2

Bob-Chuck

Bob: FRIENDS
Chuck: FRIENDS

2

2

UpdateItem,
Expect V...
Specialized Transactions
Transactions Table
Id

DeleteItem,
Expect V1=V1prev,
V2=V2prev,

Bob

1.
2.
3.
4.

Status

V1

V2...
When things go wrong

Social Gaming
Specialized Transactions
Transactions Table
Id

V1

V2

BatchGetItem

Bob

1.
2.
3.
4.

Status

FriendId

Status

V

Alice...
Specialized Transactions
Transactions Table
Id

V1

V2

Bob-Chuck

Bob: FRIENDS
Chuck: FRIENDS

2

2

PutItem,
Expect not ...
Specialized Transactions
Transactions Table
Id

V1

V2

Bob-Chuck

Bob: FRIENDS
Chuck: FRIENDS

2

2

UpdateItem,
Expect V...
Social Gaming
Specialized Transactions
Transactions Table
Id

Status

V1

V2

Bob-Chuck

Bob: FRIENDS
Chuck: FRIENDS

2

2

Scan

Sweepe...
Specialized Transactions
Transactions Table
Id

Status

V1

V2

Bob-Chuck

Bob: FRIENDS
Chuck: FRIENDS

2

2

UpdateItem,
...
Specialized Transactions
Transactions Table
Id

DeleteItem,
Expect V1=V1prev,
V2=V2prev,

Sweeper

Status

V1

V2

FriendI...
Transaction advice
•

Lock items before modifying
– Including items that don’t exist yet

•
•
•

Don’t stomp on future wri...
High-Throughput Voting
write sharding

Voting
Voting
Voter

Candidate A
Votes: 20

Candidate B
Votes: 30
Votes Table

Voting
Voting
Voter

UpdateItem
ADD 1 to “Candidate A”
(aka Atomic Increment)

Candidate A
Votes: 21

Candidate B
Votes: 30
Votes...
Scaling on DynamoDB
You

Votes Table

Voting

Need to scale
for the election
Scaling on DynamoDB
You

Provision 1200 Write Capacity Units

Votes Table

Voting
Scaling on DynamoDB
You

Provision 1200 Write Capacity Units

Partition 1

Partition 2

600 Write Capacity Units (each)
Vo...
Scaling on DynamoDB
You

Provision 1200 Write Capacity Units

Partition 1

Partition 2

(no sharing)
Votes Table

Voting
Scaling on DynamoDB
You

Provision 200,000 Write Capacity Units

Partition 1
(600 WCU)

Partition K
(600 WCU)

Partition M...
Scaling bottlenecks

Voters

Partition 1
(600 WCU)

Partition K
(600 WCU)

Partition M
(600 WCU)

Candidate B

Candidate A...
Scaling bottlenecks

Voters

Partition 1
(600 WCU)

Partition K
(600 WCU)

Partition M
(600 WCU)

Candidate B

Candidate A...
Best Practice: Uniform Workloads
“To achieve the full amount of request throughput you have
provisioned for a table, keep ...
Scaling on DynamoDB
Voter

Candidate A_1

Candidate A_4

Candidate A_7

Candidate B_5

Candidate B_1
Candidate A_5
Candida...
Scaling on DynamoDB
Voter

UpdateItem: “CandidateA_” + rand(0, 10)
ADD 1 to Votes

Candidate A_1

Candidate A_4

Candidate...
Scaling on DynamoDB

Periodic
Process

Voter

2. Store
1. Sum

Candidate A_1

Candidate A_4

Candidate A_7

Candidate B_5
...
Scaling Reads

Voters

Partition 1
(600 WCU)

Partition K
(600 WCU)

Partition M
(600 WCU)

Candidate A_Total
Votes: 2.5M
...
Scaling Reads

Voters

Partition 1
(600 WCU)

Partition K
(600 WCU)

Partition M
(600 WCU)

Candidate A_Total
Votes: 2.5M
...
Global Leaderboard
scatter-gather

Leaderboards
Game-Wide Leaderboard
•

Find the top 10 scores game-wide

HighScore

User

1000

Alice

850

Dave

580

Erin

470

Bob

3...
Game-Wide Leaderboard
•

Find the top 10 scores game-wide

HighScore
1000

Table Schemas must begin
with a Hash Key

User
...
Game-Wide Leaderboard
•

Find the top 10 scores game-wide

User
Chuck
Cannot be Queried
the way we want

HighScore
20

Ali...
Game-Wide Leaderboard
•

Use a constant Hash key?

Constant

HighScore-User

1

0001000-Alice

1

0000850-Dave

1

0000580...
Game-Wide Leaderboard
•

Use a constant Hash key?

Constant
1

0001000-Alice

1
Extremely non-uniform
workload

HighScore-...
Scatter-Gather Leading Range Key

HighScores Table
Shard

Shard

HighScore-User

2

0000600-Frank

5

0000999-Oscar

2

00...
Scatter-Gather Leading Range Key
1. Periodically Query each Shard DESC, LIMIT N

HighScores Table
Shard

Shard

HighScore-...
Scatter-Gather Leading Range Key
HighScore
1000

Alice

999

2. Keep only the top N,
Store somewhere

User

Oscar

HighSco...
Scatter-Gather Leading Range Key
User
Alice

5

Carole

HighScores Table

1

Oscar

Store the Shard id by User for high sc...
Recap
•

Basics
– Basic Game State (conditional writes)
– Save Games (hash + range)
– Social Gaming (secondary indexes)

•...
Thanks!

aws.amazon.com/dynamodb/resources
Upcoming SlideShare
Loading in...5
×

AWS Webcast - Data Modeling and Best Practices for Scaling your Application with Amazon DynamoDB

1,351

Published on

Amazon DynamoDB is a fully managed, highly scalable distributed database service. In this technical talk, we will deep dive on how to:
• Use DynamoDB to build high-scale applications like social gaming, chat, and voting.
• Model these applications using DynamoDB, including how to use building blocks such as conditional writes, consistent reads, and batch operations to build the higher-level functionality such as multi-item atomic writes and join queries.
• Incorporate best practices such as index projections, item sharding, and parallel scan for maximum scalability

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,351
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

AWS Webcast - Data Modeling and Best Practices for Scaling your Application with Amazon DynamoDB

  1. 1. Data Modeling and Best Practices in DynamoDB Peter-Mark Verwoerd Solutions Architect
  2. 2. Traditional Database Architecture Client Tier App/Web Tier RDBMS
  3. 3. One Database for All Workloads • • • • key-value access complex queries transactions analytics Client Tier App/Web Tier RDBMS
  4. 4. Cloud Data Tier Architecture Client Tier App/Web Tier Data Tier Search Cache Blob Store NoSQL RDBMS Data Warehouse
  5. 5. Workload Driven Data Store Selection rich search key/value simple query hot reads transaction processing logging Data Tier Search Cache Blob Store NoSQL RDBMS Data Warehouse analytics
  6. 6. AWS Services for the Data Tier rich search key/value simple query hot reads transaction processing logging Data Tier Amazon CloudSearch Amazon ElastiCache Amazon DynamoDB Amazon RDS Amazon S3 Amazon Redshift analytics
  7. 7. Plan • Basics – Basic Game State – Save Games – Social Gaming • Advanced – Social Gaming – Voting – Global Leaderboard
  8. 8. Plan • Basics – Basic Game State (conditional writes) – Save Games (hash + range) – Social Gaming (secondary indexes) • Advanced – Social Gaming (transactions) – Voting (write sharding) – Global Leaderboard (scatter-gather query)
  9. 9. Basic Game State conditional writes Basics
  10. 10. Tic Tac Toe Basics
  11. 11. Tic Tac Toe Alice Bob Your App DynamoDB Basics
  12. 12. Tic Tac Toe Table Game Table Id O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  13. 13. Tic Tac Toe Table Game Table Id O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  14. 14. Tic Tac Toe Table Id State IsTie [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics O abecd Item Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  15. 15. Tic Tac Toe Table Id Players O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace [ Alice, Bob ] Alice STARTED Data … Alice … … Attribute Basics Winner
  16. 16. Tic Tac Toe Table Primary Key Id O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  17. 17. Tic Tac Toe Table Id Players O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace [ Alice, Bob ] Alice STARTED Set Basics String Winner Data … Alice … … Number Binary
  18. 18. Tic Tac Toe Table { "Data" : [ [ "X", null, "O" ], [ null, "O", null], [ "O", null, "X" ] ] } Id O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace Basics Players [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  19. 19. State Transitions with Conditional Writes Alice Bob DynamoDB Basics
  20. 20. State Transitions with Conditional Writes Alice Bob UpdateItem: Top-Right = O Turn = Bob DynamoDB Basics
  21. 21. State Transitions with Conditional Writes Alice Bob UpdateItem: Top-Left = X Turn = Alice DynamoDB Basics
  22. 22. State Transitions with Conditional Writes Alice Bob (1) DynamoDB Basics Bob (2) Bob (3)
  23. 23. State Transitions with Conditional Writes Alice Bob (1) DynamoDB Basics Bob (2) Bob (3)
  24. 24. State Transitions with Conditional Writes Alice Bob (1) DynamoDB Basics Bob (2) Bob (3)
  25. 25. State Transitions with Conditional Writes Bob (1) Bob (2) State : STARTED, Turn : Bob, Top-Right : O DynamoDB Basics Bob (3)
  26. 26. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X Bob (2) Bob (3) Update: Turn : Alice Low-Right : X State : STARTED, Turn : Bob, Top-Right : O DynamoDB Basics Update: Turn : Alice Mid : X
  27. 27. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X, Mid: X, Low-Right: X Basics Bob (2) Update: Turn : Alice Mid : X Bob (3) Update: Turn : Alice Low-Right : X DynamoDB
  28. 28. Conditional Writes • • Basics Apply an update only if values are as expected Otherwise reject the write
  29. 29. Conditional Writes UpdateItem Id=abecd Game Item { Id : abecd, Players : [ Alice, Bob ], State : STARTED, Turn : Bob, Top-Right: O } Basics Updates: { Turn : Alice, Top-Left: X } Expected: { Turn : Bob, Top-Left : null, State : STARTED }
  30. 30. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Bob, Top-Right : O Basics Bob (2) DynamoDB Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Bob (3) Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null
  31. 31. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Bob, Top-Right : O Basics Bob (2) DynamoDB Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Bob (3) Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null
  32. 32. State Transitions with Conditional Writes Bob (1) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X Basics Bob (2) DynamoDB Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Bob (3) Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null
  33. 33. Save Games hash + range Save Games
  34. 34. Save Games Save Games
  35. 35. Primary Key Schemas Hash Key Schema Primary Key Id Players O State IsTie abecd [ Alice, Bob ] Alice DONE 1 fbdcc [ Alice, Bob ] Alice DONE dbace [ Alice, Bob ] Alice STARTED Winner Data … Alice … …
  36. 36. Primary Key Schemas Hash and Range Key Schema Primary Key Id Turn Players Turn State abecd 0 [ Alice, Bob ] Alice STARTED … abecd 1 [ Alice, Bob ] Bob STARTED … abecd 2 [ Alice, Bob ] Alice STARTED … abecd 3 [ Alice, Bob ] Bob STARTED … abecd 4 [ Alice, Bob ] Alice DONE dbace 0 [ Alice, Bob ] Bob STARTED dbace 1 [ Alice, Bob ] Alice STARTED Save Games IsTie Winner Alice Data … …
  37. 37. Primary Key Schemas Primary Key Id Turn Players Turn State abecd 0 [ Alice, Bob ] Alice STARTED … abecd 1 [ Alice, Bob ] Bob STARTED … abecd 2 [ Alice, Bob ] Alice STARTED … abecd 3 [ Alice, Bob ] Bob STARTED … abecd 4 [ Alice, Bob ] Alice DONE dbace 0 [ Alice, Bob ] Bob STARTED dbace 1 [ Alice, Bob ] Alice STARTED Save Games IsTie Winner Alice Data … …
  38. 38. Primary Key Schemas • Hash-only – Key/value lookups only • Hash and Range – Given a hash key value, query for items by range key – Items are sorted by range key within each hash key Save Games
  39. 39. Primary Key Schemas Query WHERE Id=abecd, ORDER BY Turn DESC, LIMIT 2 Primary Key Id Turn Players Turn State abecd 0 [ Alice, Bob ] Alice STARTED … abecd 1 [ Alice, Bob ] Bob STARTED … abecd 2 [ Alice, Bob ] Alice STARTED … abecd 3 [ Alice, Bob ] Bob STARTED … abecd 4 [ Alice, Bob ] Alice DONE dbace 0 [ Alice, Bob ] Bob STARTED dbace 1 [ Alice, Bob ] Alice STARTED Save Games IsTie Winner Alice Data … …
  40. 40. Social Gaming local secondary indexes Social Gaming
  41. 41. Social Gaming • • • • Host games Invite friends to play Find friends’ games to play See history of games Social Gaming
  42. 42. Social Gaming Hash: UserId Range: GameId Attributes: OpponentId, Date, (rest of game state) HostedGame Table UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 Social Gaming
  43. 43. Social Gaming • • • • Host games Invite friends to play Find friends’ games to play See history of games Social Gaming
  44. 44. Social Gaming: find recent games UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 Query UserId=Alice Social Gaming
  45. 45. Query cost • • Provisioned Throughput: Work / sec allowed on your table Capacity Units: Amount of provisioned throughput consumed by an operation Social Gaming
  46. 46. Query cost UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 (397 more games for Alice) Social Gaming (1 item = 600 bytes)
  47. 47. Query cost UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 (1 item = 600 bytes) (397 more games for Alice) (Items evaluated by Query) (KB per Read Capacity Unit) 400 X 600 / 1024 / 4 (bytes per item) Social Gaming (KB per byte) = 60 Read Capacity Units
  48. 48. Local Secondary Indexes • An alternate range key on a table HostedGame Table LocalSecondaryIndex on Date UserId GameId Date UserId Date GameId Carol e23f5a 2013-10-08 Carol 2013-10-08 e23f5a Alice d4e2dc 2013-10-01 Alice 2013-09-27 e9cba3 Alice e9cba3 2013-09-27 Alice 2013-10-01 d4e2dc Alice f6a3bd 2013-10-01 Alice 2013-10-01 f6a3bd Social Gaming
  49. 49. Query cost on Local Secondary Indexes UserId … Carol 2013-10-08 e23f5a … Alice (397 older games) 2013-09-27 e9cba3 … Alice 2013-10-01 d4e2dc … Alice Social Gaming GameId Alice Query for the 10 most recent games Date 2013-10-01 f6a3bd …
  50. 50. Query cost on Local Secondary Indexes UserId GameId … Carol 2013-10-08 e23f5a … Alice (397 older games) Alice 2013-09-27 e9cba3 … Alice 2013-10-01 d4e2dc … Alice Query for the 10 most recent games Date 2013-10-01 f6a3bd … (Items evaluated by Query) (KB per Read Capacity Unit) 10 X 600 / 1024 / 4 (bytes per item) Social Gaming (KB per byte) = 2 Read Capacity Units
  51. 51. Example Local Secondary Indexes • Find 10 recent matches between Alice and Bob Social Gaming
  52. 52. Example Local Secondary Indexes • Find 10 recent matches between Alice and Bob – Hash: UserId – Range: OpponentId + Date Query WHERE UserId=Alice AND OpponentAndDate STARTS_WITH “Bob-” LIMIT 10 DESC Social Gaming
  53. 53. More example Local Secondary Indexes • Find a host’s matches without an opponent Social Gaming
  54. 54. More example Local Secondary Indexes • Find a host’s matches without an opponent – Hash: UserId – Range: UnmatchedDate (sparse index) Query WHERE UserId=Alice LIMIT 10 DESC Social Gaming
  55. 55. Local Secondary Index Projections • Choose what attributes are copied into the index – ALL, SPECIFIC, KEYS • • • Substantially cheaper to Query only projection Project the attributes that your use case requires Can make writes cheaper too Social Gaming
  56. 56. Write cost for Local Secondary Index • Insert new item – 1 additional write • Setting index range key to / from null – 1 additional write • Updating a projected attribute – 1 additional write • Updating a non-projected attribute – 0 additional writes • Updating the index range key – 2 additional writes Social Gaming
  57. 57. Read cost for Query of non-projected attributes • • Regular Query cost + Single-item Get cost for each evaluated item Social Gaming
  58. 58. Example Local Secondary Index Projections • Query Alice’s 10 most recent Games UserId GameId Date OpponentId … Carol e23f5a 2013-10-08 Charlie … Alice d4e2dc 2013-10-01 Bob … Alice e9cba3 2013-09-27 Bob … Alice f6a3bd 2013-10-08 Social Gaming
  59. 59. Example Local Secondary Index Projections • Query Alice’s 10 most recent Games – Opponent, Winner, (UserId, GameId, Date) – Projected item size from 600 bytes to 40 bytes • Write cost: – 1 Write Capacity Unit for insert, opponent joining, and completion – 0 Write Capacity Units for other state transitions Social Gaming
  60. 60. Social Gaming transactions Social Gaming
  61. 61. Social Gaming: Friends • • • Query who you are friends with Ask to be friends with someone Acknowledge (or decline) friend request Social Gaming
  62. 62. Social Gaming: Friends Hash: UserId Range: FriendId Attributes: Status, Date, etc Friends Table UserId FriendId Status Date … Alice Bob FRIENDS 2013-08-20 … Bob Alice FRIENDS 2013-08-20 … Bob Chuck INCOMING 2013-10-08 … Chuck Bob SENT 2013-10-08 … Social Gaming
  63. 63. Becoming Friends: Multi-item Atomic Writes A friend request! UserId Status Bob FRIENDS Bob Alice FRIENDS Bob Chuck INCOMING Chuck Social Gaming FriendId Alice Bob Bob SENT
  64. 64. Becoming Friends: Multi-item Atomic Writes UserId Social Gaming Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record FriendId Alice Bob Alice FRIENDS Bob Chuck INCOMING Chuck Bob SENT
  65. 65. Becoming Friends: Multi-item Atomic Writes UpdateItem Status=FRIENDS Social Gaming FriendId Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record UserId Alice Bob Alice FRIENDS Bob Chuck FRIENDS Chuck Bob SENT
  66. 66. Becoming Friends: Multi-item Atomic Writes UpdateItem Status=FRIENDS Social Gaming FriendId Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record UserId Alice Bob Alice FRIENDS Bob Chuck FRIENDS Chuck Bob FRIENDS
  67. 67. When things go wrong Social Gaming
  68. 68. Becoming Friends: When things go wrong A friend request! UserId Social Gaming Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record FriendId Alice Bob Alice FRIENDS Bob Chuck INCOMING Chuck Bob SENT
  69. 69. Becoming Friends: When things go wrong UpdateItem Status=FRIENDS Social Gaming FriendId Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record UserId Alice Bob Alice FRIENDS Bob Chuck FRIENDS Chuck Bob SENT
  70. 70. Becoming Friends: When things go wrong UpdateItem Status=FRIENDS Social Gaming FriendId Status Bob FRIENDS Bob 1. Update Bob/Chuck record 2. Update Chuck/Bob record UserId Alice Bob Alice FRIENDS Bob Chuck FRIENDS Chuck Bob SENT
  71. 71. Multi-item transaction in DynamoDB • • • Scan for “stuck” transactions Use the Client Transactions Library on the AWS SDK for Java Roll your own scheme Social Gaming
  72. 72. Replayable state machines INCOMING ACCEPTING SENT FRIENDS Bob/ Chuck SENDING Chuck/ Bob Social Gaming FRIENDS
  73. 73. Client Transactions Library Bob Transactions Table Social Gaming Transaction Client Transaction Images Table Friends Table
  74. 74. Client Transactions Usage • • • • • Low contention only Don’t mix Tx Client writes with normal writes No Query support Expensive, slower But, easy to use Social Gaming
  75. 75. Specialized Transactions Transactions Table A friend request! Id Status V1 V2 Bob 1. 2. 3. 4. FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  76. 76. Specialized Transactions Transactions Table Id V1 V2 BatchGetItem Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  77. 77. Specialized Transactions Transactions Table Id V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 PutItem, Expect not exists Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  78. 78. Specialized Transactions Transactions Table Id V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 UpdateItem, Expect V=Vprev Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob FRIENDS 3
  79. 79. Specialized Transactions Transactions Table Id DeleteItem, Expect V1=V1prev, V2=V2prev, Bob 1. 2. 3. 4. Status V1 V2 FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob FRIENDS 3
  80. 80. When things go wrong Social Gaming
  81. 81. Specialized Transactions Transactions Table Id V1 V2 BatchGetItem Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  82. 82. Specialized Transactions Transactions Table Id V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 PutItem, Expect not exists Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck INCOMING 2 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  83. 83. Specialized Transactions Transactions Table Id V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 UpdateItem, Expect V=Vprev Bob 1. 2. 3. 4. Status FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob Read items Write to Tx table Apply writes Delete from Tx table Bob SENT 2
  84. 84. Social Gaming
  85. 85. Specialized Transactions Transactions Table Id Status V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 Scan Sweeper FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob 1. Scan for stuck Tx 2. Apply writes 3. Delete from Tx table Bob SENT 2
  86. 86. Specialized Transactions Transactions Table Id Status V1 V2 Bob-Chuck Bob: FRIENDS Chuck: FRIENDS 2 2 UpdateItem, Expect V=Vprev Sweeper FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob 1. Scan for stuck Tx 2. Apply writes 3. Delete from Tx table Bob FRIENDS 3
  87. 87. Specialized Transactions Transactions Table Id DeleteItem, Expect V1=V1prev, V2=V2prev, Sweeper Status V1 V2 FriendId Status V Alice Bob FRIENDS 3 Bob Alice FRIENDS 3 Chuck FRIENDS 3 Chuck Social Gaming UserId Bob 1. Scan for stuck Tx 2. Apply writes 3. Delete from Tx table Bob FRIENDS 3
  88. 88. Transaction advice • Lock items before modifying – Including items that don’t exist yet • • • Don’t stomp on future writes (use versions) Sweep for stuck transactions Avoid deadlock Social Gaming
  89. 89. High-Throughput Voting write sharding Voting
  90. 90. Voting Voter Candidate A Votes: 20 Candidate B Votes: 30 Votes Table Voting
  91. 91. Voting Voter UpdateItem ADD 1 to “Candidate A” (aka Atomic Increment) Candidate A Votes: 21 Candidate B Votes: 30 Votes Table Voting
  92. 92. Scaling on DynamoDB You Votes Table Voting Need to scale for the election
  93. 93. Scaling on DynamoDB You Provision 1200 Write Capacity Units Votes Table Voting
  94. 94. Scaling on DynamoDB You Provision 1200 Write Capacity Units Partition 1 Partition 2 600 Write Capacity Units (each) Votes Table Voting
  95. 95. Scaling on DynamoDB You Provision 1200 Write Capacity Units Partition 1 Partition 2 (no sharing) Votes Table Voting
  96. 96. Scaling on DynamoDB You Provision 200,000 Write Capacity Units Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Votes Table Voting Partition N (600 WCU)
  97. 97. Scaling bottlenecks Voters Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Candidate B Candidate A Votes Table Voting Partition N (600 WCU)
  98. 98. Scaling bottlenecks Voters Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Candidate B Candidate A Votes Table Voting Partition N (600 WCU)
  99. 99. Best Practice: Uniform Workloads “To achieve the full amount of request throughput you have provisioned for a table, keep your workload spread evenly across the hash key values.” – DynamoDB Developer Guide Voting
  100. 100. Scaling on DynamoDB Voter Candidate A_1 Candidate A_4 Candidate A_7 Candidate B_5 Candidate B_1 Candidate A_5 Candidate A_2 Candidate B_8 Candidate B_4 Candidate B_3 Candidate B_7 Candidate A_3 Candidate A_6 Candidate A_8 Voting Votes Table Candidate B_2 Candidate B_6
  101. 101. Scaling on DynamoDB Voter UpdateItem: “CandidateA_” + rand(0, 10) ADD 1 to Votes Candidate A_1 Candidate A_4 Candidate A_7 Candidate B_5 Candidate B_1 Candidate A_5 Candidate A_2 Candidate B_8 Candidate B_4 Candidate B_3 Candidate B_7 Candidate A_3 Candidate A_6 Candidate A_8 Voting Votes Table Candidate B_2 Candidate B_6
  102. 102. Scaling on DynamoDB Periodic Process Voter 2. Store 1. Sum Candidate A_1 Candidate A_4 Candidate A_7 Candidate B_5 Candidate B_1 Candidate A_5 Candidate A_2 Candidate B_8 Candidate B_4 Candidate A Total: 2.5M Candidate B_3 Candidate B_7 Candidate A_3 Candidate A_6 Candidate A_8 Voting Votes Table Candidate B_2 Candidate B_6
  103. 103. Scaling Reads Voters Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Candidate A_Total Votes: 2.5M Partition N (600 WCU) Candidate B_Total Votes: 2.1M Votes Table Voting
  104. 104. Scaling Reads Voters Partition 1 (600 WCU) Partition K (600 WCU) Partition M (600 WCU) Candidate A_Total Votes: 2.5M Partition N (600 WCU) Candidate B_Total Votes: 2.1M Votes Table Voting
  105. 105. Global Leaderboard scatter-gather Leaderboards
  106. 106. Game-Wide Leaderboard • Find the top 10 scores game-wide HighScore User 1000 Alice 850 Dave 580 Erin 470 Bob 30 Chuck Leaderboards
  107. 107. Game-Wide Leaderboard • Find the top 10 scores game-wide HighScore 1000 Table Schemas must begin with a Hash Key User Alice 850 Dave 580 Erin 470 Bob 30 Chuck Leaderboards
  108. 108. Game-Wide Leaderboard • Find the top 10 scores game-wide User Chuck Cannot be Queried the way we want HighScore 20 Alice 1000 Bob 470 Dave 850 Erin 580 Leaderboards
  109. 109. Game-Wide Leaderboard • Use a constant Hash key? Constant HighScore-User 1 0001000-Alice 1 0000850-Dave 1 0000580-Erin 1 0000470-Bob 1 0000030-Chuck Leaderboards Zero-pad strings for sort stability
  110. 110. Game-Wide Leaderboard • Use a constant Hash key? Constant 1 0001000-Alice 1 Extremely non-uniform workload HighScore-User 0000850-Dave 1 0000580-Erin 1 0000470-Bob 1 0000030-Chuck Leaderboards
  111. 111. Scatter-Gather Leading Range Key HighScores Table Shard Shard HighScore-User 2 0000600-Frank 5 0000999-Oscar 2 0000581-Trent 5 0000700-Craig 5 0000030-Chuck 0001000-Alice 1 0000980-Eve HighScore-User 1 HighScore-User 2 Shard 0000850-Dave 1 0000580-Erin Shard HighScore-User 3 0000900-Dan 3 0000850-Wendy 3 0000080-Trent Shard HighScore-User 4 0000500-Merlin 4 0000350-Carole 4 0000280-Paul Leaderboards
  112. 112. Scatter-Gather Leading Range Key 1. Periodically Query each Shard DESC, LIMIT N HighScores Table Shard Shard HighScore-User 2 0000600-Frank 5 0000999-Oscar 2 0000581-Trent 5 0000700-Craig 5 0000030-Chuck 0001000-Alice 1 0000980-Eve HighScore-User 1 HighScore-User 2 Shard 0000850-Dave 1 0000580-Erin Shard HighScore-User 3 0000900-Dan 3 0000850-Wendy 3 0000080-Trent Shard HighScore-User 4 0000500-Merlin 4 0000350-Carole 4 0000280-Paul Leaderboards
  113. 113. Scatter-Gather Leading Range Key HighScore 1000 Alice 999 2. Keep only the top N, Store somewhere User Oscar HighScores Table Shard Shard HighScore-User 2 0000600-Frank 5 0000999-Oscar 2 0000581-Trent 5 0000700-Craig 5 0000030-Chuck 0001000-Alice 1 0000980-Eve HighScore-User 1 HighScore-User 2 Shard 0000850-Dave 1 0000580-Erin Shard HighScore-User 3 0000900-Dan 3 0000850-Wendy 3 0000080-Trent Shard HighScore-User 4 0000500-Merlin 4 0000350-Carole 4 0000280-Paul Leaderboards
  114. 114. Scatter-Gather Leading Range Key User Alice 5 Carole HighScores Table 1 Oscar Store the Shard id by User for high score updates Shard 4 Shard Shard HighScore-User 2 0000600-Frank 5 0000999-Oscar 2 0000581-Trent 5 0000700-Craig 5 0000030-Chuck 0001000-Alice 1 0000980-Eve HighScore-User 1 HighScore-User 2 Shard 0000850-Dave 1 0000580-Erin Shard HighScore-User 3 0000900-Dan 3 0000850-Wendy 3 0000080-Trent Shard HighScore-User 4 0000500-Merlin 4 0000350-Carole 4 0000280-Paul Leaderboards
  115. 115. Recap • Basics – Basic Game State (conditional writes) – Save Games (hash + range) – Social Gaming (secondary indexes) • Advanced – Social Gaming (transactions) – Replication (cross-region, cross-data-store) – Global Leaderboard (scatter-gather query)
  116. 116. Thanks! aws.amazon.com/dynamodb/resources
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×