SlideShare a Scribd company logo
1 of 66
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Advanced Design Patterns for
Amazon DynamoDB
R i c k H o u l i h a n – S e n i o r P r a c t i c e M a n a g e r , N o S Q L
A W S P r o f e s s i o n a l S e r v i c e s
D A T 4 0 3
N o v e m b e r 2 9 , 2 0 1 7
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to Expect from the Session
• When to use NoSQL and why
• Brief overview of Amazon DynamoDB
• Hot keys, design considerations
• NoSQL data modeling
• Normalized versus unstructured schema
• Common NoSQL design patterns
• Time series, write-sharding, MVCC, etc.
• Amazon DynamoDB in the serverless ecosystem
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is a database?
“A place to put stuff my app needs.” – Average Developer
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Technology adoption and the hype curve
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why NoSQL?
Optimized for storage Optimized for compute
Normalized/relational Denormalized/hierarchical
Ad hoc queries Instantiated views
Scale vertically Scale horizontally
Good for OLAP Built for OLTP at scale
SQL NoSQL
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon DynamoDB
Document or key-value Scales to any workloadFully managed NoSQL
Access control Event-driven programmingFast and consistent
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Table
Table
Items
Attributes
Partition
key
Sort
key
Mandatory
Key-value access pattern
Determines data distribution
Optional
Model 1:N relationships
Enables rich query capabilities
All items for key
==, <, >, >=, <=
“begins with”
“between”
“contains”
“in”
sorted results
counts
top/bottom N values
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NoSQL data modeling
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It’s all about aggregations…
Document management Process controlSocial network
Data treesIT monitoring
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SQL vs. NoSQL design pattern
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Design patterns and best practices
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB Streams and AWS Lambda
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Triggers
Lambda function
Notify change
Item/table level metrics
Amazon CloudSearch
Kinesis Firehose
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Write sharding
Handling high velocity writes
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Partition 1
1000 WCUs
Partition K
1000 WCUs
Partition M
1000 WCUs
Partition N
1000 WCUs
Votes Table
Candidate A Candidate B
Scaling bottlenecks
Voters
Provision 200,000 WCUs
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Queue-based load leveling
Workers
Dashboard
SQS
• Write incoming votes to SQS
• Scale workers as needed to
process queue
• Avoid key pressure
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Write sharding
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4
Candidate A_7 Candidate B_8
Candidate A_6 Candidate A_8
Candidate A_5
Voter
Votes Table
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Write sharding
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4
Candidate A_7 Candidate B_8
Insert: “CandidateA_” + rand(0, 10)
Candidate A_6 Candidate A_8
Candidate A_5
Voter
Votes Table
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Votes Table
Shard aggregation
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4
Candidate A_5
Candidate A_6 Candidate A_8
Candidate A_7 Candidate B_8
Periodic
process
Candidate A
Total: 2.5M
1. Sum
2. Store Voter
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Query a single sharded item efficiently
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4
Candidate A_7 Candidate B_8
Insert: “CandidateA_” + hash(SSN)
Candidate A_6 Candidate A_8
Candidate A_5
Voter
Votes Table
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Calculating partition counts (reads)
Items per partition
Average item size
RCU Size
Requests per second
100 K * 0.2 KB / 4 KB * 10 / 3000 =
Partition max RCU
~ 17
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Calculating partition counts (writes)
Items per second
Average item size
100 K * (itemSize < 1KB ? 1KB : itemSize) / 1000 =
Partition max WCU
100
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Increase throughput with concurrency
Consider RCU/WCU per key, item size and request rate
Shard write-heavy partition keys
your write workload is not horizontally
scalable
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Time-based workflows
Processing the entire table
efficiently
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Finding expired items
Active Tickets_Table
Event_id
(Partition)
Timestamp GSIKey
Rand(0-N)
… Attribute N
Expired Tickets GSI
GSIKey
(Partition)
Timestamp
(Sort)
Archive Table
Event_id
(Partition)
Timestamp
(Sort)
Attribute1 …. Attribute N
RCUs = 10000
WCUs = 10000
RCUs = 100
WCUs = 1
Current table
HotdataColddata
Scatter query GSI for expired tickets and use TTL to archive
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Find items efficiently with AWS Lambda
Use a write-sharded GSI to selectively query the entire table
Create a Lambda “stored procedure” to process items
Migrate data between tables with TTL/Streams/Lambda
there is a need to query all items on the
table selectively
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Product catalog
Popular items (read)
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Partition 1
2000 RCUs
Partition K
2000 RCUs
Partition M
2000 RCUs
Partition 50
2000 RCU
Scaling bottlenecks
Product A Product B
Shoppers
Product Catalog Table
SELECT Id, Description, ...
FROM ProductCatalog
WHERE Id="POPULAR_PRODUCT"
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Fully managed, highly available: handles all software management,
fault tolerant, replication across multi-AZs within a region
• DynamoDB API compatible: seamlessly caches DynamoDB API
calls, no application re-writes required
• Write-through: DAX handles caching for writes
• Flexible: configure DAX for one table or many
• Scalable: scales-out to any workload with up to 10 read replicas
• Manageability: fully integrated AWS service: Amazon CloudWatch,
tagging for DynamoDB, AWS Console
• Security: Amazon VPC, AWS IAM, AWS CloudTrail, AWS
Organizations
Features
DynamoDB Accelerator (DAX)
DynamoDB
Your Applications
DynamoDB Accelerator
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Targeting queries
Query filters, composite keys, and
sparse indexes
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Secondary index
Opponent Date GameId Status Host
Alice 2014-10-02 d9bl3 DONE David
Carol 2014-10-08 o2pnb IN_PROGRESS Bob
Bob 2014-09-30 72f49 PENDING Alice
Bob 2014-10-03 b932s PENDING Carol
Bob 2014-10-03 ef9ca IN_PROGRESS David
BobPartition key Sort key
Multi-value sorts and filters
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Secondary Index
Approach 1: Query filter
Bob
Opponent Date GameId Status Host
Alice 2014-10-02 d9bl3 DONE David
Carol 2014-10-08 o2pnb IN_PROGRESS Bob
Bob 2014-09-30 72f49 PENDING Alice
Bob 2014-10-03 b932s PENDING Carol
Bob 2014-10-03 ef9ca IN_PROGRESS David
SELECT * FROM Game
WHERE Opponent='Bob'
ORDER BY Date DESC
FILTER ON Status='PENDING'
(filtered out)
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Approach 2: Composite key
StatusDate
DONE_2014-10-02
IN_PROGRESS_2014-10-08
IN_PROGRESS_2014-10-03
PENDING_2014-09-30
PENDING_2014-10-03
Status
DONE
IN_PROGRESS
IN_PROGRESS
PENDING
PENDING
Date
2014-10-02
2014-10-08
2014-10-03
2014-10-03
2014-09-30
+ =
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Secondary Index
Approach 2: Composite key
Opponent StatusDate GameId Host
Alice DONE_2014-10-02 d9bl3 David
Carol IN_PROGRESS_2014-10-08 o2pnb Bob
Bob IN_PROGRESS_2014-10-03 ef9ca David
Bob PENDING_2014-09-30 72f49 Alice
Bob PENDING_2014-10-03 b932s Carol
Partition key Sort key
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Opponent StatusDate GameId Host
Alice DONE_2014-10-02 d9bl3 David
Carol IN_PROGRESS_2014-10-08 o2pnb Bob
Bob IN_PROGRESS_2014-10-03 ef9ca David
Bob PENDING_2014-09-30 72f49 Alice
Bob PENDING_2014-10-03 b932s Carol
Secondary index
Approach 2: Composite key
Bob
SELECT * FROM Game
WHERE Opponent='Bob'
AND StatusDate BEGINS_WITH 'PENDING'
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sparse indexes
Id
(Partition)
User Game Score Date Award
1 Bob G1 1300 2012-12-23
2 Bob G1 1450 2012-12-23
3 Jay G1 1600 2012-12-24
4 Mary G1 2000 2012-10-24 Champ
5 Ryan G2 123 2012-03-10
6 Jones G2 345 2012-03-20
Game-scores-table
Award
(Partition)
Id User Score
Champ 4 Mary 2000
Award-GSI
Scan sparse GSIs
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Concatenate attributes to form useful
secondary index keys
Take advantage of sparse indexes
Replace filter with indexes
you want to optimize a query as much
as possible
Status + Date
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Vertical Partitioning
Large items
Filters vs. indexes
M:N modeling—inbox and outbox
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reports
Table
Workflow Management App
David
SELECT *
FROM Reports
WHERE Owner = ‘David’
AND State = ‘Pending’
LIMIT 50
ORDER BY Date DESC
Pending
SELECT *
FROM Reports
WHERE Owner = ‘David’
AND State = ‘Processed’
LIMIT 50
ORDER BY Date DESC
Processed
Owner StateDate Document
David Pending#2014-10-02 …
… many more Reports for David …
David Processed#2014-10-03 …
Alice Pending#2014-09-28 …
Alice Pending#2014-10-01 …
Large and small attributes mixed
(Many more report items)
David
Reports table
50 items × 256 KB each
Partition key Sort key
Large attachments
SELECT *
FROM Reports
WHERE Recipient=‘David’
AND Status=‘Pending’
LIMIT 50
ORDER BY Date DESC
Inbox
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Computing read query cost
Items evaluated by query
Average item size
Conversion ratio
Eventually consistent reads
50 * 256KB * (1 RCU / 4 KB) * (1 / 2) = 1600 RCU
Owner StateDate Summary ReportID
David Pending#2014-10-02 … afed
David Processed#2014-10-03 … 3kf8
Alice Processed#2014-09-28 … 9d2b
Alice Processed#2014-10-01 … ct7r
Separate the bulk data
Pending-GSI Reports table
ReportID Body
9d2b …
3kf8 …
ct7r …
afed …
David
1. Query Pending-GSI: 1 RCU
2. BatchGetItem messages: 1600 RCU
(50 separate items at 256 KB)
(50 sequential items at 128 bytes)
Uniformly distributes large item reads
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Messaging app
Reports
table
David
Pending GSI
Pending
Processed GSI
Processed
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reduce one-to-many item sizes
Configure secondary index projections
Use GSIs to model M:N relationship
Distribute large items
querying many large items at once
ProcessedReportsPending
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Advanced data modeling
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multi-version concurrency
Transactions in NoSQL
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How OLTP apps use data
 Mostly hierarchical
structures
 Entity driven workflows
 Data spread across tables
 Requires complex queries
 Primary driver for ACID
ItemID
(PK)
Version
(SK)
CurVer Attrs
1
v0 2 …
v1 …
v2 …
v3 …
Emulating ACID transactions
(Many more item partitions)
Item versions
Overwrite v0 Item to
Commit changes
COPY Item.v0 -> Item1.v3 IF Item.v3 == NULL
UPDATE Item1.v3 SET Attr1 += 1
UPDATE Item1.v3 SET Attr2 = …
UPDATE Item1.v3 SET Attr3 = …
COPY Item1.v3 -> Item1.v0 SET CurVer = 3
Transaction
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Versioning item sets
Get all items for a product
assembly
SELECT *
FROM Assemblies
WHERE ID=1
Order
Item
Picker
ID PartID Vendor Bin
1
1 ACME 27Z19
2 ACME 39M97
3 ACME 75B25
(Many more assemblies)
Assemblies table
Partition key Sort key
ID PartID Vendor Bin
1
1 ACME 27Z19
2 ACME 39M97
3 ACME 75B25
4 UAC 53G56
5 UAC 64B17
6 UAC 48J19
Updates are problematic
(Many more assemblies)
Assemblies table
Old items
SELECT *
FROM Assemblies
WHERE ID=1
Order Item Picker
New items
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ID PartID Vendor Bin
1
1 ACME 27Z19
2 ACME 39M97
3 ACME 75B25
4 UAC 53G56
5 UAC 64B17
6 UAC 48J19
Creating partition “locks”
(Many more assemblies)
Assemblies table
• Use metadata item for versioning
SET #ItemList.i = list append(:newList, #ItemList.i ), #Lock =
1
• Obtain locks with conditional writes
--condition-expression “Lock = 0”
• Remove old items or add version info to Sort Key
Query composite keys for specific versions
• Readers can determine how to handle “fat” reads
• Add additional metadata to manage transactional
workflow as needed
Current state, breadcrumbs, timeout, etc.
ID PartID ItemList Lock
1 0 {i:[[1,2,3],[4,5,6]]} 0
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use item partitions to manage transactional
workflows
Manage versioning across items with metadata
Tag attributes to maintain multiple versions
Code your app to recognize when updates are in progress
Implement app layer error handling and recovery logic
transactional writes across items is
required
Geo-hashing
FleetID
(PK)
Location
(SK)
GSIKey
Rand(0-N)
GUID
0201200 …
0213211 …
0233321 …
0320011 …
• Break down map area into a grid
• Continue to subdivide grid sections to
narrow down search areas
• Use range queries to find nearby items
• Configure write-sharded GSI to query
across partitions
GSIKey
(PK)
Location
(SK)
FleetID
0-N
… GUID
… GUID
… GUID
… GUID
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hierarchical data
Composite key modeling
Hierarchical data structures as items
• Use composite sort key to define a hierarchy
• Highly selective queries with sort conditions
• Reduce query complexity
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
… or as documents (JSON)
JSON data types (M, L, BOOL, NULL)
Document SDKs available
Indexing only by using DynamoDB Streams or AWS Lambda
400 KB maximum item size (limits hierarchical data structure)
Primary Key
Attributes
PK
Items
BookID
type title author genre publisher datePublished ISBN
Book Ringworld Larry Niven Science Fiction Ballantine Oct-70 0-345-02046-4
AlbumID
type title artist genre Attributes
Album
Dark Side of the
Moon
Pink Floyd Progressive Rock
{ label:"Harvest", studio: "Abbey Road", published: "3/1/73", producer: "Pink
Floyd", tracks: [{title: "Speak to Me", length: "1:30", music: "Mason", vocals:
"Instrumental"},{title: ”Breathe", length: ”2:43", music: ”Waters, Gilmour,
Wright", vocals: ”Gilmour"},{title: ”On the Run", length: “3:30", music: ”Gilmour,
Waters", vocals: "Instrumental"}]}
MovieID
type title genre writer Attributes
Movie Idiocracy Scifi Comedy Mike Judge
{ producer: "20th Century Fox", actors: [{ name: "Luke Wilson", dob: "9/21/71",
character: "Joe Bowers", image: "img2.jpg"},{ name: "Maya Rudolph", dob:
"7/27/72", character: "Rita", image: "img1.jpg"},{ name: "Dax Shepard", dob:
"1/2/75", character: "Frito Pendejo", image: "img3.jpg"}]
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
… or as nested sets
• Store data trees on a table as
array of leaf items
• Non-leaf items define start/end
index values
• Query for leaf items by
category using index ranges
• Partition table on node ID, add
edges to define adjacency list
• Define a default edge for every
node type to describe the node
itself
• Use partitioned GSIs to query large
nodes (dates, places, etc.)
• Use Dynamo DB
Streams/Lambda/EMR for graph
query projections
• Neighbor entity state
• Subtree aggregations
• Breadth first search
• Node ranking
Adjacency lists and materialized graphs
GSI Primary Key Attributes
GSIkey Data Target Type Node Projection
0-N
Jason Bourne 1
Person
1 …
James John Doe 4 4 …
20170418 2
Birthdate
1 …
4 …
Date 2 …
Finland 3
Birthplace
1 …
4 …
Place 3 …
GSI Primary Key Attributes
GSIkey Type Target Data Node Projection
0-N
Person
1 Jason Bourne 1 …
4 John Doe 4 …
Birthdate 2 20170418
1 …
4 …
Birthplace 3 Finland
1 …
4 …
Date 2 20170418 2 …
Place 3 Finland 3 …
Table Primary Key Attributes
Node Type Target Data GSIkey Projection
1
Person 1 Jason Bourne
HASH(Person.Data)
Edge/spanning
tree rollups
Birthdate 2 20170418
Birthplace 3 Finland
2 Date 2 20170418
HASH(Data) (Summary Stats)
3 Place 3 Finland
4
Person 4 John Doe
HASH(Person.Data
Edge/spanning
tree rollups
Birthdate 2 20170418
Birthplace 3 Finland
Audible eBook sync service
• Allows users to save session
state for Audible eBooks
• Maintains mappings per user
for eBooks and audio products
• Spikey load patterns require
significant overprovisioning
• Large number of access
patterns
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Access patterns
#
ACCESS PATTERNS: https://w.amazon.com/index.php/WfV/Services/DBMigration
USE CASE
1 CompanionMapping getCompanionMappingsByAsin
2 CompanionMapping getCompanionMappingsByEbookAndAudiobookContentId
3 CompanionMapping getCompanionMappingsFromCache
4 CompanionMapping getCompanionMappings
5 CompanionMapping getCompanionMappingsAvailable
6 AcrInfo getACRInfo
7 AcrInfo getACRs
8 AcrInfo getACRInfos
9 AcrInfo getACRInfosbySKU
10 AudioProduct getAudioProductsForACRs
11 AudioProduct getAudioProducts
12 AudioProduct deleteAudioProductsMatchingSkuVersions
13 AudioProduct getChildAudioProductsForSKU
14 Product getProductInfoByAsins
15 Product getParentChildDataByParentAsins
16 AudioFile getAudioFilesForACR
17 AudioFile getAudioFilesForChildACR
18 AudioFile getAudioFilesByParentAsinVersionFormat
19 AudioFile getAudioFiles
20 AudioFile getAudioFilesForChildAsin
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Primary table
T
A
B
L
E
Primary Key
Attributes
PK SK (GSI 3)
ABOOKACR1
v0#ABOOKACR1
GSI-1 GSI-2
ABOOK-ASIN1 ABOOK-SKU1
EBOOKACR1
GSI-1 GSI-2
SyncFileAcr ABOOK-ASIN1
ABOOKACR1#TRACK#1
GSI-1 GSI-2
ABOOK-ASIN1 ABOOK-SKU1
ABOOKACR1#TRACK#2
GSI-1 GSI-2
ABOOK-ASIN1 ABOOK-SKU1
EBOOKACR1 EBOOKACR1
GSI-1 EBookAsin
EBOOK-SKU1 ASIN
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Indexes
G
S
I
1
Partition Key Projected Attributes
ABOOK-ASIN1 ABOOKACR1
ABOOKACR1-v1
ABOOKACR1#TRACK#1
ABOOKACR1#TRACK#2
SyncFileAcr ABOOKACR1 MAP-EBOOKACR1
EBOOK-SKU1 ABOOKACR1 EBOOKACR1
G
S
I
2
Partition Key Projected Attributes
ABOOK-ASIN1 ABOOKACR1 MAP-EBOOKACR1
ABOOK-SKU1 ABOOKACR1
ABOOKACR1-v1
ABOOKACR1#TRACK#1
ABOOKACR1#TRACK#2
G
S
I
3
GSI Partition Key Projected Attributes
V0#ABOOKACR1 ABOOKACR1 ABOOKACR1-v1
EBOOKACR1 ABOOKACR1 MAP-EBOOKACR1
ABOOKACR1#TRACK#1 ABOOKACR1 ABOOKACR1#TRACK#1
ABOOKACR1#TRACK#2 ABOOKACR1 ABOOKACR1#TRACK#2
EBOOKACR1 ABOOKACR1 EBOOKACR1
Query conditions
#
ACCESS PATTERNS: https://w.amazon.com/index.php/WfV/Services/DBMigration
USE CASE Lookup parameters INDEX Key Conditions Filter Conditions
1 CompanionMapping getCompanionMappingsByAsin audiobookAsin/ebookSku GSI2 GSI-2=ABOOK-ASIN1 None
2 CompanionMapping
getCompanionMappingsByEbookAndAudi
obookContentId
ebookAcr/sku,version,format or
audiobookAcr/asin,version,format
GSI-3 on TargetACR attribute OR
PrimaryKey on Table
GSI-3=MAP-EBOOKACR1 version=v and format=f
3 CompanionMapping getCompanionMappingsFromCache
ebookAcr/sku,version,format or
audiobookAcr/asin,version,format
GSI-3 on TargetACR attribute OR
PrimaryKey on Table
GSI-3=MAP-EBOOKACR1 version=v and format=f
4 CompanionMapping getCompanionMappings
syncfileAcr, ebookAcr?,
audiobookAcr?
GSI1 GSI-1=SyncFileAcr None
5 CompanionMapping getCompanionMappingsAvailable ebookAcr, audiobookAcr Primary Key on Table
Acr=ABOOKACR1 and
TargetACR beginswith "MAP-"
6 AcrInfo getACRInfo acr Primary Key on Table
Acr=ABOOKACR1 and
TargetACR beginswith "ABOOKACR1-v"
7 AcrInfo getACRs acr / asin,version,format Primary Key on Table Acr=ABOOKACR1 version=v and format=f
8 AcrInfo getACRInfos acr Primary Key on table
Acr=ABOOKACR1 and
TargetACR beginswith "ABOOKACR1"
9 AcrInfo getACRInfosbySKU sku GSI2 GSI-2=ABOOK-SKU1
10 AudioProduct getAudioProductsForACRs acr Primary Key on table
Acr=ABOOKACR1 and TargetACR
beginswith "ABOOKACR1"
11 AudioProduct getAudioProducts sku, version, format GSI2 GSI-2=ABOOK-SKU1 version=v and format=f
12 AudioProduct deleteAudioProductsMatchingSkuVersions sku, version GSI2 GSI-2=ABOOK-SKU1 version=v
13 AudioProduct getChildAudioProductsForSKU sku GSI2 GSI-2=ABOOK-SKU1
14 Product getProductInfoByAsins asin GSI1 GSI-1=ABOOK-ASIN1
15 Product getParentChildDataByParentAsins asin GSI1 GSI-1=ABOOK-ASIN1
16 AudioFile getAudioFilesForACR acr Primary Key on table
Acr=ABOOKACR1 and
TargetACR beginswith "ABOOKACR1#"
17 AudioFile getAudioFilesForChildACR acr, parent_asin Primary Key on table Acr=ABOOKACR1 version=v and format=f
18 AudioFile getAudioFilesByParentAsinVersionFormat parent_asin, version, format GSI1 GSI-1=ABOOK-ASIN1 version=v and format=f
19 AudioFile getAudioFiles sku, version, format GSI2 GSI-2=ABOOK-SKU1 version=v and format=f
20 AudioFile getAudioFilesForChildAsin asin, parent_asin, version, format GSI1 GSI-1=ABOOK-ASIN1 version=v and format=f
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!

More Related Content

What's hot

Amazon DynamoDB Deep Dive Advanced Design Patterns for DynamoDB (DAT401) - AW...
Amazon DynamoDB Deep Dive Advanced Design Patterns for DynamoDB (DAT401) - AW...Amazon DynamoDB Deep Dive Advanced Design Patterns for DynamoDB (DAT401) - AW...
Amazon DynamoDB Deep Dive Advanced Design Patterns for DynamoDB (DAT401) - AW...Amazon Web Services
 
Aurora는 어떻게 다른가 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 Gaming
Aurora는 어떻게 다른가 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 GamingAurora는 어떻게 다른가 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 Gaming
Aurora는 어떻게 다른가 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 GamingAmazon Web Services Korea
 
Diving into Delta Lake: Unpacking the Transaction Log
Diving into Delta Lake: Unpacking the Transaction LogDiving into Delta Lake: Unpacking the Transaction Log
Diving into Delta Lake: Unpacking the Transaction LogDatabricks
 
AWS로 사용자 천만 명 서비스 만들기 (윤석찬)- 클라우드 태권 2015
AWS로 사용자 천만 명 서비스 만들기 (윤석찬)- 클라우드 태권 2015 AWS로 사용자 천만 명 서비스 만들기 (윤석찬)- 클라우드 태권 2015
AWS로 사용자 천만 명 서비스 만들기 (윤석찬)- 클라우드 태권 2015 Amazon Web Services Korea
 
Percona xtrabackup - MySQL Meetup @ Mumbai
Percona xtrabackup - MySQL Meetup @ MumbaiPercona xtrabackup - MySQL Meetup @ Mumbai
Percona xtrabackup - MySQL Meetup @ MumbaiNilnandan Joshi
 
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정PgDay.Seoul
 
AWS CloudFormation Best Practices
AWS CloudFormation Best PracticesAWS CloudFormation Best Practices
AWS CloudFormation Best PracticesAmazon Web Services
 
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...confluent
 
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
 
Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...
Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...
Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...Amazon Web Services
 
Deep Dive on MySQL Databases on Amazon RDS (DAT322) - AWS re:Invent 2018
Deep Dive on MySQL Databases on Amazon RDS (DAT322) - AWS re:Invent 2018Deep Dive on MySQL Databases on Amazon RDS (DAT322) - AWS re:Invent 2018
Deep Dive on MySQL Databases on Amazon RDS (DAT322) - AWS re:Invent 2018Amazon Web Services
 
Serverless with IAC - terraform과 cloudformation 비교
Serverless with IAC - terraform과 cloudformation 비교Serverless with IAC - terraform과 cloudformation 비교
Serverless with IAC - terraform과 cloudformation 비교재현 신
 
[Pgday.Seoul 2017] 2. PostgreSQL을 위한 리눅스 커널 최적화 - 김상욱
[Pgday.Seoul 2017] 2. PostgreSQL을 위한 리눅스 커널 최적화 - 김상욱[Pgday.Seoul 2017] 2. PostgreSQL을 위한 리눅스 커널 최적화 - 김상욱
[Pgday.Seoul 2017] 2. PostgreSQL을 위한 리눅스 커널 최적화 - 김상욱PgDay.Seoul
 
How to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScaleHow to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScaleMariaDB plc
 
Serverless Streaming Data Processing using Amazon Kinesis Analytics
Serverless Streaming Data Processing using Amazon Kinesis AnalyticsServerless Streaming Data Processing using Amazon Kinesis Analytics
Serverless Streaming Data Processing using Amazon Kinesis AnalyticsAmazon Web Services
 
Deep Dive on PostgreSQL Databases on Amazon RDS (DAT324) - AWS re:Invent 2018
Deep Dive on PostgreSQL Databases on Amazon RDS (DAT324) - AWS re:Invent 2018Deep Dive on PostgreSQL Databases on Amazon RDS (DAT324) - AWS re:Invent 2018
Deep Dive on PostgreSQL Databases on Amazon RDS (DAT324) - AWS re:Invent 2018Amazon Web Services
 
InnoDB Performance Optimisation
InnoDB Performance OptimisationInnoDB Performance Optimisation
InnoDB Performance OptimisationMydbops
 
Using Apache Spark to Solve Sessionization Problem in Batch and Streaming
Using Apache Spark to Solve Sessionization Problem in Batch and StreamingUsing Apache Spark to Solve Sessionization Problem in Batch and Streaming
Using Apache Spark to Solve Sessionization Problem in Batch and StreamingDatabricks
 
Amazon EMR Deep Dive & Best Practices
Amazon EMR Deep Dive & Best PracticesAmazon EMR Deep Dive & Best Practices
Amazon EMR Deep Dive & Best PracticesAmazon Web Services
 

What's hot (20)

Amazon DynamoDB Deep Dive Advanced Design Patterns for DynamoDB (DAT401) - AW...
Amazon DynamoDB Deep Dive Advanced Design Patterns for DynamoDB (DAT401) - AW...Amazon DynamoDB Deep Dive Advanced Design Patterns for DynamoDB (DAT401) - AW...
Amazon DynamoDB Deep Dive Advanced Design Patterns for DynamoDB (DAT401) - AW...
 
Aurora는 어떻게 다른가 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 Gaming
Aurora는 어떻게 다른가 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 GamingAurora는 어떻게 다른가 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 Gaming
Aurora는 어떻게 다른가 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 Gaming
 
Diving into Delta Lake: Unpacking the Transaction Log
Diving into Delta Lake: Unpacking the Transaction LogDiving into Delta Lake: Unpacking the Transaction Log
Diving into Delta Lake: Unpacking the Transaction Log
 
AWS로 사용자 천만 명 서비스 만들기 (윤석찬)- 클라우드 태권 2015
AWS로 사용자 천만 명 서비스 만들기 (윤석찬)- 클라우드 태권 2015 AWS로 사용자 천만 명 서비스 만들기 (윤석찬)- 클라우드 태권 2015
AWS로 사용자 천만 명 서비스 만들기 (윤석찬)- 클라우드 태권 2015
 
Percona xtrabackup - MySQL Meetup @ Mumbai
Percona xtrabackup - MySQL Meetup @ MumbaiPercona xtrabackup - MySQL Meetup @ Mumbai
Percona xtrabackup - MySQL Meetup @ Mumbai
 
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
[pgday.Seoul 2022] 서비스개편시 PostgreSQL 도입기 - 진소린 & 김태정
 
AWS CloudFormation Best Practices
AWS CloudFormation Best PracticesAWS CloudFormation Best Practices
AWS CloudFormation Best Practices
 
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
 
Deep Dive on AWS CloudFormation
Deep Dive on AWS CloudFormationDeep Dive on AWS CloudFormation
Deep Dive on AWS CloudFormation
 
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
 
Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...
Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...
Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...
 
Deep Dive on MySQL Databases on Amazon RDS (DAT322) - AWS re:Invent 2018
Deep Dive on MySQL Databases on Amazon RDS (DAT322) - AWS re:Invent 2018Deep Dive on MySQL Databases on Amazon RDS (DAT322) - AWS re:Invent 2018
Deep Dive on MySQL Databases on Amazon RDS (DAT322) - AWS re:Invent 2018
 
Serverless with IAC - terraform과 cloudformation 비교
Serverless with IAC - terraform과 cloudformation 비교Serverless with IAC - terraform과 cloudformation 비교
Serverless with IAC - terraform과 cloudformation 비교
 
[Pgday.Seoul 2017] 2. PostgreSQL을 위한 리눅스 커널 최적화 - 김상욱
[Pgday.Seoul 2017] 2. PostgreSQL을 위한 리눅스 커널 최적화 - 김상욱[Pgday.Seoul 2017] 2. PostgreSQL을 위한 리눅스 커널 최적화 - 김상욱
[Pgday.Seoul 2017] 2. PostgreSQL을 위한 리눅스 커널 최적화 - 김상욱
 
How to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScaleHow to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScale
 
Serverless Streaming Data Processing using Amazon Kinesis Analytics
Serverless Streaming Data Processing using Amazon Kinesis AnalyticsServerless Streaming Data Processing using Amazon Kinesis Analytics
Serverless Streaming Data Processing using Amazon Kinesis Analytics
 
Deep Dive on PostgreSQL Databases on Amazon RDS (DAT324) - AWS re:Invent 2018
Deep Dive on PostgreSQL Databases on Amazon RDS (DAT324) - AWS re:Invent 2018Deep Dive on PostgreSQL Databases on Amazon RDS (DAT324) - AWS re:Invent 2018
Deep Dive on PostgreSQL Databases on Amazon RDS (DAT324) - AWS re:Invent 2018
 
InnoDB Performance Optimisation
InnoDB Performance OptimisationInnoDB Performance Optimisation
InnoDB Performance Optimisation
 
Using Apache Spark to Solve Sessionization Problem in Batch and Streaming
Using Apache Spark to Solve Sessionization Problem in Batch and StreamingUsing Apache Spark to Solve Sessionization Problem in Batch and Streaming
Using Apache Spark to Solve Sessionization Problem in Batch and Streaming
 
Amazon EMR Deep Dive & Best Practices
Amazon EMR Deep Dive & Best PracticesAmazon EMR Deep Dive & Best Practices
Amazon EMR Deep Dive & Best Practices
 

Similar to AWS re:Invent 2017 - Advanced Design Patterns for Amazon DynamoDB

Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...Amazon Web Services
 
SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
 SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ... SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...Amazon Web Services
 
Applying AWS Purpose-Built Database Strategy - SRV307 - Anaheim AWS Summit
Applying AWS Purpose-Built Database Strategy - SRV307 - Anaheim AWS SummitApplying AWS Purpose-Built Database Strategy - SRV307 - Anaheim AWS Summit
Applying AWS Purpose-Built Database Strategy - SRV307 - Anaheim AWS SummitAmazon Web Services
 
Applying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS Summit
Applying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS SummitApplying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS Summit
Applying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS SummitAmazon Web Services
 
Tinder and DynamoDB: It's a Match! Massive Data Migration, Zero Down Time - D...
Tinder and DynamoDB: It's a Match! Massive Data Migration, Zero Down Time - D...Tinder and DynamoDB: It's a Match! Massive Data Migration, Zero Down Time - D...
Tinder and DynamoDB: It's a Match! Massive Data Migration, Zero Down Time - D...Amazon Web Services
 
Journey Towards Scaling Your API to 10 Million Users
Journey Towards Scaling Your API to 10 Million UsersJourney Towards Scaling Your API to 10 Million Users
Journey Towards Scaling Your API to 10 Million UsersAdrian Hornsby
 
Amazon.com - Replacing 100s of Oracle DBs with Just One: DynamoDB - ARC406 - ...
Amazon.com - Replacing 100s of Oracle DBs with Just One: DynamoDB - ARC406 - ...Amazon.com - Replacing 100s of Oracle DBs with Just One: DynamoDB - ARC406 - ...
Amazon.com - Replacing 100s of Oracle DBs with Just One: DynamoDB - ARC406 - ...Amazon Web Services
 
RET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdf
RET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdfRET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdf
RET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdfAmazon Web Services
 
DynamoDB adaptive capacity: smooth performance for chaotic workloads - DAT327...
DynamoDB adaptive capacity: smooth performance for chaotic workloads - DAT327...DynamoDB adaptive capacity: smooth performance for chaotic workloads - DAT327...
DynamoDB adaptive capacity: smooth performance for chaotic workloads - DAT327...Amazon Web Services
 
How to build a data lake with aws glue data catalog (ABD213-R) re:Invent 2017
How to build a data lake with aws glue data catalog (ABD213-R)  re:Invent 2017How to build a data lake with aws glue data catalog (ABD213-R)  re:Invent 2017
How to build a data lake with aws glue data catalog (ABD213-R) re:Invent 2017Amazon Web Services
 
Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...
Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...
Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...Amazon Web Services
 
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018Amazon Web Services
 
Deep Learning at AWS: Embedding & Attention Models
Deep Learning at AWS: Embedding & Attention ModelsDeep Learning at AWS: Embedding & Attention Models
Deep Learning at AWS: Embedding & Attention ModelsAmazon Web Services
 
Data Warehousing and Data Lake Analytics, Together - AWS Online Tech Talks
Data Warehousing and Data Lake Analytics, Together - AWS Online Tech TalksData Warehousing and Data Lake Analytics, Together - AWS Online Tech Talks
Data Warehousing and Data Lake Analytics, Together - AWS Online Tech TalksAmazon Web Services
 
ABD307_Deep Analytics for Global AWS Marketing Organization
ABD307_Deep Analytics for Global AWS Marketing OrganizationABD307_Deep Analytics for Global AWS Marketing Organization
ABD307_Deep Analytics for Global AWS Marketing OrganizationAmazon Web Services
 
Real-time Analytics using Data from IoT Devices - AWS Online Tech Talks
Real-time Analytics using Data from IoT Devices - AWS Online Tech TalksReal-time Analytics using Data from IoT Devices - AWS Online Tech Talks
Real-time Analytics using Data from IoT Devices - AWS Online Tech TalksAmazon Web Services
 
GAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
GAM310_Build a Telemetry and Analytics Pipeline for Game BalancingGAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
GAM310_Build a Telemetry and Analytics Pipeline for Game BalancingAmazon Web Services
 
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...Amazon Web Services
 
FSV305-Optimizing Payments Collections with Containers and Machine Learning
FSV305-Optimizing Payments Collections with Containers and Machine LearningFSV305-Optimizing Payments Collections with Containers and Machine Learning
FSV305-Optimizing Payments Collections with Containers and Machine LearningAmazon Web Services
 
Building mobile apps that can automatically scale globally to millions of use...
Building mobile apps that can automatically scale globally to millions of use...Building mobile apps that can automatically scale globally to millions of use...
Building mobile apps that can automatically scale globally to millions of use...AWS Germany
 

Similar to AWS re:Invent 2017 - Advanced Design Patterns for Amazon DynamoDB (20)

Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
Building with AWS Databases: Match Your Workload to the Right Database (DAT30...
 
SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
 SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ... SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
 
Applying AWS Purpose-Built Database Strategy - SRV307 - Anaheim AWS Summit
Applying AWS Purpose-Built Database Strategy - SRV307 - Anaheim AWS SummitApplying AWS Purpose-Built Database Strategy - SRV307 - Anaheim AWS Summit
Applying AWS Purpose-Built Database Strategy - SRV307 - Anaheim AWS Summit
 
Applying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS Summit
Applying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS SummitApplying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS Summit
Applying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS Summit
 
Tinder and DynamoDB: It's a Match! Massive Data Migration, Zero Down Time - D...
Tinder and DynamoDB: It's a Match! Massive Data Migration, Zero Down Time - D...Tinder and DynamoDB: It's a Match! Massive Data Migration, Zero Down Time - D...
Tinder and DynamoDB: It's a Match! Massive Data Migration, Zero Down Time - D...
 
Journey Towards Scaling Your API to 10 Million Users
Journey Towards Scaling Your API to 10 Million UsersJourney Towards Scaling Your API to 10 Million Users
Journey Towards Scaling Your API to 10 Million Users
 
Amazon.com - Replacing 100s of Oracle DBs with Just One: DynamoDB - ARC406 - ...
Amazon.com - Replacing 100s of Oracle DBs with Just One: DynamoDB - ARC406 - ...Amazon.com - Replacing 100s of Oracle DBs with Just One: DynamoDB - ARC406 - ...
Amazon.com - Replacing 100s of Oracle DBs with Just One: DynamoDB - ARC406 - ...
 
RET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdf
RET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdfRET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdf
RET305-Turbo Charge Your E-Commerce Site wAmazon Cache and Search Solutions.pdf
 
DynamoDB adaptive capacity: smooth performance for chaotic workloads - DAT327...
DynamoDB adaptive capacity: smooth performance for chaotic workloads - DAT327...DynamoDB adaptive capacity: smooth performance for chaotic workloads - DAT327...
DynamoDB adaptive capacity: smooth performance for chaotic workloads - DAT327...
 
How to build a data lake with aws glue data catalog (ABD213-R) re:Invent 2017
How to build a data lake with aws glue data catalog (ABD213-R)  re:Invent 2017How to build a data lake with aws glue data catalog (ABD213-R)  re:Invent 2017
How to build a data lake with aws glue data catalog (ABD213-R) re:Invent 2017
 
Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...
Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...
Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...
 
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018
 
Deep Learning at AWS: Embedding & Attention Models
Deep Learning at AWS: Embedding & Attention ModelsDeep Learning at AWS: Embedding & Attention Models
Deep Learning at AWS: Embedding & Attention Models
 
Data Warehousing and Data Lake Analytics, Together - AWS Online Tech Talks
Data Warehousing and Data Lake Analytics, Together - AWS Online Tech TalksData Warehousing and Data Lake Analytics, Together - AWS Online Tech Talks
Data Warehousing and Data Lake Analytics, Together - AWS Online Tech Talks
 
ABD307_Deep Analytics for Global AWS Marketing Organization
ABD307_Deep Analytics for Global AWS Marketing OrganizationABD307_Deep Analytics for Global AWS Marketing Organization
ABD307_Deep Analytics for Global AWS Marketing Organization
 
Real-time Analytics using Data from IoT Devices - AWS Online Tech Talks
Real-time Analytics using Data from IoT Devices - AWS Online Tech TalksReal-time Analytics using Data from IoT Devices - AWS Online Tech Talks
Real-time Analytics using Data from IoT Devices - AWS Online Tech Talks
 
GAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
GAM310_Build a Telemetry and Analytics Pipeline for Game BalancingGAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
GAM310_Build a Telemetry and Analytics Pipeline for Game Balancing
 
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
 
FSV305-Optimizing Payments Collections with Containers and Machine Learning
FSV305-Optimizing Payments Collections with Containers and Machine LearningFSV305-Optimizing Payments Collections with Containers and Machine Learning
FSV305-Optimizing Payments Collections with Containers and Machine Learning
 
Building mobile apps that can automatically scale globally to millions of use...
Building mobile apps that can automatically scale globally to millions of use...Building mobile apps that can automatically scale globally to millions of use...
Building mobile apps that can automatically scale globally to millions of use...
 

More from Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

More from Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

AWS re:Invent 2017 - Advanced Design Patterns for Amazon DynamoDB

  • 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Advanced Design Patterns for Amazon DynamoDB R i c k H o u l i h a n – S e n i o r P r a c t i c e M a n a g e r , N o S Q L A W S P r o f e s s i o n a l S e r v i c e s D A T 4 0 3 N o v e m b e r 2 9 , 2 0 1 7
  • 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What to Expect from the Session • When to use NoSQL and why • Brief overview of Amazon DynamoDB • Hot keys, design considerations • NoSQL data modeling • Normalized versus unstructured schema • Common NoSQL design patterns • Time series, write-sharding, MVCC, etc. • Amazon DynamoDB in the serverless ecosystem
  • 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is a database? “A place to put stuff my app needs.” – Average Developer
  • 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Technology adoption and the hype curve
  • 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why NoSQL? Optimized for storage Optimized for compute Normalized/relational Denormalized/hierarchical Ad hoc queries Instantiated views Scale vertically Scale horizontally Good for OLAP Built for OLTP at scale SQL NoSQL
  • 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon DynamoDB Document or key-value Scales to any workloadFully managed NoSQL Access control Event-driven programmingFast and consistent
  • 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Table Table Items Attributes Partition key Sort key Mandatory Key-value access pattern Determines data distribution Optional Model 1:N relationships Enables rich query capabilities All items for key ==, <, >, >=, <= “begins with” “between” “contains” “in” sorted results counts top/bottom N values
  • 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. NoSQL data modeling
  • 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. It’s all about aggregations… Document management Process controlSocial network Data treesIT monitoring
  • 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. SQL vs. NoSQL design pattern
  • 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Design patterns and best practices
  • 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DynamoDB Streams and AWS Lambda
  • 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Triggers Lambda function Notify change Item/table level metrics Amazon CloudSearch Kinesis Firehose
  • 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Write sharding Handling high velocity writes
  • 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Partition 1 1000 WCUs Partition K 1000 WCUs Partition M 1000 WCUs Partition N 1000 WCUs Votes Table Candidate A Candidate B Scaling bottlenecks Voters Provision 200,000 WCUs
  • 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Queue-based load leveling Workers Dashboard SQS • Write incoming votes to SQS • Scale workers as needed to process queue • Avoid key pressure
  • 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Write sharding Candidate A_2 Candidate B_1 Candidate B_2 Candidate B_3 Candidate B_5 Candidate B_4 Candidate B_7 Candidate B_6 Candidate A_1 Candidate A_3 Candidate A_4 Candidate A_7 Candidate B_8 Candidate A_6 Candidate A_8 Candidate A_5 Voter Votes Table
  • 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Write sharding Candidate A_2 Candidate B_1 Candidate B_2 Candidate B_3 Candidate B_5 Candidate B_4 Candidate B_7 Candidate B_6 Candidate A_1 Candidate A_3 Candidate A_4 Candidate A_7 Candidate B_8 Insert: “CandidateA_” + rand(0, 10) Candidate A_6 Candidate A_8 Candidate A_5 Voter Votes Table
  • 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Votes Table Shard aggregation Candidate A_2 Candidate B_1 Candidate B_2 Candidate B_3 Candidate B_5 Candidate B_4 Candidate B_7 Candidate B_6 Candidate A_1 Candidate A_3 Candidate A_4 Candidate A_5 Candidate A_6 Candidate A_8 Candidate A_7 Candidate B_8 Periodic process Candidate A Total: 2.5M 1. Sum 2. Store Voter
  • 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Query a single sharded item efficiently Candidate A_2 Candidate B_1 Candidate B_2 Candidate B_3 Candidate B_5 Candidate B_4 Candidate B_7 Candidate B_6 Candidate A_1 Candidate A_3 Candidate A_4 Candidate A_7 Candidate B_8 Insert: “CandidateA_” + hash(SSN) Candidate A_6 Candidate A_8 Candidate A_5 Voter Votes Table
  • 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Calculating partition counts (reads) Items per partition Average item size RCU Size Requests per second 100 K * 0.2 KB / 4 KB * 10 / 3000 = Partition max RCU ~ 17
  • 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Calculating partition counts (writes) Items per second Average item size 100 K * (itemSize < 1KB ? 1KB : itemSize) / 1000 = Partition max WCU 100
  • 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Increase throughput with concurrency Consider RCU/WCU per key, item size and request rate Shard write-heavy partition keys your write workload is not horizontally scalable
  • 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Time-based workflows Processing the entire table efficiently
  • 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Finding expired items Active Tickets_Table Event_id (Partition) Timestamp GSIKey Rand(0-N) … Attribute N Expired Tickets GSI GSIKey (Partition) Timestamp (Sort) Archive Table Event_id (Partition) Timestamp (Sort) Attribute1 …. Attribute N RCUs = 10000 WCUs = 10000 RCUs = 100 WCUs = 1 Current table HotdataColddata Scatter query GSI for expired tickets and use TTL to archive
  • 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Find items efficiently with AWS Lambda Use a write-sharded GSI to selectively query the entire table Create a Lambda “stored procedure” to process items Migrate data between tables with TTL/Streams/Lambda there is a need to query all items on the table selectively
  • 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Product catalog Popular items (read)
  • 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Partition 1 2000 RCUs Partition K 2000 RCUs Partition M 2000 RCUs Partition 50 2000 RCU Scaling bottlenecks Product A Product B Shoppers Product Catalog Table SELECT Id, Description, ... FROM ProductCatalog WHERE Id="POPULAR_PRODUCT"
  • 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Fully managed, highly available: handles all software management, fault tolerant, replication across multi-AZs within a region • DynamoDB API compatible: seamlessly caches DynamoDB API calls, no application re-writes required • Write-through: DAX handles caching for writes • Flexible: configure DAX for one table or many • Scalable: scales-out to any workload with up to 10 read replicas • Manageability: fully integrated AWS service: Amazon CloudWatch, tagging for DynamoDB, AWS Console • Security: Amazon VPC, AWS IAM, AWS CloudTrail, AWS Organizations Features DynamoDB Accelerator (DAX) DynamoDB Your Applications DynamoDB Accelerator
  • 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Targeting queries Query filters, composite keys, and sparse indexes
  • 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Secondary index Opponent Date GameId Status Host Alice 2014-10-02 d9bl3 DONE David Carol 2014-10-08 o2pnb IN_PROGRESS Bob Bob 2014-09-30 72f49 PENDING Alice Bob 2014-10-03 b932s PENDING Carol Bob 2014-10-03 ef9ca IN_PROGRESS David BobPartition key Sort key Multi-value sorts and filters
  • 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Secondary Index Approach 1: Query filter Bob Opponent Date GameId Status Host Alice 2014-10-02 d9bl3 DONE David Carol 2014-10-08 o2pnb IN_PROGRESS Bob Bob 2014-09-30 72f49 PENDING Alice Bob 2014-10-03 b932s PENDING Carol Bob 2014-10-03 ef9ca IN_PROGRESS David SELECT * FROM Game WHERE Opponent='Bob' ORDER BY Date DESC FILTER ON Status='PENDING' (filtered out)
  • 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Approach 2: Composite key StatusDate DONE_2014-10-02 IN_PROGRESS_2014-10-08 IN_PROGRESS_2014-10-03 PENDING_2014-09-30 PENDING_2014-10-03 Status DONE IN_PROGRESS IN_PROGRESS PENDING PENDING Date 2014-10-02 2014-10-08 2014-10-03 2014-10-03 2014-09-30 + =
  • 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Secondary Index Approach 2: Composite key Opponent StatusDate GameId Host Alice DONE_2014-10-02 d9bl3 David Carol IN_PROGRESS_2014-10-08 o2pnb Bob Bob IN_PROGRESS_2014-10-03 ef9ca David Bob PENDING_2014-09-30 72f49 Alice Bob PENDING_2014-10-03 b932s Carol Partition key Sort key
  • 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Opponent StatusDate GameId Host Alice DONE_2014-10-02 d9bl3 David Carol IN_PROGRESS_2014-10-08 o2pnb Bob Bob IN_PROGRESS_2014-10-03 ef9ca David Bob PENDING_2014-09-30 72f49 Alice Bob PENDING_2014-10-03 b932s Carol Secondary index Approach 2: Composite key Bob SELECT * FROM Game WHERE Opponent='Bob' AND StatusDate BEGINS_WITH 'PENDING'
  • 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sparse indexes Id (Partition) User Game Score Date Award 1 Bob G1 1300 2012-12-23 2 Bob G1 1450 2012-12-23 3 Jay G1 1600 2012-12-24 4 Mary G1 2000 2012-10-24 Champ 5 Ryan G2 123 2012-03-10 6 Jones G2 345 2012-03-20 Game-scores-table Award (Partition) Id User Score Champ 4 Mary 2000 Award-GSI Scan sparse GSIs
  • 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Concatenate attributes to form useful secondary index keys Take advantage of sparse indexes Replace filter with indexes you want to optimize a query as much as possible Status + Date
  • 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vertical Partitioning Large items Filters vs. indexes M:N modeling—inbox and outbox
  • 41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reports Table Workflow Management App David SELECT * FROM Reports WHERE Owner = ‘David’ AND State = ‘Pending’ LIMIT 50 ORDER BY Date DESC Pending SELECT * FROM Reports WHERE Owner = ‘David’ AND State = ‘Processed’ LIMIT 50 ORDER BY Date DESC Processed
  • 42. Owner StateDate Document David Pending#2014-10-02 … … many more Reports for David … David Processed#2014-10-03 … Alice Pending#2014-09-28 … Alice Pending#2014-10-01 … Large and small attributes mixed (Many more report items) David Reports table 50 items × 256 KB each Partition key Sort key Large attachments SELECT * FROM Reports WHERE Recipient=‘David’ AND Status=‘Pending’ LIMIT 50 ORDER BY Date DESC Inbox © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Computing read query cost Items evaluated by query Average item size Conversion ratio Eventually consistent reads 50 * 256KB * (1 RCU / 4 KB) * (1 / 2) = 1600 RCU
  • 44. Owner StateDate Summary ReportID David Pending#2014-10-02 … afed David Processed#2014-10-03 … 3kf8 Alice Processed#2014-09-28 … 9d2b Alice Processed#2014-10-01 … ct7r Separate the bulk data Pending-GSI Reports table ReportID Body 9d2b … 3kf8 … ct7r … afed … David 1. Query Pending-GSI: 1 RCU 2. BatchGetItem messages: 1600 RCU (50 separate items at 256 KB) (50 sequential items at 128 bytes) Uniformly distributes large item reads © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Messaging app Reports table David Pending GSI Pending Processed GSI Processed
  • 46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reduce one-to-many item sizes Configure secondary index projections Use GSIs to model M:N relationship Distribute large items querying many large items at once ProcessedReportsPending
  • 47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Advanced data modeling
  • 48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Multi-version concurrency Transactions in NoSQL
  • 49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How OLTP apps use data  Mostly hierarchical structures  Entity driven workflows  Data spread across tables  Requires complex queries  Primary driver for ACID
  • 50. ItemID (PK) Version (SK) CurVer Attrs 1 v0 2 … v1 … v2 … v3 … Emulating ACID transactions (Many more item partitions) Item versions Overwrite v0 Item to Commit changes COPY Item.v0 -> Item1.v3 IF Item.v3 == NULL UPDATE Item1.v3 SET Attr1 += 1 UPDATE Item1.v3 SET Attr2 = … UPDATE Item1.v3 SET Attr3 = … COPY Item1.v3 -> Item1.v0 SET CurVer = 3 Transaction © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Versioning item sets Get all items for a product assembly SELECT * FROM Assemblies WHERE ID=1 Order Item Picker ID PartID Vendor Bin 1 1 ACME 27Z19 2 ACME 39M97 3 ACME 75B25 (Many more assemblies) Assemblies table Partition key Sort key
  • 52. ID PartID Vendor Bin 1 1 ACME 27Z19 2 ACME 39M97 3 ACME 75B25 4 UAC 53G56 5 UAC 64B17 6 UAC 48J19 Updates are problematic (Many more assemblies) Assemblies table Old items SELECT * FROM Assemblies WHERE ID=1 Order Item Picker New items © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 53. ID PartID Vendor Bin 1 1 ACME 27Z19 2 ACME 39M97 3 ACME 75B25 4 UAC 53G56 5 UAC 64B17 6 UAC 48J19 Creating partition “locks” (Many more assemblies) Assemblies table • Use metadata item for versioning SET #ItemList.i = list append(:newList, #ItemList.i ), #Lock = 1 • Obtain locks with conditional writes --condition-expression “Lock = 0” • Remove old items or add version info to Sort Key Query composite keys for specific versions • Readers can determine how to handle “fat” reads • Add additional metadata to manage transactional workflow as needed Current state, breadcrumbs, timeout, etc. ID PartID ItemList Lock 1 0 {i:[[1,2,3],[4,5,6]]} 0 © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Use item partitions to manage transactional workflows Manage versioning across items with metadata Tag attributes to maintain multiple versions Code your app to recognize when updates are in progress Implement app layer error handling and recovery logic transactional writes across items is required
  • 55. Geo-hashing FleetID (PK) Location (SK) GSIKey Rand(0-N) GUID 0201200 … 0213211 … 0233321 … 0320011 … • Break down map area into a grid • Continue to subdivide grid sections to narrow down search areas • Use range queries to find nearby items • Configure write-sharded GSI to query across partitions GSIKey (PK) Location (SK) FleetID 0-N … GUID … GUID … GUID … GUID
  • 56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Hierarchical data Composite key modeling
  • 57. Hierarchical data structures as items • Use composite sort key to define a hierarchy • Highly selective queries with sort conditions • Reduce query complexity © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. … or as documents (JSON) JSON data types (M, L, BOOL, NULL) Document SDKs available Indexing only by using DynamoDB Streams or AWS Lambda 400 KB maximum item size (limits hierarchical data structure) Primary Key Attributes PK Items BookID type title author genre publisher datePublished ISBN Book Ringworld Larry Niven Science Fiction Ballantine Oct-70 0-345-02046-4 AlbumID type title artist genre Attributes Album Dark Side of the Moon Pink Floyd Progressive Rock { label:"Harvest", studio: "Abbey Road", published: "3/1/73", producer: "Pink Floyd", tracks: [{title: "Speak to Me", length: "1:30", music: "Mason", vocals: "Instrumental"},{title: ”Breathe", length: ”2:43", music: ”Waters, Gilmour, Wright", vocals: ”Gilmour"},{title: ”On the Run", length: “3:30", music: ”Gilmour, Waters", vocals: "Instrumental"}]} MovieID type title genre writer Attributes Movie Idiocracy Scifi Comedy Mike Judge { producer: "20th Century Fox", actors: [{ name: "Luke Wilson", dob: "9/21/71", character: "Joe Bowers", image: "img2.jpg"},{ name: "Maya Rudolph", dob: "7/27/72", character: "Rita", image: "img1.jpg"},{ name: "Dax Shepard", dob: "1/2/75", character: "Frito Pendejo", image: "img3.jpg"}]
  • 59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. … or as nested sets • Store data trees on a table as array of leaf items • Non-leaf items define start/end index values • Query for leaf items by category using index ranges
  • 60. • Partition table on node ID, add edges to define adjacency list • Define a default edge for every node type to describe the node itself • Use partitioned GSIs to query large nodes (dates, places, etc.) • Use Dynamo DB Streams/Lambda/EMR for graph query projections • Neighbor entity state • Subtree aggregations • Breadth first search • Node ranking Adjacency lists and materialized graphs GSI Primary Key Attributes GSIkey Data Target Type Node Projection 0-N Jason Bourne 1 Person 1 … James John Doe 4 4 … 20170418 2 Birthdate 1 … 4 … Date 2 … Finland 3 Birthplace 1 … 4 … Place 3 … GSI Primary Key Attributes GSIkey Type Target Data Node Projection 0-N Person 1 Jason Bourne 1 … 4 John Doe 4 … Birthdate 2 20170418 1 … 4 … Birthplace 3 Finland 1 … 4 … Date 2 20170418 2 … Place 3 Finland 3 … Table Primary Key Attributes Node Type Target Data GSIkey Projection 1 Person 1 Jason Bourne HASH(Person.Data) Edge/spanning tree rollups Birthdate 2 20170418 Birthplace 3 Finland 2 Date 2 20170418 HASH(Data) (Summary Stats) 3 Place 3 Finland 4 Person 4 John Doe HASH(Person.Data Edge/spanning tree rollups Birthdate 2 20170418 Birthplace 3 Finland
  • 61. Audible eBook sync service • Allows users to save session state for Audible eBooks • Maintains mappings per user for eBooks and audio products • Spikey load patterns require significant overprovisioning • Large number of access patterns © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Access patterns # ACCESS PATTERNS: https://w.amazon.com/index.php/WfV/Services/DBMigration USE CASE 1 CompanionMapping getCompanionMappingsByAsin 2 CompanionMapping getCompanionMappingsByEbookAndAudiobookContentId 3 CompanionMapping getCompanionMappingsFromCache 4 CompanionMapping getCompanionMappings 5 CompanionMapping getCompanionMappingsAvailable 6 AcrInfo getACRInfo 7 AcrInfo getACRs 8 AcrInfo getACRInfos 9 AcrInfo getACRInfosbySKU 10 AudioProduct getAudioProductsForACRs 11 AudioProduct getAudioProducts 12 AudioProduct deleteAudioProductsMatchingSkuVersions 13 AudioProduct getChildAudioProductsForSKU 14 Product getProductInfoByAsins 15 Product getParentChildDataByParentAsins 16 AudioFile getAudioFilesForACR 17 AudioFile getAudioFilesForChildACR 18 AudioFile getAudioFilesByParentAsinVersionFormat 19 AudioFile getAudioFiles 20 AudioFile getAudioFilesForChildAsin
  • 63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Primary table T A B L E Primary Key Attributes PK SK (GSI 3) ABOOKACR1 v0#ABOOKACR1 GSI-1 GSI-2 ABOOK-ASIN1 ABOOK-SKU1 EBOOKACR1 GSI-1 GSI-2 SyncFileAcr ABOOK-ASIN1 ABOOKACR1#TRACK#1 GSI-1 GSI-2 ABOOK-ASIN1 ABOOK-SKU1 ABOOKACR1#TRACK#2 GSI-1 GSI-2 ABOOK-ASIN1 ABOOK-SKU1 EBOOKACR1 EBOOKACR1 GSI-1 EBookAsin EBOOK-SKU1 ASIN
  • 64. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Indexes G S I 1 Partition Key Projected Attributes ABOOK-ASIN1 ABOOKACR1 ABOOKACR1-v1 ABOOKACR1#TRACK#1 ABOOKACR1#TRACK#2 SyncFileAcr ABOOKACR1 MAP-EBOOKACR1 EBOOK-SKU1 ABOOKACR1 EBOOKACR1 G S I 2 Partition Key Projected Attributes ABOOK-ASIN1 ABOOKACR1 MAP-EBOOKACR1 ABOOK-SKU1 ABOOKACR1 ABOOKACR1-v1 ABOOKACR1#TRACK#1 ABOOKACR1#TRACK#2 G S I 3 GSI Partition Key Projected Attributes V0#ABOOKACR1 ABOOKACR1 ABOOKACR1-v1 EBOOKACR1 ABOOKACR1 MAP-EBOOKACR1 ABOOKACR1#TRACK#1 ABOOKACR1 ABOOKACR1#TRACK#1 ABOOKACR1#TRACK#2 ABOOKACR1 ABOOKACR1#TRACK#2 EBOOKACR1 ABOOKACR1 EBOOKACR1
  • 65. Query conditions # ACCESS PATTERNS: https://w.amazon.com/index.php/WfV/Services/DBMigration USE CASE Lookup parameters INDEX Key Conditions Filter Conditions 1 CompanionMapping getCompanionMappingsByAsin audiobookAsin/ebookSku GSI2 GSI-2=ABOOK-ASIN1 None 2 CompanionMapping getCompanionMappingsByEbookAndAudi obookContentId ebookAcr/sku,version,format or audiobookAcr/asin,version,format GSI-3 on TargetACR attribute OR PrimaryKey on Table GSI-3=MAP-EBOOKACR1 version=v and format=f 3 CompanionMapping getCompanionMappingsFromCache ebookAcr/sku,version,format or audiobookAcr/asin,version,format GSI-3 on TargetACR attribute OR PrimaryKey on Table GSI-3=MAP-EBOOKACR1 version=v and format=f 4 CompanionMapping getCompanionMappings syncfileAcr, ebookAcr?, audiobookAcr? GSI1 GSI-1=SyncFileAcr None 5 CompanionMapping getCompanionMappingsAvailable ebookAcr, audiobookAcr Primary Key on Table Acr=ABOOKACR1 and TargetACR beginswith "MAP-" 6 AcrInfo getACRInfo acr Primary Key on Table Acr=ABOOKACR1 and TargetACR beginswith "ABOOKACR1-v" 7 AcrInfo getACRs acr / asin,version,format Primary Key on Table Acr=ABOOKACR1 version=v and format=f 8 AcrInfo getACRInfos acr Primary Key on table Acr=ABOOKACR1 and TargetACR beginswith "ABOOKACR1" 9 AcrInfo getACRInfosbySKU sku GSI2 GSI-2=ABOOK-SKU1 10 AudioProduct getAudioProductsForACRs acr Primary Key on table Acr=ABOOKACR1 and TargetACR beginswith "ABOOKACR1" 11 AudioProduct getAudioProducts sku, version, format GSI2 GSI-2=ABOOK-SKU1 version=v and format=f 12 AudioProduct deleteAudioProductsMatchingSkuVersions sku, version GSI2 GSI-2=ABOOK-SKU1 version=v 13 AudioProduct getChildAudioProductsForSKU sku GSI2 GSI-2=ABOOK-SKU1 14 Product getProductInfoByAsins asin GSI1 GSI-1=ABOOK-ASIN1 15 Product getParentChildDataByParentAsins asin GSI1 GSI-1=ABOOK-ASIN1 16 AudioFile getAudioFilesForACR acr Primary Key on table Acr=ABOOKACR1 and TargetACR beginswith "ABOOKACR1#" 17 AudioFile getAudioFilesForChildACR acr, parent_asin Primary Key on table Acr=ABOOKACR1 version=v and format=f 18 AudioFile getAudioFilesByParentAsinVersionFormat parent_asin, version, format GSI1 GSI-1=ABOOK-ASIN1 version=v and format=f 19 AudioFile getAudioFiles sku, version, format GSI2 GSI-2=ABOOK-SKU1 version=v and format=f 20 AudioFile getAudioFilesForChildAsin asin, parent_asin, version, format GSI1 GSI-1=ABOOK-ASIN1 version=v and format=f
  • 66. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!