A visual introduction to Dynamo DB indexes and how to use them effectively. Presented at the Auckland AWS community meetup, August 2021. Copyright Centrapay 2021.

1. Dynamo DB
Indexes

2. Nathan Jones
Principal Engineer @ Centrapay
~15 Years Building Enterprise Software
Oracle, Postgres, Mongo DB, MySQL, Firebase
~16 Months Architecting Centrapay Platform
Dynamo DB
_ncjones
ncjones
nathan-c-jones

3. We unlock better ways for businesses and
consumers to connect, engage and pay.
Digital Assets & Payments
“Spend anything anywhere”
~70% NZ payment terminals now QR code enabled
centrapay.com

4. 1. Core Concepts
2. Indexes
3. 10 Patterns for Success
Contents

5. Core Concepts

6. ✔ Launched 2012
✔ Schemaless
✔ Serverless
✔ Atomic Writes
✔ Highly Scalable
✘ Reporting
✘ Search
Dynamo DB

7. Cassandra Firebase
Developer Experience cf.
Mongo
Dynamo DB

8. Users
User
Id: 543
Name: Isaac
User
Id: 856
Name: Matthew
Table
Item
Item
Max 400KB
Max 400KB
Unbounded
Items + Tables

9. Partition Key
ID = PK + SK
(Optional)
Sort Key
“Hash Attribute” “Range Attribute”
Item Identification

10. Users
User
PK Id: 543
Name: Isaac
User
PK Id: 856
Name: Matthew
Item Identification: PK

11. Item Identification: PK + SK
User Prefs
User Pref
PK UserId: 543
SK Name: email-alerts
Value: true
User Pref
PK UserId: 543
SK Name: email-frequency
Value: daily

12. User Pref
PK UserId: 543
SK Name: email-frequency
Value: weekly
Replace or Reject
Item Identification: Collision
User Prefs
User Pref
PK UserId: 543
SK Name: email-alerts
Value: true
User Pref
PK UserId: 543
SK Name: email-frequency
Value: daily

13. Users
User Pref
PK UserId: 543
SK Type: UserPref#email-alerts
Value: true
User
PK UserId: 543
SK Type: User
Name: Isaac
Heterogeneous Tables

14. Queries
Find PK = 543
User Prefs
User Pref
PK UserId: 543
SK Name: email-alerts
Value: true
User Pref
PK UserId: 543
SK Name: email-frequency
Value: daily
User Pref
PK UserId: 543
SK Name: session-ttl
Value: true

15. Queries
User Prefs
User Pref
PK UserId: 543
SK Name: email-alerts
Value: true
User Pref
PK UserId: 543
SK Name: email-frequency
Value: daily
User Pref
PK UserId: 543
SK Name: session-ttl
Value: true
Find PK = 543
and SK = session-ttl

16. User Prefs
User Pref
PK UserId: 543
SK Name: email-alerts
Value: true
User Pref
PK UserId: 543
SK Name: email-frequency
Value: daily
User Pref
PK UserId: 543
SK Name: session-ttl
Value: true
Find PK = 543
and SK starts with email
Queries

17. User Prefs Table
Partitions
Partition 2
User Pref
PK UserId: 543
SK Name: email-alerts
Value: true
Partition 1 Partition 3
User Pref
PK UserId: 856
SK Name: email-alerts
Value: true
User Pref
PK UserId: 543
SK Name: email-frequency
Value: daily
User Pref
PK UserId: 856
SK Name: email-frequency
Value: daily
User Pref
PK UserId: 238
SK Name: email-alerts
Value: true
User Pref
PK UserId: 238
SK Name: email-frequency
Value: daily
User Pref
PK UserId: 945
SK Name: email-alerts
Value: true

18. Request Router
...
hash(PK) → Partition ID
Read / Write Request PK
Partition 2
Partition 1 Partition N
Partitions

19. Max
Capacity
Partition 1 Partition 2 Partition 3 Partition 4
Write: 1000WCU
Read: 3000RCU
Capacity
Exceeded!
Read/Write Capacity

20. Partition
Leader
Replica Replica
Partition
Leader
Replica Replica
Partition
Leader
Replica Replica
Write Request
AZ 2 AZ 3
AZ 1
Replication

21. User Prefs (ap-southeast-1)
User Prefs (us-west-1)
Async Replication
User Pref
PK UserId: 543
SK Name: session-ttl
Value: true
User Pref
PK UserId: 543
SK Name: session-ttl
Value: true
��
��
Global Tables

22. Indexes

23. User
Id: 543
Name: Isaac
User
Id: 238
Name: Nathan
User
Id: 945
Name: Raymond
User
Id: 856
Name: Matthew
543
238
945
856
Index Records
Id = 856? Id = 856?
🏎
Indexed Query
O(log n)
🐢
Scan
O(n)
Indexes 101

24. Partition Key (PK)
Sort Key (SK)
Projected Attributes
🔐 Permissions
🐘 Read/Write Capacity
Dynamo Index Anatomy
+

25. Type Identifying Consistency
Primary ✔ Strongly Consistent
Global
Secondary
✘ Eventually Consistent
Local
Secondary
✘ Strongly Consistent
Dynamo Index Types

26. ✔ Identifying
✔ Strongly Consistent
Primary Index

27. User
Device
Session
SessionId 123
UserId 945
DeviceId 1342
Created 2021-07-14
Expires 2021-07-15 Id 1342
Description iPhone X
Id 945
Name Raymond
Primary Index: Example

28. Session
Index: Get Session by Id
Partition Key Session Id
Sort Key n/a
Projected Attributes All
PK SessionId 123
UserId 945
DeviceId 1342
Created 2021-07-14
Expires 2021-07-15
Primary Index: Example

29. ✔ Non Identifying
✔ Eventually Consistent
Global Secondary Index (GSI)

30. Session
Index: Get Sessions by Device Id
Partition Key Device Id
Sort Key Created Date
Projected Attributes UserId, SessionId
PK SessionId 123
UserId 945
PK DeviceId 1342
SK Created 2021-07-14
Expires 2021-07-15
Global Secondary Index: Example

31. Primary Index Partition Global Secondary Index Partition
Session
PK SessionId: 123
UserId: 945
DeviceId: 1342
Created: 2021-07-14
Expires: 2021-07-15
Session
SessionId: 123
UserId: 945
PK DeviceId: 1342
SK Created: 2021-07-14
Async Replication
<1s Normally
GSI Eventual Consistency

32. ✔ Non Identifying
✔ Strongly Consistent
Local Secondary Index

33. 🤔 Share PK with Primary Index
🙁 Max 10GB
😧 No adding to existing table
😧 No partition rebalancing
Local Secondary Index

34. 10 Patterns for Success

35. User Pref
PK pk User#945
SK sk Pref#email-frequency
value daily
User Pref
PK UserId 945
SK Name email-frequency
value daily
✔
1. Generic Primary PK, SK

36. Customer
PK pk Customer#10034
SK sk #Customer
orgId 687
Customer
PK pk Org#687
SK sk Customer#10034
orgId 687
✔
2. Immutable Primary PK

37. Order
PK pk Order#10034
SK sk #Order
PK orgId 687
status waiting
updatedAt 2021-07-28T10:00:00Z
SK
sk.status.
updatedAt
waiting#2021-07-28T10:00:00Z
select * from Order
where
orgId = "687" and
status = "waiting"
order by updatedAt;
✔
3. Computed Secondary Keys

38. Order
PK pk Order#23043
SK sk #Order
PK orgId 687
status waiting
updatedAt 2021-07-28T10:00:00Z
SK
sk.status.
updatedAt
waiting#2021-07-28T10:00:00Z
Order
PK pk Order#23043
SK sk #Order
PK orgId 687
status processed
updatedAt 2021-07-28T10:01:05Z
SK
sk.status.
updatedAt
✔
(DELETED)
(Sparse Index)
4. Delete Keys to Drop from GSI

39. Unique User Handle
PK pk UserHandle#rayray
SK sk #Unique
User
PK pk User#945
SK sk #User
handle rayray
✔
5. Unique Constraints
Atomic Transaction

40. User Version
User
PK pk User#945
SK sk #User
version 2
PK pk User#945
SK sk Version#2
✔
Atomic Transaction
6. Optimistic Locking

41. Customer
PK pk Customer#f9c82a6b-7b7a-4d6f-bc48-8fffc88fc78b
SK sk #Customer
orgId bb7ecd0f-f856-499b-9fdf-7d4b058a2b3b
✔
7. Random IDs

42. ��
��
��
��
��
��
��
��
��
��
8. Understand Access Patterns

43. Import Job
PK pk Job#250688
SK sk #ImportJob
PK status pending
SK createdAt 2021-07-23T10:00:01
Import Job
PK pk Job#250688
SK sk #ImportJob
status pending
statusShard 15
PK pk.status pending.15
SK createdAt 2021-07-23T10:00:01
✔
(RANDOM / HASHED)
9. Use “Shard” PK Suffix

44. 10. Scripted Data Migrations

45. Conclusion
🔑 Choose keys for indexes carefully
💕 Use heterogeneous tables and multi-item transactions
⚖ Distribute partition load evenly

46. Official Best Practices
https://docs.aws.amazon.com/amazondynamodb/latest/develope
rguide/best-practices.html
Official Blogs
Dynamic Shards:
https://aws.amazon.com/blogs/database/choosing-the-right-num
ber-of-shards-for-your-large-scale-amazon-dynamodb-table
Adaptive Capacity:
https://aws.amazon.com/blogs/database/how-amazon-dynamod
b-adaptive-capacity-accommodates-uneven-data-access-pattern
s-or-why-what-you-know-about-dynamodb-might-be-outdated/
Important Resources
Alex DeBrie
Data Modelling:
https://www.youtube.com/watch?
v=DIQVJqiSUkE
🔥🔥🔥👍👍👍
Partitions:
https://www.alexdebrie.com/post
s/dynamodb-partitions/

47. Centrapay is Hiring
https://www.linkedin.com/company/centrapay/jobs/
We are always on the lookout for
talented individuals to join our team.
Lead Software Engineer