1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replacing 100s of Oracle DBs
with Just 1: DynamoDB
M i k e T h o m a s , S o f t w a r e D e v e l o p m e n t M a n a g e r
P i y u s h K a l a n i , S r . S o f t w a r e D e v e l o p m e n t E n g i n e e r
H o w A m a z o n ' s O r d e r P r o c e s s i n g W o r k f l o w S y s t e m L e a r n e d t o
S t o p W o r r y i n g a n d L o v e I t s S c a l e
A m a z o n . c o m
D e c e m b e r 1 , 2 0 1 7
AWS re:INVENT
A R C 4 0 6

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect from this session
Design challenges faced moving off Oracle and onto Amazon DynamoDB
We hope you will:
• Take away 5 “power” techniques that you may be able to use…
…if you find that not everything is provided “out of the box”
This talk is not:
• An introduction to Oracle
• An introduction to DynamoDB
• An exhaustive comparison of all designs, choices
• About how order processing at Amazon works

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
1. Introduce system and key functionality—what are our requirements?
2. Oracle-based architecture—where did it break down?
3. Design challenges moving to DynamoDB—how did we overcome them?
4. Results

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Herd
• Distributed workflow orchestration system
• Behind-the-scenes coordinator across the Amazon service-oriented architecture
• Stitch together individual services together, step by step
• For example: OrderProcessing workflow
Charge
Start
Ship
OrderApproved …
Email
Customer
Declined …

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Herd
Tell us to run a workflow, and we’ll drive it through to completion:
1. Record everything that happens, and guarantee it won’t get lost
• No order left behind
2. Retry when there’s a failure
• Stuff breaks
3. Help you monitor and debug errors (individually and in aggregate)
• Amazon has bugs, too
Hence, “Herd”

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data model
Work Item
Identifier Order:555-1111111-0000000
Workflow OrderProcessing
Scheduled Time 2017-11-03 17:08:00 UTC
Current Step Start
History [ ]

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Work Item
Identifier Order:555-1111111-0000000
Workflow OrderProcessing
Scheduled Time 2017-11-03 17:09:00 UTC
Current Step Start
History [ ]
Work Item
Identifier Order:555-1111111-0000000
Workflow OrderProcessing
Scheduled Time 2017-11-03 17:09:00 UTC
Current Step Charge
History [Start → Charge]
Work Item
Identifier Order:555-1111111-0000000
Workflow OrderProcessing
Scheduled Time 2017-11-03 17:09:00 UTC
Current Step Charge
History [Start → Charge,
“Declined”]
Work Item
Identifier Order:555-1111111-0000000
Workflow OrderProcessing
Scheduled Time 2017-11-03 17:09:00 UTC
Current Step Email Customer
History [Start → Charge,
“Declined”,
Charge → Email Customer]
Key functionality
1. History store
• Store a log of all events that occurred

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Work Item
Identifier Order:555-1111111-0000000
Workflow OrderProcessing
Scheduled Time 2017-11-03 17:09:00 UTC
Current Step Email Customer
History [Start → Charge,
“Declined”,
Charge → Email Customer]
Work Item
Identifier Order:555-1111111-0000000
Workflow OrderProcessing
Scheduled Time 2017-11-03 17:09:00 UTC
Current Step Email Customer
History [Start → Charge,
“Declined”,
Charge → Email Customer,
“InternalServerError”]
Key functionality
1. History store
• Store a log of all events that occurred
2. Scheduled queue
• Find the next work item that’s ready to run
Work Item
Identifier Order:555-1111111-0000000
Workflow OrderProcessing
Scheduled Time 2017-11-03 17:12:00 UTC
Current Step Email Customer
History [Start → Charge,
“Declined”,
Charge → Email Customer,
“InternalServerError”]

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Work Item
Identifier Order:555-1111111-0000000
Workflow OrderProcessing
Scheduled Time 2017-11-03 17:12:00 UTC
Current Step Email Customer
History [Start → Charge,
“Declined”,
Charge → Email Customer,
“InternalServerError”]
Key functionality
1. History store
• Store a log of all events that occurred
2. Scheduled queue
• Find the next work item that’s ready to run
3. Group, count, and list
• Show real-time aggregated views

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key functionality
1. History store
• Store a log of all events that occurred
2. Scheduled queue
• Find the next work item that’s ready to run
3. Group, count, and list
• Show real-time aggregated views
Step Count
Charge 54
Email Customer 80
Ship Order 22
Step Count
Charge 54
Email Customer 8,000
Ship Order 22
Step: Email Customer
Work Item Identifiers
Order:555-1111111-0000000
Order:555-1111111-0000001
Order:555-1111111-0000002
… (page 1 of many)
Step Count
Charge 54
Email Customer 8,000
Ship Order 22

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key functionality
1. History store
• Store a log of all events that occurred
2. Scheduled queue
• Find the next work item that’s ready to run
3. Group, count, and list
• Show real-time aggregated views

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Work Item Storage

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2009
Work Item Storage
DB 3
DB Service 1 DB Service 2 DB Service 3
DB 2DB 1
(Router)

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2011
Work Item Storage

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2013
Work Item Storage
…… … … ………

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2015
Work Item Storage
…… … … ………
…… … … ………
…… … … ………

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scaling effort
• Provision Oracle DB hardware
• Create DB credentials
• Create table schemas
• Tune Oracle DB settings
• Create new service
• Get new JDBC connection strings
• Update application configuration
• Provision load balancer
• Define new alarms
• Functional tests
• Performance tests
• Repartition data seamlessly
Ongoing maintenance effort
• OS patches
• Oracle DB software patches and upgrades
• 20 min downtime
• Index rebuild jobs
• Fine-tune their schedule
• Why is one DB slower than the others?
• Check hardware metrics
• Bad query plan?
• Configuration out of sync?
• Hardware failures
• Disks, I/O controllers
• Automated fail-over failures
• Lots of “tender love and care”
Where our Oracle architecture broke down

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
DynamoDB

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How we moved to DynamoDB

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key functionality
History store
Scheduled queue
Group, count, and list

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2015
Work Item Storage
…… … … ………
…… … … ………
…… … … ………

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
History store—on Oracle
Table: HISTORY
Identifier
Created Date
Serialized History
Table: WORK ITEM HEADER
Identifier
Workflow Name
Scheduled Time
Current Step
…
1 N

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
History store—on Oracle
Identifier Workflow Name Step …
… … … …
Identifier Workflow Name Step …
… … … …
xyz OrderProcessing Charge …
Identifier Created Date History
… … …
Identifier Created Date History
… … …
xyz T1 […, →Charge]
Transaction: OpenClose
Work Item Header Table Work Item History Table

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
History store—on DynamoDB
Identifier Workflow Name Step …
… … … …
Identifier Workflow Name Step …
… … … …
xyz OrderProcessing Charge …
Identifier Workflow Name Step …
… … … …
xyz OrderProcessing Ship …
Identifier Created Date History
… … …
Identifier Created Date History
… … …
xyz T1 […, →Charge]
Identifier Created Date History
… … …
xyz T1 […, →Charge]
xyz T2 […, →Ship]
No Transactions
Work Item Header Table Work Item History Table

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
History store—on DynamoDB
Table: WORK ITEMS
String Identifier
String Workflow Name
Date Scheduled Time
String Current Step
Binary History

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
$

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB cost model
• Provision your throughput
• DynamoDB charges per KB read/write

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
History accumulates
2 KB
Actual: 6 KB of writes Charged: 12 KB of writes
2 KB
2 KB
2 KB
4 KB
6 KB
Work Item Records

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Minimize writes
Keep the size of frequently written records really small
• ≤ 1 KB
Only keys and indexed attributes

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
History store—on DynamoDB
2 KB
2 KB
2 KB
Header
2 KB Cruft
Header Records History Records
6 KB written, not 12 KB
(excluding cruft)

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Technique #1
Separate (large) immutable payloads from (small) mutable data

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
History store—on DynamoDB
2 KB
2 KB
2 KB
Header
Header Records History Records
2 KB

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
History store: Many records
2 KB
2 KB
2 KB
Header
Header Records History Records
2 KB
2 KB
2 KB
2 KB
2 KB
2 KB
2 KB
2 KB
2 KB
…
…
…

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
10 KB10 MB
History store: Compaction
2 KB
2 KB
2 KB
2 KB
Header
Header Records History Records
2 KB
New Request
Max # History Records: 4
If > DynamoDB Limit:
Put in S3

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
History store: What about the cruft?
2 KB
2 KB
2 KB
Header
Header Records History Records
2 KB
To clean up cruft:
List & delete
10 MB

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Technique #2
Be a little inefficient, rarely…
(…and have a way to monitor that inefficiency!)

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB architecture
Work Item Storage

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key functionality
History store
Scheduled queue
Group, count, and list

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scheduled queue—challenges
• 100s of millions of active work items
• Biggest workflow has about 60 million active work items
• Need to start execution within couple of seconds of the scheduled time
• 10s of thousands of work items processed/second for a workflow

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB
DynamoDB
DynamoDB Partitions
DynamoDB Load Balancer
A
2Put
Partition Key: A
Sort Key: 2
Partition Key A
Put
Partition Key: A
Sort Key: 1
A
1
Put
Partition Key: A
Sort Key: 3
A
3
Put
Partition Key: B
Sort Key: 1
B
1
Put
Partition Key: C
Sort Key: 2
C
2

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB Global Secondary Index (GSI)
Table: WORK ITEM HEADER
Identifier
Workflow Name
Scheduled Time
Step
…
Partition Key
GSI: TIMER
Identifier
Workflow Name
Scheduled Time
…
Partition Key
Sort Key

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1 Partition Key
DynamoDB

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How can we make it scale?
OrderProcessing
OrderProcessing/N
OrderProcessing/1
OrderProcessing/2
OrderProcessing/3
Partition Key
.
.
.

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
5 Partition Keys
DynamoDB
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
~100 Partition Keys
DynamoDB
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Partition key space
1 N
Distribution

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Technique #3
Avoid hot DynamoDB partitions by manufacturing additional keys

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Partition key space
1 N
Distribution

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
~100K Partition Keys
DynamoDB
…

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Partition key space
1 N
Distribution
Discovery

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
~1000 Partition Keys
DynamoDB
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cache prefetching

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RAM

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
…..
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
Host
…

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
…..
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
Host 3Host 2 Host KHost 1 …..

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
…..
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
Host 1 …..Host 2 Host 3 Host K
OrderProcessing/1
OrderProcessing/2
OrderProcessing/3
OrderProcessing/4
OrderProcessing/5
OrderProcessing/6
OrderProcessing/7
OrderProcessing/8
OrderProcessing/9
OrderProcessing/[N-2]
OrderProcessing/[N-1]
OrderProcessing/[N]

57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
…..… …… …
Host 1 …..Host 2 Host 3 Host K
Scheduled Time = “Current time + X mins”

58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
…..… …… …
Host 1 …..Host 2 Host 3 Host K
Timer Router
poll(“OrderProcessing”)

59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
…..
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
Host 1 …..Host 2 Host 3 Host K
Timer Router Partition Assigner

60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Technique #4
Move hard problems into memory, where everything is easy
(Kind of)

61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GSI propagation delay

62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GSI propagation delay—issue
Work Item Storage
Timer Router
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Hosts
DynamoDB
Work Item Header
Timer GSI

63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Work Item Header
Timer GSI
GSI propagation delay—issue
Work Item Storage
Timer Router
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Hosts
Insert
Inserted: T1
Replicated: T2
Available: T3
DynamoDB
Time

64. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Work Item Header
Timer GSI
GSI propagation delay—solution
Work Item Storage
Timer Router
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Hosts
Insert
Inserted: T1
Available: T2
DynamoDB
Time

65. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Technique #5
Prime caches to minimize eventual consistency delays

66. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB architecture
Work Item Storage
Partition Assigner
Timer Router
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Hosts

67. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key functionality
History store
Scheduled queue
Group, count, and list

68. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Group, count, and list
Need to aggregate across 100s of millions to build the above view
Step Count
Charge 54
Email Customer 8,000
Ship Order 22

69. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB GSI—aggregate views
Table: WORK ITEM HEADER
Identifier
Workflow Name
Scheduled Time
Step
…
Partition Key
GSI: VIEW
Identifier
Workflow Name
Step
…
Partition Key

70. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Recap: Technique #3
Avoid hot DynamoDB partitions by manufacturing additional keys

71. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB GSI—aggregate views
Table: WORK ITEM HEADER
Identifier
Workflow Name/Partition
Scheduled Time
Step
…
Partition Key
GSI: VIEW
Identifier
Workflow Name/Partition
Step
…
Partition Key

72. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Recap: Technique #4
Move hard problems into memory

73. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
…..
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
.
.
.
.
.
.
.
.
.
…
Host 1 …..Host 2 Host 3 Host K
View Router
Aggregate(“OrderProcessing”)

74. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Recap: Technique #5
Prime caches to minimize eventual consistency delays

75. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB architecture
Work Item Storage
Partition Assigner
Timer Router
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Hosts
View Router
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Nodes
View
Hosts

76. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key functionality
History store
Scheduled queue
Group, count, and list

77. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Recap: Techniques you could use
1. Separate (large) immutable payloads from (small) mutable data
2. Be a little inefficient, rarely
3. Avoid hot DynamoDB partitions by manufacturing additional keys
4. Move hard problems into memory
5. Prime caches to minimize eventual consistency delays

78. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Oracle architecture
Work Item Storage
…… … … ………
…… … … ………
…… … … ………

79. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scaling effort
• Provision Oracle DB hardware
• Create DB credentials
• Create table schemas
• Tune Oracle DB settings
• Create new service
• Get new JDBC connection strings
• Update application configuration
• Provision load balancer
• Define new alarms
• Functional tests
• Performance tests
• Repartition data seamlessly
Ongoing maintenance effort
• OS patches
• Oracle DB software patches and upgrades
• 20 min downtime
• Index rebuild jobs
• Fine-tune their schedule
• Why is one DB slower than the others?
• Check hardware metrics
• Bad query plan?
• Configuration out of sync?
• Hardware failures
• Disks, I/O controllers
• Automated fail-over failures
• Lots of “tender love and care”
Where our Oracle architecture broke down

80. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB architecture
Work Item Storage
Partition Assigner
Timer Router
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Hosts
View Router
Timer
Nodes
Timer
Nodes
Timer
Nodes
Timer
Nodes
View
Hosts

81. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bonus results:
• Workflow delays reduced 10x:
1 s to 100 ms @ p50
• Techniques:
1) Querying in-memory
2) Priming caches
• Latency and availability improved
• DB failovers, index rebuilds
• We can scale down ($$$)
• No more durable state
• Architectural isolation
• Scale and fail independently
Key result:
• Maintenance effort reduced 10x:
60 developer weeks/year à 6
• Would have needed 1000+ Oracle hosts
for Prime Day 2017
• Soon: nothing but scaling
• Instead: innovate on behalf of
customers
DynamoDB architecture: Results

82. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
( A f t e r )( B e f o r e )
R e p lacin g 1 0 0 s o f O racle D Bs w it h Ju st 1 D y n am o D B
O r a c l e A r c h i t e c t u r e D y n a m o D B A r c h i t e c t u r e

83. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!