1. November 12, 2014 | Las Vegas, NV
David Yanacek, Amazon DynamoDB
Jason Lambert, Here by Nokia

4. App/Web Tier
Client TierRDBMS

5. App/Web Tier
Client TierRDBMS

6. App/Web Tier
Client TierData Tier
Search
Cache
Blob Store
RDBMS
NoSQL
Data Warehouse

7. Data Tier
Search
Cache
Blob Store
RDBMS
NoSQL
Data Warehouse
logging
analytics
key/value
simple query
rich search
hot reads
complex queries
and transactions

8. Data Tier
Amazon
DynamoDB
Amazon
RDS
Amazon
ElastiCache
Amazon
S3
Amazon
Redshift
Amazon
CloudSearch
logging
analytics
key/value
simple query
rich search hot reads
complex queries
and transactions

10. “DynamoDB has scaled effortlessly to match our company's explosive growth, doesn't burden our operations staff, and integrates beautifully with our other AWS assets”.
“I love how DynamoDB enables us to provision our desired throughput, and achieve low latency and seamless scale, even with our constantly growing workloads.”

12. RDBMS
Poor Availability Limited Scalability High Cost

13. lack of strong
operational consistency
every engineer needs to
learn distributed systems
complexity

14. Non-Relational
Fast & Predictable Performance
Seamless Scalability
Easy Administration

15. database service
automated operationspredictable performance
fast development
always durable
low latencycost effective
=

16. table

19. WRITES
Continuously replicated to 3 zones
Quorum acknowledgment
Persisted to disk (custom SSD)
READS
Strongly or eventually consistent
No trade-off in latency

20. “Our previous NoSQL database required almost a full time administrator to run.
Now AWS takes care of it.”
“Being optimized at AdRollmeans we spend more every month on snacks than we do on DynamoDB–and almost nothing on an ops team”
Save MoneyReduce Effort

21. Weatherbugmobile app
Lightning detection & alerting for 40M users/month
Developed and tested in weeks, at “1/20th of the cost of the traditional DB approach”
Super Bowl promotion
Millions of interactions over a relatively short period of time
Built the app in 3 days, from design to production-ready

22. database service
automated operationspredictable performance
fast development
always durable
low latencycost effective
=

24. https://www.example.com/maps/Pike+Place+Market,+Seattle,+ WA/47.6206924,-122.3492979/@47.6151852,- 122.3516502,16z/data=!3m1!4b1!4m8!4m7!1m5!1m1!1s0x54906ab2c151d6cb:0x60441d5f633e9f1c!2m2!1d- 122.3420567!2d47.6101359!1m0

25. https://__________________________Place+Market,+Seattle,+ WA/47.6206924,-122.3492979/@47.6151852,- 122.3516502,16z/data=!3m1!4b1!4m8!4m7!1m5!1m1!1s0x54906ab2c151d6cb:0x60441d5f633e9f1c!2m2!1d- 122.3420567!2d47.6101359!1m0
https://short.example.com/c3b87de1

26. client
Amazon DynamoDB
Web Service
Elastic Load Balancing
URL Shortener App
Target Website
example.com
website

27. client
example.com
website
Web Service
Elastic Load Balancing
GET /u/d83
HTTP 1.1 301 Moved Permanently
Location: http://www.example.com/maps/...
Amazon DynamoDB

28. client
example.com
website
Web Service
Elastic Load Balancing
GET /maps/Pike+Place...
Amazon DynamoDB

29. ShortUrl
RedirectTo
CreatedBy
CreateDate
example.com/u/d83
<long url>
David
2014-09-09
example.com/u/abc
<long url>
Alice
2013-08-17
example.com/u/123
<long url>
2013-12-21
example.com/u/ea2
<long url>
Bob
2014-03-30
Url Table
Table

30. ShortUrl
RedirectTo
CreatedBy
CreateDate
example.com/u/d83
<long url>
David
2014-09-09
example.com/u/abc
<long url>
Alice
2013-08-17
example.com/u/123
<long url>
2013-12-21
example.com/u/ea2
<long url>
Bob
2014-03-30
Url Table
Item

31. ShortUrl
RedirectTo
CreatedBy
CreateDate
example.com/u/d83
<long url>
David
2014-09-09
example.com/u/abc
<long url>
Alice
2013-08-17
example.com/u/123
<long url>
2013-12-21
example.com/u/ea2
<long url>
Bob
2014-03-30
Url Table
Attributes (schema-less)

32. ShortUrl
RedirectTo
CreatedBy
CreateDate
example.com/u/d83
<long url>
David
2014-09-09
example.com/u/abc
<long url>
Alice
2013-08-17
example.com/u/123
<long url>
2013-12-21
example.com/u/ea2
<long url>
Bob
2014-03-30
Url Table
Primary Key (Hash Index)

33. Item (JSON Document)

34. String Set
String
Number
Map
Not pictured:
•Binary
•List
•Null
•Number Set
•Binary Set
Boolean

35. 64 KB
400 KB

37. dynamodb.putItem({
TableName:'Url',
Item:{
ShortUrl:"short.example.com/4ua93",
RedirectTo:"long.example.com/maps/Pike+Place...",
IsLive:true,
Tags:["map","directions"],
Metadata:{
CreatedBy:"David",
CreateDate:"2014-19-21T08:49",
TargetViews:100
},
Stats:{
LastVisited:"2014-10-30T10:31",
PageViews:326
}
}
},function(err,data){
if(err)print(err);// an error occurred
elseprint(data);// successful response
});
Request
Response (empty)

38. dynamodb.getItem({
TableName:'Url',
Key:{
"ShortUrl":"short.example.com/4ua93"
},
ConsistentRead:true
},function(err,data){
if(err)print(err);
elseprint(data);
});
Optional read-your-writes consistency

39. dynamodb.scan({
TableName:'Url',
ProjectionExpression:'ShortUrl, '
+'RedirectTo, Stats.LastVisited',
Limit:2
}).eachPage(function(err,data){
if(err)print(err);
elseif(data)print(data);
});
SQL-like expressions
Paginated (max 1 MB per page)

40. dynamodb.scan({
TableName:'Url',
FilterExpression:
'begins_with(RedirectTo, :domain) '+
'AND contains(Tags, :tag)'+
'AND ('+
' Stats.PageViews> Metadata.TargetViews'+
' OR '+
' Stats.StarCount> Metadata.TargetStars'+
')',
ExpressionAttributeValues:{
':domain':'long.example.com/',
':tag':'directions'
}
}).eachPage(function(err,data){
if(err)print(err);
elseif(data)print(data);
});
Nested conditions
Compare existing attributes
Filter Expressions
and Improved Conditions!

44. Update when viewed

45. dynamodb.updateItem({
TableName:'Url',
Key:{
"ShortUrl":"short.example.com/4ua93"
},
UpdateExpression:
'SET Usage.Views= Usage.Views+ :increment_by, '
+'Usage.LastVisited= :now',
ExpressionAttributeValues:{
':increment_by':1,
':now':newDate().toISOString()
},
ReturnValues:'ALL_NEW',
ConditionExpression:'attribute_exists(ShortUrl)'
},function(err,data){
if(err)print(err);
elseprint(data);
});
Increment views by 1

46. dynamodb.updateItem({
TableName:'Url',
Key:{
"ShortUrl":"short.example.com/4ua93"
},
UpdateExpression:
'SET Usage.Views= Usage.Views+ :increment_by, '
+'Usage.LastVisited= :now',
ExpressionAttributeValues:{
':increment_by':1,
':now':newDate().toISOString()
},
ReturnValues:'ALL_NEW',
ConditionExpression:'attribute_exists(ShortUrl)'
},function(err,data){
if(err)print(err);
elseprint(data);
});
Writes are also free reads!

51. Images Table

52. Images Table
User
Image
Date
Link
Bob
aed4c
2013-10-01
s3://…
Bob
cf2e2
2013-09-05
s3://…
Bob
f93bae
2013-10-08
s3://…
Alice
ca61a
2013-09-12
s3://…
Hash and range primary key schema (composite key)

53. Images Table
User
Image
Date
Link
Bob
aed4c
2013-10-01
s3://…
Bob
cf2e2
2013-09-05
s3://…
Bob
f93bae
2013-10-08
s3://…
Alice
ca61a
2013-09-12
s3://…
Range: Index is sorted by range key, can do range queries
Hash: Equality queries only

54. Images Table
User
Image
Date
Link
Bob
aed4c
2013-10-01
s3://…
Bob
cf2e2
2013-09-05
s3://…
Bob
f93bae
2013-10-08
s3://…
Alice
ca61a
2013-09-12
s3://…
Bob
Query for Bob’s Images
Query
FROM Images
WHERE User=Bob

57. Images Table
User
Image
Date
Link
Bob
aed4c
2013-10-01
s3://…
Bob
cf2e2
2013-09-05
s3://…
Bob
f93bae
2013-10-08
s3://…
Alice
ca61a
2013-09-12
s3://…
User
Date
Image
Bob
2013-09-05
cf2e2
Bob
2013-10-01
aed4c
Bob
2013-10-08
f93bae
Alice
2013-09-12
ca61a
Table
ByDate Local Secondary Index
Local secondary index on Date

58. Images Table
User
Image
Date
Link
Bob
aed4c
2013-10-01
s3://…
Bob
cf2e2
2013-09-05
s3://…
Bob
f93bae
2013-10-08
s3://…
Alice
ca61a
2013-09-12
s3://…
User
Date
Image
Bob
2013-09-05
cf2e2
Bob
2013-10-01
aed4c
Bob
2013-10-08
f93bae
Alice
2013-09-12
ca61a
Table
ByDate Local Secondary Index
Query for Bob’s
two most recent images
Query
FROM Image-Index
WHERE User=Bob, DESC, LIMIT 2

60. Images
Table
ImageTags
Table

61. ImageTags Table
Image
User
aed4c
Alice
aed4c
Bob
f93bae
Bob

62. ImageTags Table
Image
User
aed4c
Alice
aed4c
Bob
f93bae
Bob
Hash and range primary key schema

63. Image
User
aed4c
Alice
aed4c
Bob
f93bae
Bob
ImageTags Table
Bob
Query
FROM ImageTags
WHERE Image=aed4c

64. ImageTags Table
Image
User
aed4c
Alice
aed4c
Bob
f93bae
Alice
f93bae
Bob
Bob
Tag Alice in Image f93bae

67. and/or range keyfor your table
•Even moreflexible query patterns
Distributed transaction managed for you by DynamoDB

68. ImageTags Table
Image
User
aed4c
Alice
aed4c
Bob
f93bae
Alice
f93bae
Bob
Table

69. ImageTags Table
Global secondary index on User, Image
User
Image
Bob
aed4c
Bob
f93bae
Alice
aed4c
Alice
f93bae
ByUser Global Secondary Index
Image
User
aed4c
Alice
aed4c
Bob
f93bae
Alice
f93bae
Bob
Table
Alternate hash and range keys

70. ImageTags Table
Alice
Query for images tagged Alice
User
Image
Bob
aed4c
Bob
f93bae
Alice
aed4c
Alice
f93bae
ByUser Global Secondary Index
Image
User
aed4c
Alice
aed4c
Bob
f93bae
Alice
f93bae
Bob
Table
Query
FROM ImageTags-Index
WHERE User=Alice

74. {
Id: abecd,
Players : [ Alice, Bob ],
State : STARTED,
Turn : Bob,
Top-Right : O
}
Game Item

75. Amazon
DynamoDB
Alice
Bob

76. Amazon
DynamoDB
Alice
Bob
Update:
Top-Right : O
Turn : Bob

77. Amazon
DynamoDB
Alice
Bob
Update:
Top-Left : X
Turn : Alice

78. Alice
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)

79. Alice
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)

80. Alice
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)

81. Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O

82. Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Low-Right : X

83. Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X,
Mid: X,
Low-Right: X
Update:
Turn : Alice
Low-Right : X

85. Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null

86. Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null

87. Bob (1)
Amazon
DynamoDB
Bob (2)
Bob (3)
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null

88. Amazon
DynamoDB
Bob
Alice
Update:
Turn : Alice
Top-Right : X
Expect:
Turn : Bob
Top-Right : null
Update:
Turn : Bob
Top-Left : X
Expect:
Turn : Alice
Top-Left : null

89. {
Id : abecd,
Players : [ Alice, Bob ],
State : STARTED,
Turn : Bob,
Top-Right: O,
Version : 2
}

91. Bob (1)
Bob (2)
Time

92. Bob (1)
Bob (2)
Read
Version: 1
Turn: Bob
Version: 1
Turn: Bob
Time

93. Bob (1)
Bob (2)
Read
Read
Version: 1
Turn: Bob
Version: 1
Turn: Bob
Version: 1
Turn: Bob
Time

94. Bob (1)
Bob (2)
Time
Read
Read
Version: 1
Turn: Bob
Version: 1
Turn: Bob
Version: 1
Turn: Bob
Version: 2
Turn: Alice
Top-Right: X
Version: 2
Turn: Alice
Top-Right: X

95. Bob (1)
Bob (2)
Read
Read
Version: 1
Turn: Bob
Version: 1
Turn: Bob
Version: 1
Turn: Bob
Version: 2
Turn: Alice
Top-Right: X
Version: 2
Turn: Alice
Top-Right: X
Version: 2
Turn: Alice
Top-Right: X
Put
If V=1
Time

96. Bob (1)
Bob (2)
Read
Read
Version: 1
Turn: Bob
Version: 1
Turn: Bob
Version: 1
Turn: Bob
Version: 2
Turn: Alice
Top-Right: X
Version: 2
Turn: Alice
Top-Right: X
Version: 2
Turn: Alice
Top-Right: X
Put
If V=1
Put
If V=1
Time

97. @DynamoDBTable(tableName="TicTacToeGame")
publicclassGame {
privateString id;
// ... other attributes
privateLong version;
@DynamoDBHashKey
publicString getId(){...}
@DynamoDBVersionAttribute
publicLong getVersion(){...}
}

98. @DynamoDBTable(tableName="TicTacToeGame")
publicclassGame {
privateString id;
// ... other attributes
privateLong version;
@DynamoDBHashKey
publicString getId(){...}
@DynamoDBVersionAttribute
publicLong getVersion(){...}
}

99. // 1. Load Game
Game game=dynamoDBMapper.load(Game.class,"cde3f");
// 2. See if the move is valid
// 3. Update the Game object
// 4. Save the game
dynamoDBMapper.save(game);

103. Amazon Redshift
Amazon EMR
DynamoDB
Amazon RDS
Amazon CloudSearch
dynamoDBMapper.save(game);
transaction.commit();

107. Addresses to Geolocations

108. records, tracking and analytics

109. high performance run time data retrieval

110. –Preparing a new wave of data to hit the service, tripling our coverage and data count

112. •Chop the big data stream files into small files
•Store those billions of files on a big CDN file system

113. http://navteq-maps.ovi.com.edgesuite.net/144/0206/23/87/53/206238753_r10_f5_x0_y2.jpg
We split the world into chunks –lots of those chunks have no streets
We couldn’t implement our own tracking and analytics
$$$$$$$$$$$$

114. Future flexible API

115. Keeps an index of all large files
Stores large files
Origin Service

118. RDS

120. 406311934325785de138c33-fa48-4418- 8ffa-54cc05e72adabigFile8098.big
A server can look up a DynamoDB record by key (~10ms), find the s3 location and use the Amazon S3 API to read file chunks (~60ms)

124. ~50% 80%

126. http://bit.ly/awsevals