Hadoop and DynamoDB provide scalable data storage and analytics capabilities. Hadoop uses Elastic MapReduce to process large datasets stored in S3 and HDFS. It allows querying live data stored in DynamoDB or archived data stored in S3. Results can be exported between the two services. DynamoDB is a NoSQL database that provides unlimited storage, consistent performance, and ability to scale without downtime. It is integrated with Hadoop for analytics use cases.

1. Hadoop and
DynamoDB
Dr. Matt Wood
T E C H N O LO G Y E VA N G E L I S T

2. Hello.

3. Thank you.

4. Consumer Seller
business business

5. Decades of experience
Operations, management and scale

6. Programmatic access

7. Unexpected innovation

8. Blinding flash of the
obvious

10. 5 years young

11. Compute Storage
Services
Databases & Support

12. Analytics

13. Available
Low cost
Flexible

14. Data

15. Data is valuable

16. Data is plentiful

17. Data is complex

18. Data is in flux

19. Data is fast moving

20. Capturing and managing
data is challenging

21. Scalable storage
can help

22. Database is a
bottleneck

23. Degraded performance

24. A very common problem

25. DynamoDB

26. NoSQL database service

27. Unlimited storage
Consistent performance

28. Provisioned throughput

29. Scale up and down

30. Scale without downtime

31. Read unit

32. Write unit

33. Provision read and write
performance

34. Highly available

35. Consistent reads

36. Eventual consistency

37. Preset alarms

38. NoSQL data model

40. “ImageID” = “1”
“Date” =
“20100915”
“Title” = “flower”
“Tags” = “flower”,
“jasmine”, “white”

41. “ImageID” = “1” “ImageID” =”2” “ImageID” =”3”
“Date” = “Date” = “Date” =
“20100915” “20100916” “20100917”
“Title” = “flower” “Title” = “car” “Title” = “coffee”
“Tags” = “flower”, “Tags” = “car”, “Tags” = “coffee”,
“jasmine”, “white” “italian” “drink”, “delicious”

42. “ImageID” = “1” Primary key
“Date” =
“20100915”
“Title” = “flower”
“Tags” = “flower”,
“jasmine”, “white”

43. “ImageID” = “1” Composite keys
“Date” =
“20100915”
“Title” = “flower”
“Tags” = “flower”,
“jasmine”, “white”

44. Range queries

47. Compute

48. Elastic Map
Reduce

49. Managed
Hadoop

50. Without the
‘muck’

51. S3
Input data

52. S3
Input data
Code Elastic
MapReduce

53. S3
Input data
Code Elastic Name
MapReduce node

54. S3
Input data
Code Elastic Name
MapReduce node
Elastic
cluster

55. S3
Input data
Code Elastic Name
MapReduce node
HDFS
Elastic
cluster

56. S3
Input data
Code Elastic Name
MapReduce node
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

57. S3
Input data
Code Elastic Name Output
MapReduce node S3 + SimpleDB
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

58. S3
Input data
Output
S3 + SimpleDB

73. It’s all just
Hadoop

74. Hive, Pig,
Cascading,
Streaming

75. API driven

76. Data movement

77. Import/Export

78. Multipart upload

79. Scale control

80. Resize running
job flows

81. 14 hours
Time remaining: 14 hours

82. 14 hours
Time remaining: 7 hours

83. Time remaining: 3 hours

84. Balance cost and
performance

85. Resize based on
usage patterns

86. Steady state Steady state
Batch processing

87. Integrated with
DynamoDB

88. Analytics

89. Integrate

90. Backup and
restore

91. HiveQL

92. Live data in DynamoDB
CREATE EXTERNAL TABLE orders_ddb_2012_01 ( order_id
string, customer_id string, order_date bigint, total
double )
STORED BY
'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler
' TBLPROPERTIES (
"dynamodb.table.name" = "Orders-2012-01",
"dynamodb.column.mapping" = "order_id:Order
ID,customer_id:Customer ID,order_date:Order
Date,total:Total"
);

93. Query DynamoDB
SELECT customer_id, sum(total) spend, count(*)
order_count
FROM orders_ddb_2012_01
WHERE order_date >= unix_timestamp('2012-01-01', 'yyyy-
MM-dd')
AND order_date < unix_timestamp('2012-01-08', 'yyyy-MM-
dd')
GROUP BY customer_id
ORDER BY spend desc
LIMIT 5 ;

94. Archived data in S3
CREATE EXTERNAL TABLE orders_s3_export ( order_id
string, customer_id string, order_date int, total
double )
PARTITIONED BY (year string, month string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY 't'
LOCATION 's3://elastic-mapreduce/samples/ddb-orders' ;

95. Query S3
SELECT year, month, customer_id, sum(total) spend,
count(*) order_count
FROM orders_s3_export
WHERE customer_id = 'c-2cC5fF1bB'
AND month >= 6
AND year = 2011
GROUP BY customer_id, year, month
ORDER by month desc;

96. Export to S3
CREATE EXTERNAL TABLE orders_s3_new_export ( order_id
string, customer_id string, order_date int, total
double )
PARTITIONED BY (year string, month string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION 's3://';
INSERT OVERWRITE TABLE
orders_s3_new_export
PARTITION (year='2012', month='01')
SELECT * from orders_ddb_2012_01;

97. Perfect match

98. Thank you!

99. Q U E S T I O N S + C O M M E N T S
matthew@amazon.com
@mza
O N T W I T T E R