Hadoop and DynamoDB

Hadoop and DynamoDBDr. Matt WoodT E C H N O LO G Y E VA N G E L I S T

Hello.

Thank you.

Consumer Seller business business

Decades of experience Operations, management and scale

Programmatic access

Unexpected innovation

Blinding ﬂash of the obvious

5 years young

Compute Storage ServicesDatabases & Support

Analytics

AvailableLow cost Flexible

Data is valuable

Data is plentiful

Data is complex

Data is in ﬂux

Data is fast moving

Capturing and managing data is challenging

Scalable storage can help

Database is a bottleneck

Degraded performance

A very common problem

DynamoDB

NoSQL database service

Unlimited storageConsistent performance

Provisioned throughput

Scale up and down

Scale without downtime

Read unit

Write unit

Provision read and write performance

Highly available

Consistent reads

Eventual consistency

Preset alarms

NoSQL data model

“ImageID” = “1” “Date” = “20100915”“Title” = “ﬂower”“Tags” = “ﬂower”,“jasmine”, “white”

“ImageID” = “1” “ImageID” =”2” “ImageID” =”3” “Date” = “Date” = “Date” = “20100915” “20100916” “20100917”“Title” = “ﬂower” “Title” = “car” “Title” = “coffee”“Tags” = “ﬂower”, “Tags” = “car”, “Tags” = “coffee”,“jasmine”, “white” “italian” “drink”, “delicious”

“ImageID” = “1” Primary key “Date” = “20100915”“Title” = “ﬂower”“Tags” = “ﬂower”,“jasmine”, “white”

“ImageID” = “1” Composite keys “Date” = “20100915”“Title” = “ﬂower”“Tags” = “ﬂower”,“jasmine”, “white”

Range queries

Compute

Elastic Map Reduce

ManagedHadoop

Without the ‘muck’

S3Input data

S3 Input dataCode Elastic MapReduce

S3 Input dataCode Elastic Name MapReduce node

S3 Input dataCode Elastic Name MapReduce node Elastic cluster

S3 Input dataCode Elastic Name MapReduce node HDFS Elastic cluster

S3 Input dataCode Elastic Name MapReduce node Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster

S3 Input dataCode Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster

S3Input data Output S3 + SimpleDB

It’s all just Hadoop

Hive, Pig,Cascading,Streaming

API driven

Data movement

Import/Export

Multipart upload

Scale control

Resize running job ﬂows

14 hoursTime remaining: 14 hours

14 hoursTime remaining: 7 hours

Time remaining: 3 hours

Balance cost and performance

Resize based on usage patterns

Steady state Steady state Batch processing

Integrated with DynamoDB

Analytics

Integrate

Backup and restore

HiveQL

Live data in DynamoDBCREATE EXTERNAL TABLE orders_ddb_2012_01 ( order_idstring, customer_id string, order_date bigint, totaldouble )STORED BYorg.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler TBLPROPERTIES ("dynamodb.table.name" = "Orders-2012-01","dynamodb.column.mapping" = "order_id:OrderID,customer_id:Customer ID,order_date:OrderDate,total:Total");

Query DynamoDBSELECT customer_id, sum(total) spend, count(*)order_countFROM orders_ddb_2012_01WHERE order_date >= unix_timestamp(2012-01-01, yyyy-MM-dd)AND order_date < unix_timestamp(2012-01-08, yyyy-MM-dd)GROUP BY customer_idORDER BY spend descLIMIT 5 ;

Archived data in S3CREATE EXTERNAL TABLE orders_s3_export ( order_idstring, customer_id string, order_date int, totaldouble )PARTITIONED BY (year string, month string)ROW FORMAT DELIMITEDFIELDS TERMINATED BY tLOCATION s3://elastic-mapreduce/samples/ddb-orders ;

Query S3SELECT year, month, customer_id, sum(total) spend,count(*) order_countFROM orders_s3_exportWHERE customer_id = c-2cC5fF1bBAND month >= 6AND year = 2011GROUP BY customer_id, year, monthORDER by month desc;

Export to S3CREATE EXTERNAL TABLE orders_s3_new_export ( order_idstring, customer_id string, order_date int, totaldouble )PARTITIONED BY (year string, month string)ROW FORMAT DELIMITED FIELDS TERMINATED BY ,LOCATION s3://;INSERT OVERWRITE TABLEorders_s3_new_exportPARTITION (year=2012, month=01)SELECT * from orders_ddb_2012_01;

Perfect match

Thank you!

Q U E S T I O N S + C O M M E N T Smatthew@amazon.com @mza O N T W I T T E R

http://softwarestrategiesblog.com	2967
http://lanyrd.com	160
http://a0.twimg.com	4
http://paper.li	3
http://translate.googleusercontent.com	2
http://us-w1.rockmelt.com	1
http://lcolumbus.wordpress.com	1

Hadoop and DynamoDB

by Amazon Web Services

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 3,138

Statistics