slide share on aws data pipe line

6. Amazon S3
Amazon
DynamoDB
Amazon
RDS
Amazon
Redshift
On
Premise
HDFS
(Amazon EMR)

7. Amazon DynamoDB Amazon S3

9. Amazon S3
Amazon
DynamoDB
Amazon
RDS
Amazon
Redshift
On
Premise
HDFS
(Amazon EMR)

10. Amazon S3
Amazon
DynamoDB
Amazon
RDS
Amazon
Redshift
On
Premise
HDFS
(Amazon EMR)

11. Amazon S3
Amazon
DynamoDB
Amazon
RDS
Amazon
Redshift
On
Premise
HDFS
(Amazon EMR)

12. Amazon S3
Amazon
DynamoDB
Amazon
RDS
Amazon
Redshift
On
Premise
HDFS
(Amazon EMR)

13. Amazon S3
Amazon
DynamoDB
Amazon
RDS
Amazon
Redshift
On
Premise
HDFS
(Amazon EMR)

16. Input Datanode
Activity
[Output Datanode]

17. Input Datanode with precondition check
Activity with failure & delay notifications
Ouput Datanode

20. Compute Resources
Data Data
Data Stores Data Stores

22. Start
Interval
[End]

23. Noon Today
1 hour

24. …..
12-1pm
1-2pm
2-3pm
X

25. …..
12-1pm
1-2pm
2-3pm
1 dayX
X

26. Hourly
Daily
Weekly
Monthly
Yearly
Quarterly

29. S3 logs (hourly) Geolocation data
Per-geography
usage computation
(hourly)
Redshift
results

30. S3 logs (hourly)
Precondition: files exist
Geolocation data
Precondition: ./geo_available
Per-geography
usage computation
(hourly)
Redshift
results

32. Dynamo
event data
RDS
demographics
Hive-based
analysis (hourly)
Redshift
results

34. Hourly click updates Hourly event analysis
Daily reporting SQL

36. Amazon S3
logs
Custom
Precondition
EMR usage-by-geo job
Amazon EC2
report generation
Amazon
DynamoDB
event data
Amazon RDS
demographics
Amazon Redshift
DW table
Amazon
Redshift
DW table
Hive
script

37. Amazon S3
logs
Custom
Precondition
EMR usage-by-geo job
Amazon EC2
report generation
Amazon
DynamoDB
event data
Amazon RDS
demographics
Amazon Redshift
DW table
Amazon
Redshift
DW table
Hive
script

39. We Manage You Manage
EC2
Instances
EMR Clusters
On Premise Resources
EC2
Instances
EMR Clusters

45. {
"objects" : [
{
"name" : “My Copy”,
"type" : “Copy Action”,
“input”: {“ref” : “My RDS Data”},
“output”: {“ref” : “My S3 Data”},
”runsOn” : {“ref”: “My Instance”},
"schedule" : { "ref" : “My Schedule" } },
{
"name" : ”My Instance”,
"type" : ”EC2Instance”,
"instanceType" : "m1.small”,
"schedule" : { "ref” : “My Schedule" } },
…..
}

47. On AWS On Premise
High
Frequency
$1/month $2.50/month
Low Frequency $.60/month $1.50/month

50. We are sincerely eager to
hear your feedback on this
presentation and on re:Invent.
Please fill out an evaluation
form when you have a
chance.