In this session, we'll review the features and architecture of the new AWS Data Pipeline service and explain how you can use it to better manage your data-driven workloads. We'll then go over a few examples of setting up and provisioning a pipeline in the system.

6. Amazon S3 Amazon
RDS
Amazon Amazon
DynamoDB Redshift
HDFS
On
(Amazon EMR)
Premise

7. Amazon DynamoDB Amazon S3

9. Amazon S3 Amazon
RDS
Amazon Amazon
DynamoDB Redshift
HDFS
On
(Amazon EMR)
Premise

10. Amazon S3 Amazon
RDS
Amazon Amazon
DynamoDB Redshift
HDFS
On
(Amazon EMR)
Premise

11. Amazon S3 Amazon
RDS
Amazon Amazon
DynamoDB Redshift
HDFS
On
(Amazon EMR)
Premise

12. Amazon S3 Amazon
RDS
Amazon Amazon
DynamoDB Redshift
HDFS
On
(Amazon EMR)
Premise

13. Amazon S3 Amazon
RDS
Amazon Amazon
DynamoDB Redshift
HDFS
On
(Amazon EMR)
Premise

16. Input Datanode
Activity
[Output Datanode]

17. Input Datanode with precondition check
Activity with failure & delay notifications
Ouput Datanode

20. Data Data
Data Stores Data Stores
Compute Resources

22. Start
Interval
[End]

23. Noon Today
1 hour

24. 12-1pm X
1-2pm
2-3pm
…..

25. 12-1pm X
1-2pm
2-3pm
X 1 day
…..

26. Monthly
Daily
Hourly
Quarterly
Yearly
Weekly

29. S3 logs (hourly) Geolocation data
Per-geography
usage computation
(hourly)
Redshift
results

30. S3 logs (hourly) Geolocation data
Precondition: files exist Precondition: ./geo_available
Per-geography
usage computation
(hourly)
Redshift
results

32. Dynamo RDS
event data demographics
Hive-based
analysis (hourly)
Redshift
results

34. Hourly click updates Hourly event analysis
Daily reporting SQL

36. Custom Amazon RDS
Amazon S3 Amazon demographics
logs Precondition DynamoDB
event data
Hive
script
EMR usage-by-geo job
Amazon
Redshift
DW table
Amazon Redshift Amazon EC2
DW table report generation

37. Custom Amazon RDS
Amazon S3 Amazon demographics
logs Precondition DynamoDB
event data
Hive
script
EMR usage-by-geo job
Amazon
Redshift
DW table
Amazon Redshift Amazon EC2
DW table report generation

39. We Manage You Manage
EMR Clusters EC2
EC2
Instances
Instances
EMR Clusters
On Premise Resources

45. {
"objects" : [
{
"name" : “My Copy”,
"type" : “Copy Action”,
“input”: {“ref” : “My RDS Data”},
“output”: {“ref” : “My S3 Data”},
”runsOn” : {“ref”: “My Instance”},
"schedule" : { "ref" : “My Schedule" } },
{
"name" : ”My Instance”,
"type" : ”EC2Instance”,
"instanceType" : "m1.small”,
"schedule" : { "ref” : “My Schedule" } },
…..
}

47. On AWS On Premise
High $1/month $2.50/month
Frequency
Low Frequency $.60/month $1.50/month

50. We are sincerely eager to
hear your feedback on this
presentation and on re:Invent.
Please fill out an evaluation
form when you have a
chance.