AWS provides a broad platform of managed services to help you build, secure, and seamlessly scale end-to-end Big Data applications quickly and with ease. Want to get ramped up on how to use Amazon's big data web services? Learn when to use which service? Want to write your first big data application on AWS? Join us in this session as we discuss reference architecture, design patterns, and best practices for pulling together various AWS services to meet your big data challenges.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Unni Pillai
Solutions Architect
unni_k_pillai
Building a Data Processing Pipeline
on AWS

2. • Big Data Challenges
• Architectural Principles
• Stages in a Data Processing Pipeline
• Demo : Build a data processing pipeline
• Design Patterns
Agenda

3. Ever Increasing Big Data
Volume
Velocity
Variety
Veracity
Value

4. Plethora of Tools
Amazon
Glacier
S3 DynamoDB
RDS
EMR
Amazon
Redshift
Data Pipeline
Amazon
Kinesis
Lambda Amazon ML
SQS
ElastiCache
DynamoDB
Streams
Amazon Elasticsearch
Service
AmazonKinesis
Analytics
Amazon Athena

5. Big Data Challenges
Why?
How?
What tools should I use?
Is there a reference architecture?

6. Architectural Principles
Build decoupled systems
• Data → Store → Process → Store → Analyze → Answers
Use the right tool for the job
• Data structure, latency, throughput, access patterns
Leverage AWS managed services
• Scalable/elastic, available, reliable, secure, no/low admin
Use log-centric design patterns
• Immutable logs, materialized views
Be cost-conscious
• Big data ≠ big cost

7. COLLECT STORE PROCESS/
ANALYZE
CONSUME
Time to answer (Latency)
Throughput
Simplify Big Data Processing

8. COLLECT STORE PROCESS/
ANALYZE
CONSUME
Simplify Big Data Processing
Transactions
Connected
devices
Web logs /
cookies
Social media
ERP
Answers &
Insights
Data analysts
Data scientists
Business users
Engagements
Automation / events
Data

9. PROCESS
STORE
ANALYZE & VISUALIZE
COLLECT
Building a pipeline - DEMO

10. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
LoggingIoTApplicationsTransportMessaging
Streaming App

11. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache

12. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message
Devices
Sensors &
IoT platforms
AWS IoT
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
MESSAGES
STREAMS

13. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
STREAMS

14. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache

15. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache

16. Weblogs – Common Log Format (CLF)
75.35.230.210 - - [20/Jul/2016:22:22:42 -0700]
"GET /images/pigtrihawk.jpg HTTP/1.1" 200 29236
"http://www.swivel.com/graphs/show/1163466"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)"

17. Amazon Kinesis
Streams
• For Technical Developers
• Build your own custom
applications that process
or analyze streaming
data
Amazon Kinesis
Firehose
• For all developers, data
scientists
• Easily load massive
volumes of streaming data
into S3, Amazon Redshift
and Amazon Elasticsearch
Amazon Kinesis
Analytics
• For all developers, data
scientists
• Easily analyze data
streams using standard
SQL queries
Amazon Kinesis: Streaming Data Made Easy
Services make it easy to capture, deliver and process streams on AWS

18. Why Is Amazon S3 Good for Big Data?
• Unlimited number of objects and volume of data
• Very high bandwidth – no aggregate throughput limit
• Natively supported by big data frameworks (Spark, Hive, Presto, etc.)
• No need to run compute clusters for storage (unlike HDFS)
• Multiple & heterogeneous analysis clusters can use the same data
• Designed for 99.99% availability – can tolerate zone failure
• Designed for 99.999999999% durability
• No need to pay for data replication
• Native support for versioning
• Tiered-storage (Standard, IA, Amazon Glacier) via life-cycle policies
• Secure – SSL, client/server-side encryption at rest
• Low cost

19. PROCESS
STORE
ANALYZE & VISUALIZE
COLLECT:
Amazon Kinesis Firehose
Building a pipeline - DEMO

20. • Create Amazon S3 Bucket
• Create Amazon Kinesis Firehose delivery stream
• Publish logs to Amazon Kinesis Firehose
Demo

21. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
PROCESS / ANALYZE

22. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
PROCESS / ANALYZE
Streaming
Amazon Kinesis
Analytics
KCL
apps
AWS Lambda
Stream
Amazon EC2
Amazon EMR
Amazon EC2

23. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
PROCESS / ANALYZE
Streaming
Amazon Kinesis
Analytics
KCL
apps
AWS Lambda
Stream
Amazon EC2
Amazon EMR
Amazon EC2
Amazon SQS apps
Message

24. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
PROCESS / ANALYZE
Streaming
Amazon Kinesis
Analytics
KCL
apps
AWS Lambda
Stream
Amazon EC2
Amazon EMR
Amazon EC2
Amazon SQS apps
Message
Amazon
EMR
BatchInteractive
Presto
Amazon Athena

25. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
PROCESS / ANALYZE
Streaming
Amazon Kinesis
Analytics
KCL
apps
AWS Lambda
Stream
Amazon EC2
Amazon EMR
Amazon EC2
Amazon SQS apps
Message
Amazon Redshift
Amazon
EMR
BatchInteractive
Presto
Amazon Athena

26. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
PROCESS / ANALYZE
Streaming
Amazon Kinesis
Analytics
KCL
apps
AWS Lambda
Stream
Amazon EC2
Amazon EMR
Amazon EC2
Amazon SQS apps
Message
Amazon
Machine Learning
ML
Amazon Redshift
Amazon
EMR
BatchInteractive
Presto
Amazon Athena

27. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
PROCESS / ANALYZE
Streaming
Amazon Kinesis
Analytics
KCL
apps
AWS Lambda
Stream
Amazon EC2
Amazon EMR
Amazon EC2
Amazon SQS apps
Message
Amazon
Machine Learning
ML
Amazon Redshift
Amazon
EMR
BatchInteractive
Presto
Amazon Athena

28. PROCESS:
Amazon EMR with Spark & Hive
STORE
ANALYZE & VISUALIZE
COLLECT:
Amazon Kinesis Firehose
Building a pipeline - DEMO

29. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
PROCESS / ANALYZE
Streaming
Amazon Kinesis
Analytics
KCL
apps
AWS Lambda
Stream
Amazon EC2
Amazon EMR
Amazon EC2
Amazon SQS apps
Message
Amazon
Machine Learning
ML
ETL
Amazon Redshift
Amazon
EMR
BatchInteractive
Presto
Amazon Athena
CONSUME

30. COLLECT STORE
Mobile apps
Web apps
Data centers
AWS Direct
Connect
RECORDS
AWS Import/Export
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
DOCUMENTS
FILES
Messaging
Message MESSAGES
Devices
Sensors &
IoT platforms
AWS IoT STREAMS
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon DynamoDB
Streams
Stream
Amazon SQS
Message
LoggingIoTApplicationsTransportMessaging
Streaming App
Amazon S3
FileSearch
Amazon Elasticsearch
Service
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
SQLNoSQLCache
PROCESS / ANALYZE
Streaming
Amazon Kinesis
Analytics
KCL
apps
AWS Lambda
Stream
Amazon EC2
Amazon EMR
Amazon EC2
Amazon SQS apps
Message
Amazon
Machine Learning
ML
ETL
Amazon Redshift
Amazon
EMR
BatchInteractive
Presto
Amazon Athena
CONSUME
Amazon QuickSight
Apps & Services
Analysis&visualizationNotebooksIDEAPI

31. Building a pipeline
PROCESS:
Amazon EMR with Spark & Hive
STORE
ANALYZE & VISUALIZE:
Amazon Redshift and Amazon QuickSight
COLLECT:
Amazon Kinesis Firehose

32. • Check the files which were ingested into Amazon S3
• Clean the data using Amazon EMR (Spark)
• Create a table in Amazon Athena
• Query data using SQL
Demo

33. • Amazon QuickSight Demo
Demo

34. Design Patterns

35. Primitive: Decoupled Data Bus
Storage decoupled from processing
Multiple stages
Store Process Store Process
process
store

36. Primitive: Pub/Sub
Amazon
Kinesis
AWS
Lambda
Amazon Kinesis
Connector
Library
process
store
Apache
Spark
Parallel stream consumption/processing

37. Amazon EMR
Real-time Analytics
Amazon
Kinesis
KCL app
AWS Lambda
Spark
Streaming
Amazon
SNS
Amazon
ML
Notifications
Amazon
ElastiCache
(Redis)
Amazon
DynamoDB
Amazon
RDS
Amazon
ES
Alerts
App state
Real-time prediction
KPI
process
store
Amazon Kinesis
Analytics
Amazon
S3
Log
Amazon
KinesisFan out

38. Interactive
&
Batch
Analytics
Amazon S3
Amazon EMR
Hive
Pig
Spark
Amazon
ML
process
store
Consume
Amazon Redshift
Amazon EMR
Presto
Spark
Batch
Interactive
Batch prediction
Real-time prediction
Amazon
Kinesis
Firehose
Amazon Athena
Files
Amazon Kinesis
Analytics

39. Interactive
&
Batch
Amazon S3
Amazon
Redshift
Amazon EMR
Presto
Hive
Pig
Spark
Amazon
ElastiCache
Amazon
DynamoDB
Amazon
RDS
Amazon
ES
AWS Lambda
Storm
Spark Streaming
on Amazon EMR
Applications
Amazon
Kinesis
App state
KCL
Amazon
ML
Real-time
Amazon
DynamoDB
Amazon
RDS
Change Data
Capture
Transactions
Stream
Files
Data Lake
Amazon Kinesis
Analytics
Amazon
Athena
Amazon
Kinesis
Firehose

40. http://bit.ly/DataLakeOnAWS
Data Lake Solution Architecture on AWS

41. Thank You
http://blogs.aws.amazon.com/bigdata/
unni_k_pillai