Ran Tessler discusses building big data applications on AWS. He outlines architectural principles like using a decoupled data bus to store, process and retrieve data and answers. He demonstrates using AWS services like Kinesis Firehose for data collection, EMR with Spark and Hive for processing, and Redshift and QuickSight for analysis and visualization. Tessler references the Lambda architecture and its batch, speed and serving layers that can be implemented using AWS services like S3, DynamoDB, ElastiCache and Lambda.

1. Ran Tessler, Manager, Solutions Architecture
Building Big Data Applications on AWS

2. A Modern Take on Alchemy
Turning Data into Actionable Insights

3. What to Expect from this Session
Big Data architectural principles
Reference Lambda Architecture
Live demo

4. Architectural Principles
• Decoupled “data bus”
Data → Store → Process → Answers
• Use the right tool for the job
Latency, throughput, access patterns
• Apply Lambda architecture ideas
Immutable (append-only) log, batch/speed/serving layer
• Leverage AWS managed services
No/low admin
• Be cost conscious
Big data ≠ big cost

5. Simplify Big Data Processing
Ingest /
collect
store process /
analyze
consume /
visualize
Time to Answer (data freshness)
Throughput

6. Demo
http://aws.amazon.com/big-data/use-cases/

7. AccessLog - Common Log Format (CLF)
75.35.230.210 - - [20/Jul/2009:22:22:42 -0700]
"GET /images/pigtrihawk.jpg HTTP/1.1" 200 29236

8. Did NASA’s STS-69 Mission Land …
… On the right homepage?

9. Your First Big Data Application on AWS
PROCESS
STORE
ANALYZE & VISUALIZE
COLLECT

10. Your First Big Data Application on AWS
PROCESS
STORE
COLLECT:
Amazon Kinesis Firehose
ANALYZE & VISUALIZE

11. Your First Big Data Application on AWS
STORE
COLLECT:
Amazon Kinesis Firehose
ANALYZE & VISUALIZE
PROCESS:
Amazon EMR with Spark & Hive

12. Your First Big Data Application on AWS
PROCESS:
Amazon EMR with Spark & Hive
STORE
ANALYZE & VISUALIZE:
Amazon Redshift and Amazon QuickSight
COLLECT:
Amazon Kinesis Firehose

13. Reference
Lambda Architecture

14. process
store
Apps
Batch Layer
Amazon
Kinesis S3
Connector
Amazon S3
Amazon
Redshift
Amazon EMR
Presto
Hive
Pig
Spark
Lambda
Architecture
Serving
Layer
Amazon
ElastiCache
Amazon
DynamoDB
Amazon
RDS
Amazon
ES
Amazon
Kinesis Speed Layer
KCL
AWS Lambda
Spark Streaming
Storm
Amazon
MLdata

15. Back to our demo…
PROCESS:
Amazon EMR with Spark & Hive
STORE
ANALYZE & VISUALIZE:
Amazon Redshift and Amazon QuickSight
COLLECT:
Amazon Kinesis Firehose

16. DIY
Download all steps: http://bit.ly/29fhcwu
http://aws.amazon.com/big-data/use-cases/

17. tesslerr@amazon.com