Building a Data Processing Pipeline on AWS - AWS Summit SG 2017
•
1 like•501 views
AWS provides a broad platform of managed services to help you build, secure, and seamlessly scale end-to-end Big Data applications quickly and with ease. Want to get ramped up on how to use Amazon's big data web services? Learn when to use which service? Want to write your first big data application on AWS? Join us in this session as we discuss reference architecture, design patterns, and best practices for pulling together various AWS services to meet your big data challenges.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building a Data Processing Pipeline on AWS - AWS Summit SG 2017
Similar to Building a Data Processing Pipeline on AWS - AWS Summit SG 2017 (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (20)
Building a Data Processing Pipeline on AWS - AWS Summit SG 2017
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Building a Data Processing Pipeline on AWS Eng-Hwa Tan Solutions Architect, Amazon Web Services, ASEAN
- 2. • Big Data Challenges • Architectural Principles • Stages in a Data Processing Pipeline • Build a data processing pipeline • Design Patterns Agenda
- 3. Ever Increasing Big Data Volume Velocity Variety Veracity Value
- 4. Plethora of Tools Amazon Glacier S3 DynamoDB RDS EMR Amazon Redshift Data Pipeline Amazon Kinesis Lambda Amazon ML SQS ElastiCache DynamoDB Streams Amazon Elasticsearch Service Amazon Kinesis Analytics Amazon Athena
- 5. Big Data Challenges Why? How? What tools should I use? Is there a reference architecture?
- 6. Architectural Principles Build decoupled systems • Data → Store → Process → Store → Analyze → Answers Use the right tool for the job • Data structure, latency, throughput, access patterns Leverage AWS managed services • Scalable/elastic, available, reliable, secure, no/low admin Use log-centric design patterns • Immutable logs, materialized views Be cost-conscious • Big data ≠ big cost
- 7. COLLECT STORE PROCESS/ ANALYZE CONSUME Time to answer (Latency) Throughput Simplify Big Data Processing
- 8. COLLECT STORE PROCESS/ ANALYZE CONSUME Simplify Big Data Processing Transactions Connected devices Web logs / cookies Social media ERP Answers & Insights Data analysts Data scientists Business users Engagements Automation / events Data
- 9. PROCESS STORE ANALYZE & VISUALIZE COLLECT Building a pipeline - DEMO
- 10. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS LoggingIoTApplicationsTransportMessaging Streaming App
- 11. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS LoggingIoTApplicationsTransportMessaging Streaming App Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache
- 12. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message Devices Sensors & IoT platforms AWS IoT LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache MESSAGES STREAMS
- 13. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache STREAMS
- 14. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache
- 15. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache
- 16. Weblogs – Common Log Format (CLF) 75.35.230.210 - - [20/Jul/2016:22:22:42 -0700] "GET /images/pigtrihawk.jpg HTTP/1.1" 200 29236 "http://www.swivel.com/graphs/show/1163466" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)"
- 17. Amazon Kinesis Streams • For Technical Developers • Build your own custom applications that process or analyze streaming data Amazon Kinesis Firehose • For all developers, data scientists • Easily load massive volumes of streaming data into S3, Amazon Redshift and Amazon Elasticsearch Amazon Kinesis Analytics • For all developers, data scientists • Easily analyze data streams using standard SQL queries Amazon Kinesis: Streaming Data Made Easy Services make it easy to capture, deliver and process streams on AWS
- 18. Why Is Amazon S3 Good for Big Data? • Unlimited number of objects and volume of data • Very high bandwidth – no aggregate throughput limit • Natively supported by big data frameworks (Spark, Hive, Presto, etc.) • No need to run compute clusters for storage (unlike HDFS) • Multiple & heterogeneous analysis clusters can use the same data • Designed for 99.99% availability – can tolerate zone failure • Designed for 99.999999999% durability • No need to pay for data replication • Native support for versioning • Tiered-storage (Standard, IA, Amazon Glacier) via life-cycle policies • Secure – SSL, client/server-side encryption at rest • Low cost
- 19. PROCESS STORE ANALYZE & VISUALIZE COLLECT: Amazon Kinesis Firehose Building a pipeline - DEMO
- 20. • Create Amazon S3 Bucket • Create Amazon Kinesis Firehose delivery stream • Publish logs to Amazon Kinesis Firehose Demo
- 21. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache PROCESS / ANALYZE
- 22. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache PROCESS / ANALYZE Streaming Amazon Kinesis Analytics KCL apps AWS Lambda Stream Amazon EC2 Amazon EMR Amazon EC2
- 23. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache PROCESS / ANALYZE Streaming Amazon Kinesis Analytics KCL apps AWS Lambda Stream Amazon EC2 Amazon EMR Amazon EC2 Amazon SQS apps Message
- 24. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache PROCESS / ANALYZE Streaming Amazon Kinesis Analytics KCL apps AWS Lambda Stream Amazon EC2 Amazon EMR Amazon EC2 Amazon SQS apps Message Amazon EMR BatchInteractive Presto Amazon Athena
- 25. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache PROCESS / ANALYZE Streaming Amazon Kinesis Analytics KCL apps AWS Lambda Stream Amazon EC2 Amazon EMR Amazon EC2 Amazon SQS apps Message Amazon Redshift Amazon EMR BatchInteractive Presto Amazon Athena
- 26. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache PROCESS / ANALYZE Streaming Amazon Kinesis Analytics KCL apps AWS Lambda Stream Amazon EC2 Amazon EMR Amazon EC2 Amazon SQS apps Message Amazon Machine Learning ML Amazon Redshift Amazon EMR BatchInteractive Presto Amazon Athena
- 27. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache PROCESS / ANALYZE Streaming Amazon Kinesis Analytics KCL apps AWS Lambda Stream Amazon EC2 Amazon EMR Amazon EC2 Amazon SQS apps Message Amazon Machine Learning ML Amazon Redshift Amazon EMR BatchInteractive Presto Amazon Athena
- 28. PROCESS: Amazon EMR with Spark & Hive STORE ANALYZE & VISUALIZE COLLECT: Amazon Kinesis Firehose Building a pipeline - DEMO
- 29. • Check the files which were ingested into Amazon S3 • Clean the data using Amazon EMR (Spark) • Create a table in Amazon Athena • Query data using SQL Demo
- 30. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache PROCESS / ANALYZE Streaming Amazon Kinesis Analytics KCL apps AWS Lambda Stream Amazon EC2 Amazon EMR Amazon EC2 Amazon SQS apps Message Amazon Machine Learning ML ETL Amazon Redshift Amazon EMR BatchInteractive Presto Amazon Athena CONSUME
- 31. COLLECT STORE Mobile apps Web apps Data centers AWS Direct Connect RECORDS AWS Import/Export Snowball Logging Amazon CloudWatch AWS CloudTrail DOCUMENTS FILES Messaging Message MESSAGES Devices Sensors & IoT platforms AWS IoT STREAMS Apache Kafka Amazon Kinesis Streams Amazon Kinesis Firehose Amazon DynamoDB Streams Stream Amazon SQS Message LoggingIoTApplicationsTransportMessaging Streaming App Amazon S3 FileSearch Amazon Elasticsearch Service Amazon DynamoDB Amazon ElastiCache Amazon RDS SQLNoSQLCache PROCESS / ANALYZE Streaming Amazon Kinesis Analytics KCL apps AWS Lambda Stream Amazon EC2 Amazon EMR Amazon EC2 Amazon SQS apps Message Amazon Machine Learning ML ETL Amazon Redshift Amazon EMR BatchInteractive Presto Amazon Athena CONSUME Amazon QuickSight Apps & Services Analysis&visualizationNotebooksIDEAPI
- 32. Building a pipeline PROCESS: Amazon EMR with Spark & Hive STORE ANALYZE & VISUALIZE: Amazon Redshift and Amazon QuickSight COLLECT: Amazon Kinesis Firehose
- 33. • Amazon QuickSight Demo Demo
- 34. Design Patterns
- 35. Primitive: Decoupled Data Bus Storage decoupled from processing Multiple stages Store Process Store Process process store
- 36. Primitive: Pub/Sub Amazon Kinesis AWS Lambda Amazon Kinesis Connector Library process store Apache Spark Parallel stream consumption/processing
- 37. Amazon EMR Real-time Analytics Amazon Kinesis KCL app AWS Lambda Spark Streaming Amazon SNS Amazon ML Notifications Amazon ElastiCache (Redis) Amazon DynamoDB Amazon RDS Amazon ES Alerts App state Real-time prediction KPI process store Amazon Kinesis Analytics Amazon S3 Log Amazon KinesisFan out
- 38. Interactive & Batch Analytics Amazon S3 Amazon EMR Hive Pig Spark Amazon ML process store Consume Amazon Redshift Amazon EMR Presto Spark Batch Interactive Batch prediction Real-time prediction Amazon Kinesis Firehose Amazon Athena Files Amazon Kinesis Analytics
- 39. Interactive & Batch Amazon S3 Amazon Redshift Amazon EMR Presto Hive Pig Spark Amazon ElastiCache Amazon DynamoDB Amazon RDS Amazon ES AWS Lambda Storm Spark Streaming on Amazon EMR Applications Amazon Kinesis App state KCL Amazon ML Real-time Amazon DynamoDB Amazon RDS Change Data Capture Transactions Stream Files Data Lake Amazon Kinesis Analytics Amazon Athena Amazon Kinesis Firehose
- 40. http://bit.ly/DataLakeOnAWS Data Lake Solution Architecture on AWS