Uploaded byAWS Germany
Wild Rydes with Big Data/Kinesis focus: AWS Serverless Workshop
The document outlines a serverless data processing architecture utilizing AWS services for a fictional company, Wild Rydes. It covers various modules including real-time streaming, data aggregation, processing, and data lake implementation, highlighting tools like Amazon Kinesis and Amazon Athena. Additionally, it provides setup instructions, coding examples, and pricing details for AWS services used in the framework.
More Related Content
Similar to Wild Rydes with Big Data/Kinesis focus: AWS Serverless Workshop
![[판교에서 만나는 아마존웹서비스] 새롭게 출시된 AWS 제품 소개](https://cdn.slidesharecdn.com/ss_thumbnails/2013reinventnewservices-131203201509-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Wild Rydes with Big Data/Kinesis focus: AWS Serverless Workshop
- 1. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Markus Kaiser, Solutions Architect Daniel Geske, Solutions Architect Wild Rydes: Dawn of a new unicorn Serverless data processing on AWS
- 2. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark What to expect 1. Serverless architectures 2. Wild Rydes scenario 3. Modules and relevant AWS services • Setup • Real-time data streaming • Stream aggregation • Stream processing • Data lake • Extra credit (optional) 4. Summary and clean-up
- 3. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Serverless Build and run applications and services without thinking of servers Fully Managed Developer Productivity Continuous Scaling
- 4. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- 5. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Help Wild Rydes Disrupt Transportation! So how does this magic work?
- 6. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Wild Rydes is Backed by Leading Investors THE BARN ACCELERATOR TENDERLOIN CAPITAL PENGLAI COMMUNICATIONS AND POST NEW CENTURY TECHNOLOGY CORP LIMITED
- 7. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Your Task: Process and visualize data in real-time Welcome to Wild Rydes Inc., Employee #3!
- 8. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Serverless data processing on AWS http://github.com/aws -samples/aws- serverless-workshops
- 9. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Setup Configure your AWS Cloud9 IDE and setup pre-requisites like an AWS Account.
- 10. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Real-time Streaming Data Create a stream in Kinesis and write to and read from the stream to track Wild Rydes unicorns on a live map. In this module you’ll also create an Amazon Cognito identity pool to grant the live map access to your stream. https://dataprocessing.wildrydes.com/streaming-data.html
- 11. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Kinesis Data Streams • Easy administration & low cost • Secure, durable storage • Build real time applications with framework of choice
- 12. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Kinesis Data Streams Producers Consumers Kinesis Agent Apache Kafka AWS SDK LOG4J Flume Fluentd AWS Mobile SDK Kinesis Producer Library Get* APIs Kinesis Client Library + Connector Library Apache Storm Amazon EMR AWS Lambda Apache Spark Amazon Kinesis
- 13. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Stream Aggregation Build an Kinesis Data Analytics application to read from the stream and aggregate metrics like unicorn health and distance traveled each minute. https://dataprocessing.wildrydes.com/streaming-aggregation.html
- 14. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Kinesis Analytics • Connect to streaming source • Write SQL code to process streaming data • Continuously deliver SQL results • Fully managed, automatic elasticity
- 15. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Stream Processing Persist aggregate data from application to a backend database stored in DynamoDB and run queries against those data. https://dataprocessing.wildrydes.com/stream-processing.html
- 16. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Data Lake Use Kinesis Data Firehose to flush the raw sensor data to an S3 bucket for archival purposes. Using Athena, you’ll run SQL queries against the raw data for ad-hoc analysis. https://dataprocessing.wildrydes.com/data-lake.html
- 17. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Kinesis Data Firehose • Zero administration and seamless elasticity • Direct-to-data store integration • Serverless, continuous data transformations
- 18. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Athena No loading of data, analyze data directly from Amazon S3 Standard SQL • Support for complex data types (arrays, structs) • Support for partitioning of data by any key Query data in its raw format • Athena supports multiple data formats: Text, CSV, TSV, JSON, weblogs, AWS service logs • or convert to an optimized form like ORC or Parquet for the best performance and lowest cost
- 19. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Extra credit AWS SDK Amazon Kinesis Data Analytics AWS Lambda Amazon Athena
- 20. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Summary
- 21. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Serverless Data Processing Architecture
- 22. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Pricing AWS Service Price* Example Kinesis Firehose Data Volume: $0.033 per GB 1350 GB: $44.55 S3 Data Volume: $0.0245 per GB PUT Requests: $0.0054 per 1.000 27 GB: $0.66 45.000 PUT: $0.243 Athena Scanned Data: $5 per TB 5.4 TB: $27.00 AWS Service Price* Example (Monthly) Kinesis Streams Shard: $0.018 per hour PUT Units: $0.0175 per Mio 2 x Shard: $27.00 270M PUTs: $4.725 Kinesis Analystics KPU: $0.132 per hour 1 x KPU: $99 Lambda Requests: $0.20 per Mio Execution: $0,00001667 per GB/s (negligible) 1.000 Unicorns, 10s Status Msg, Msg Size 1KB eu-central-1 30 days retention, 1:10 compression, 200 queries
- 23. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Clean up 1. Amazon Athena 2. Amazon Kinesis Data Analytics 3. Amazon Kinesis Data Firehose 4. Amazon Kinesis Data Streams 5. Amazon S3 6. AWS Lambda 7. Amazon DynamoDB 8. AWS IAM 9. Amazon Cognito 10.Cloud9 IDE
- 24. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Go build Markus Kaiser, Solutions Architect @markuskaiser Daniel Geske, Solutions Architect @btx94