Uploaded byAdrian Hornsby
Serverless Streaming Data Processing using Amazon Kinesis Analytics
This document provides an overview of building a real-time streaming data visualization system with Amazon Kinesis Analytics. It begins with an introduction and outlines what will be covered, including streaming data, the Amazon Kinesis platform, Kinesis Analytics patterns and features, and a walkthrough of an end-to-end streaming data example. It then provides details on the Amazon Kinesis platform and services, demonstrates how to write SQL queries to analyze streaming data in Kinesis Analytics, and shows how to output results to destinations like Kinesis streams, S3, and DynamoDB. It concludes with notes on deploying Kinesis Analytics applications and testing.
More Related Content
Similar to Serverless Streaming Data Processing using Amazon Kinesis Analytics
Serverless Streaming Data Processing using Amazon Kinesis Analytics
- 1. Build a Real-TimeStreaming Data Visualization System with Amazon Kinesis Analytic Adrian Hornsby (@adhorn) Technical Evangelist with AWS San Francisco Loft - 2017
- 2. • Technical Evangelist,Developer Advocate, … Software Engineer • My @home is in Finland • Previously: • Solutions Architect @AWS • Lead Cloud Architect @Dreambroker • Director of Engineering, Software Engineer, DevOps, Manager, ... @Hdm • Researcher @Nokia Research Center • and a bunch of other stuff. • Love climbing and ginger shots.
- 3. What to Expectfrom the Session • Streaming data overview • Amazon Kinesis platform review • Amazon Kinesis Analytics Overview • Amazon Kinesis Analytics patterns • Streaming data end-to-end example and walk- through
- 4.
- 5. Amazon Kinesis PlatformOverview
- 6. Amazon Kinesis makesit easy to work with real-time streaming data Amazon Kinesis Streams • For Technical Developers • Collect and stream data for ordered, replayable, real-time processing Amazon Kinesis Firehose • For all developers, data scientists • Easily load massive volumes of streaming data into Amazon S3, Redshift, ElasticSearch Amazon Kinesis Analytics • For all developers, data scientists • Easily analyze data streams using standard SQL queries
- 7. Amazon Kinesis Streams •Reliably ingest and durably store streaming data at low cost • Build custom real-time applications to process streaming data
- 8. Amazon Kinesis Producers Consumers Shard1 Shard 2 Shard n Shard 3 … … Write: 1MB Read: 2MB ** A shard is a group of data records in a stream
- 9. Amazon Kinesis (underthe hood)
- 10. Amazon Kinesis Analytics •Interact with streaming data in real-time using SQL • Build fully managed and elastic stream processing applications that process data for real-time visualizations and alarms
- 11.
- 12. Kinesis Analytics Pay foronly what you use Automatic elasticity Standard SQL for analytics Real-time processing Easy to use
- 13. Use SQL tobuild real-time applications Easily write SQL code to process streaming data Connect to streaming source Continuously deliver SQL results
- 14. Connect to streamingsource • Streaming data sources include Kinesis Firehose or Kinesis Streams • Input formats include JSON, .csv, variable column, unstructured text • Each input has a schema; schema is inferred, but you can edit • Reference data sources (S3) for data enrichment
- 15. Write SQL code •Build streaming applications with one-to-many SQL statements • Robust SQL support and advanced analytic functions • Extensions to the SQL standard to work seamlessly with streaming data • Support for at-least-once processing semantics
- 16. Continuously deliver SQLresults • Send processed data to multiple destinations • S3, Amazon Redshift, Amazon ES (through Firehose) • Streams (with AWS Lambda integration for custom destinations) • End-to-end processing speed as low as sub- second • Separation of processing and data delivery
- 17. What are commonuses for Kinesis Analytics?
- 18. Generate time seriesanalytics • Compute key performance indicators over time periods • Combine with static or historical data in S3 or Amazon Redshift Analytics Streams Firehose Amazon Redshift S3 Streams Firehose Custom, real- time destinations
- 19. Create real-time alarmsand notifications • Build sequences of events from the stream, like user sessions in a clickstream or app behavior through logs • Identify events (or a series of events) of interest, and react to the data through alarms and notifications Analytics Streams Firehose Streams Amazon SNS Amazon CloudWatch Lambda
- 20. Feed real-time dashboards •Validate and transform raw data, and then process to calculate meaningful statistics • Send processed data downstream for visualization in BI and visualization services Amazon QuickSight Analytics Amazon ES Amazon Redshift Amazon RDS Streams Firehose
- 21.
- 22. Scenario Requirements Data tocapture every second: • Total distinct users • Number of users for each Operating System • Number of users in each quadrant Output Requirements • Update DynamoDB table every second, with each aggregate values
- 23.
- 25. Data Input Source JSONData • Once per second, using JavaScript SDK: • Unique Cognito ID (anonymous user) • OS • Quadrant • Data sent to Kinesis Stream Amazon Kinesis Stream Amazon Cognito Amazon S3 JavaScript SDK { "recordTime": 1486505943.204, "cognitoId": "us-east-1:3626e211-d2a3-447b-8231-e1f4e0486f44", "os": "Android", "quadrant": "A" }
- 26. How is rawdata mapped to a schema? Amazon Kinesis stream Amazon KinesisAnalytics cognitoID os quadrant <guid1> Android A <guid2> iOS B Source Data for Kinesis Analytics { "recordTime": 1486505943.204, "cognitoId": "us-east-1:<guid>", "os": "Android", "quadrant": "A" }
- 27. How is streamingdata accessed with SQL? STREAM • Analogous to a TABLE • Represents continuous data flow CREATE OR REPLACE STREAM DISTINCT_USER_STREAM( COGNITO_ID VARCHAR(64), DEVICE VARCHAR(32), OS VARCHAR(32), QUADRANT char(1), DT TIMESTAMP);
- 28. How is streamingdata accessed with SQL? PUMP • Continuous INSERT query • Inserts data from one in-application stream to another CREATE OR REPLACE PUMP "DISTINCT_USER_PUMP" AS INSERT INTO "DISTINCT_USER_STREAM" SELECT STREAM DISTINCT "cognitoId", ...
- 29. How do wemodel our data? DISTINCT_USERS_STREAM •COGNITO_ID •OS •QUADRANT •DT DESTINATION_SQL_STREAM •UNIQUE_USER_COUNT •ANDROID_COUNT •IOS_COUNT •OTHER_OS_COUNT •QUADRANT_A_COUNT •QUADRANT_B_COUNT •QUADRANT_C_COUNT •QUADRANT_C_COUNT SOURCE_STREAM •cognitoID •os •quadrant Kinesis stream Kinesis Stream Pump Kinesis Analytics Application
- 30. How do weget distinct user records? Use PUMP to insert distinct records into in-app STREAM CREATE OR REPLACE PUMP "DISTINCT_USER_PUMP" AS INSERT INTO "DISTINCT_USER_STREAM" SELECT STREAM DISTINCT "cognitoId", "device", "os", "quadrant", FLOOR(s.ROWTIME TO SECOND) FROM "SOURCE_SQL_STREAM_001" s; DISTINCT_USERS_STREAM •COGNITO_ID •OS •QUADRANT •DT SOURCE_STREAM •cognitoID •os •quadrant
- 31. How do weaggregate streaming data? • A common requirement in streaming analytics is to perform set-based operation(s) (count, average, max, min,..) over events that arrive within a specified period of time • Cannot simply aggregate over an entire table like typical static database • How do we define a subset in a potentially infinite stream? • Windowing functions!
- 32. Windowing Concepts • Windowscan be tumbling or sliding • Windows are fixed length Output record will have the timestamp of the end of the window 1 5 4 26 8 6 4 t1 t2 t5 t6t3 t4 Time Window1 Window2 Window3 Aggregate Function (Sum) 18 14 Output Events
- 33. Comparing Types ofWindows • Output created at the end of the window • The output of the window will be single event based on the aggregate function used Tumbling window Aggregate per time interval Sliding window Windows constantly re-evaluated
- 34. How do weaggregate per second? • Tumbling window, group by time period CREATE OR REPLACE PUMP "OUTPUT_PUMP" AS INSERT INTO "DESTINATION_SQL_STREAM" SELECT STREAM COUNT(dus.COGNITO_ID) AS UNIQUE_USER_COUNT, COUNT((CASE WHEN dus.OS = 'Android' THEN COGNITO_ID ELSE null END)) AS ANDROID_COUNT, COUNT((CASE WHEN dus.OS = 'iOS' THEN COGNITO_ID ELSE null END)) AS IOS_COUNT, COUNT((CASE WHEN dus.OS = 'Windows Phone' THEN COGNITO_ID ELSE null END)) AS WINDOWS_PHONE_COUNT, COUNT((CASE WHEN dus.OS = 'other' THEN COGNITO_ID ELSE null END)) AS OTHER_OS_COUNT, COUNT((CASE WHEN dus.QUADRANT = 'A' THEN COGNITO_ID ELSE null END)) AS QUADRANT_A_COUNT, COUNT((CASE WHEN dus.QUADRANT = 'B' THEN COGNITO_ID ELSE null END)) AS QUADRANT_B_COUNT, COUNT((CASE WHEN dus.QUADRANT = 'C' THEN COGNITO_ID ELSE null END)) AS QUADRANT_C_COUNT, COUNT((CASE WHEN dus.QUADRANT = 'D' THEN COGNITO_ID ELSE null END)) AS QUADRANT_D_COUNT, ROWTIME FROM "DISTINCT_USER_STREAM" dus GROUP BY FLOOR(dus.ROWTIME TO SECOND);
- 35. Output to KinesisStream MENTION_COUNT_STREAM •UNIQUE_USER_COUNT •ANDROID_COUNT •… Amazon Kinesis Stream { "unique_user_count": 96, "android_count": 50, "ios_count": 46, "android_count": 50, "quadrant_a_count": 80, "quadrant_b_count ": 10, "quadrant_c_count ": 3, "quadrant_d_count ": 3 } 1 record, every second
- 36. Processing a KinesisStreams with AWS Lambda Shard 1 Shard 2 Shard 3 Shard 4 Shard n Kinesis Stream . . . . . . • Single instance of Lambda function per shard • Polls shard once per second • Lambda function instances created and removed automatically as stream is scaled Gets Records 1x per sec 10k records
- 37. Persist aggregated datain DynamoDB Amazon Kinesis Stream Lambda event source mapping Lambda Function Amazon DynamoDB event.Records.forEach((record) => { const payload = new Buffer(record.kinesis.data, 'base64').toString('ascii'); var docClient = new AWS.DynamoDB.DocumentClient(); var table = "user-quadrant-data"; var data = JSON.parse(payload); var params = { TableName: table, Item:{ "dataType": "quadrantRollup", "windowtime": (new Date(data.WINDOW_TIME)).getTime(), "userCount": data.UNIQUE_USER_COUNT, "quadrantA": data.QUADRANT_A_COUNT, "quadrantB": data.QUADRANT_B_COUNT, ... } }; docClient.put(params, function(err, data) { ...
- 38. Render Dashboard Amazon DynamoDB Amazon S3 JavaScript SDK Static HTML, JavaScript GetNew Data, Once per Second
- 39.
- 43.
- 45.
Editor's Notes
- #9 Narrative: The reality is that most data is produced continuously and is coming at us at lightning speeds due to an explosive growth of real-time data sources. TP: Machine data will make up 40% of our digital universe by 2020 Narrative: Whether it is log data coming from mobile and web applications, purchase data from ecommerce sites, or sensor data from IoT devices, it all delivers information that can help companies learn about what their customers, organization, and business are doing right now. TP: Customer Benefits Improve operational efficiencies, improve customer experiences, new business models Smart building: reduce energy costs, cut maintenance, increase safety and security Smart textiles: monitor skin temperature, monitor stress
- #10 Narrative: So how much is this data worth? Well, it depends… Recent data is highly valuable If you act on it in time Perishable Insights (M. Gualtieri, Forrester) Old + Recent data is more valuable If you have the means to combine them Narrative: Processing real-time data as it arrives can let you make decisions much faster and get the most value from your data. But, building your own custom applications to process streaming data is complicated and resource intensive. You need to train or hire developers with the right skillsets, and then wait for months for the applications to be built and fine-tuned, and the operate and scale the application as the business grows. All of this takes lots of time and money, and, at the end of the day, lots of companies just never get there, settle for the status-quo, and live with information that is hours or days old.
- #11 Narrative: You need a different set of analytical tools to collect and analyze real-time streaming data than what you have traditionally used for data at rest. With traditional analytics, you gather the information, store it in a database, and analyze it hours, days, or weeks later. Analyzing real-time data requires a different approach. Instead of running database queries on stored data, streaming analytics platforms have to process the data continuously and before the data lands in a database. And streaming data comes in at an incredible rate that can vary up and down all the time. Streaming analytics platforms have to be able to process this data when it arrives, often at speeds of millions and even tens of millions of events per hour. Key requirements of stream processing Durable: Durable ingest so that processing can be repeatable; Continuous - Need to always be processing the latest data Fast: Frequency (micro batches, size of batches, true streaming), and speed (sub-second, minute, hour) Correct: at most once, at least once, and exactly once processing; event time, ingest time, processing time. Reactive: Ability to process and respond in near real-time; feedback mechanisms to send processed data to live applications Reliable: Highly available, fast failovers
- #13 Since Amazon Kinesis launch in 2013, the ecosystem evolved and we introduced Kinesis Firehose and Kinesis Analytics. Streams was launched in GA at re:Invent 2014, Firehose at re:Invent 2015, and Analytics was launched in August 2016 We have continuously iterated to make it easier for customers to use streaming data, as well as expand the functionality of real-time processing Together, these three products make up the Amazon Kinesis streaming data platform
- #14 Easy administration: Simply create a new stream, and set the desired level of capacity with shards. Scale to match your data throughput rate and volume. Build real-time applications: Perform continual processing on streaming data using Kinesis Client Library (KCL), Apache Spark/Storm, AWS Lambda, and more. Low cost: Cost-efficient for workloads of any scale.
- #15 A shard is a group of data records in a stream. When you create a stream, you specify the number of shards for the stream. Each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys). The total capacity of a stream is the sum of the capacities of its shards. You can increase or decrease the number of shards in a stream as needed. However, note that you are charged on a per-shard basis.
- #17 Apply SQL on streams: Easily connect to a Kinesis Stream or Firehose Delivery Stream and apply SQL skills. Build real-time applications: Perform continual processing on streaming big data with sub-second processing latencies Easy Scalability : Elastically scales to match data throughput for most workloads Easy and interactive experience: Complete most stream processing use cases in minutes, and easily progress toward sophisticated scenarios
- #18 Zero Admin: Capture and deliver streaming data into S3, Redshift, ElasticCache and other AWS destinations without writing an application or managing infrastructure Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery into S3, and other destinations in as little as 60 secs, set up in minutes Seamless elasticity: Seamlessly scales to match data throughput (feedback: add bullet to discuss why firehose created. Major use case)
- #19 Since Amazon Kinesis launch in 2013, the ecosystem evolved and we introduced Kinesis Firehose and Kinesis Analytics. Streams was launched in GA at re:Invent 2014, Firehose at re:Invent 2015, and Analytics was launched in August 2016 We have continuously iterated to make it easier for customers to use streaming data, as well as expand the functionality of real-time processing Together, these three products make up the Amazon Kinesis streaming data platform
- #20 Please stay within brand by using the attached template. I’d recommend being visual – use imagery, font color, bold font, etc. in your slides. Be concise – limit your number of slides and content in them. It’s always good to have a few slides with backup information in case needed. Please also make sure there’s VP/Service Leader approval in place for all the content disclosed in the slides and in your call.
- #22 Connect to streaming source Select Kinesis Streams or Firehoses as input data for your SQL code Easily write SQL code to process streaming data - Author applications with a single SQL statement, or build sophisticated applications with multi-step pipelines with advanced analytical functions Continuously deliver SQL results - Configure one to many outputs for your processed data for real-time alerts, visualizations, and batch analytics The are three major components of a streaming application; connecting to an input stream, SQL code, and an output destinations. Inputs include Kinesis Streams and Firehose. Standard SQL code consists of one or more SQL queries which make up an application. Most applications will have one to three SQL statements that perform ETL after initial schematization, and then several SQL queries to generate real-time analytics on the transformed data. This illustrates the point of the “streaming application” concept; you are chaining SQL statements together in serial and parallel through filters, joins, and merges to build a streaming graph within your application. We believe most customers will have between 5-10 SQL statements, but some customers will build sophisticated applications with 50-100 statements. Outputs includes Kinesis Streams and Firehose (S3, Redshift, and Elasticsearch(coming Chicago Summit)). The next outputs will be CloudWatch Metrics and Quicksight, but we have yet to confirm whether these will be place for GA.
- #23 complex, nested JSON structures supported Inputs are continuously read, processed, and then written to output destination. This is not like a traditional database; data is continuously processed as opposed to running a query and then seeing a single result set. 1 level of nesting is supported completely; 2-levels of nesting is supported for a single structure. Deeper nesting is supported Deep nesting (2+ levels) is supported by specifying row paths of desired fields Multiple record types are supported through canonical schema and/or by manipulating structures in SQL using string functions
- #24 Simple mathematical operators (AVG, STDEV, etc.) String manipulations (SUBSTRING, POSITION) Tumbling, Sliding, and Custom Windows Advanced analytics (random sampling, anomaly detection) Define (streaming) SQL queries against streams. Join streams to other streams and tables. Aggregate queries over a time range or a series of rows. Register specific SQL statements as (stream) views. STREAM is used to represent both input and in-application data that is similar to a continuously updated table Windows are simplified to make them more approachable to the average SQL ROWTIME is a special column for streams to allow user to specify processing windows; this can be used to specify windows by time of processing (recommended) or user event time At least once processing - data will be re-processed until it has been successfully delivered to a configured destination
- #25 Delivery speed heavily dependent upon your SQL queries (i.e. streaming ETL will be very fast versus performing 10 minute aggregations) Output formats include JSON and CSV complex, nested JSON structures supported Inputs are continuously read, processed, and then written to output destination. This is not like a traditional database; data is continuously processed as opposed to running a query and then seeing a single result set. 1 level of nesting is supported completely; 2-levels of nesting is supported for a single structure. Deeper nesting is supported Deep nesting (2+ levels) is supported by specifying row paths of desired fields Multiple record types are supported through canonical schema and/or by manipulating structures in SQL using string functions
- #29 This type of application is useful in many scenarios, such as: Continuously computing business metrics like the number of purchases, sending results to a data warehouse like Amazon Redshift for visualization with a BIT tool like Amazon Quicksight. OR calculating the number application errors and sending results to Amazon Elasticsearch for visualization with Kibana. Amazon Kinesis Analytics provides the mechanisms to take raw, real-time data and output processed, meaningful information to a variety of destinations in real-time.
- #40 Feedback: break into two, flow for introducing windowing needs revision
- #42 Tumbling: Repeat Are non-overlapping An event can belong to only one tumbling window Sliding: Continuously moves forward in time Produces an output only during the occurrence of an event Every window will have at least one event Events can belong to more than one sliding window
- #43 Feedback: break into two, flow for introducing windowing needs revision
- #45 AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information. AWS Lambda starts running your code within milliseconds of an event such as an image upload, in-app activity, website click, or output from a connected device. You can also use AWS Lambda to create new back-end services where compute resources are automatically triggered based on custom requests. With AWS Lambda you pay only for the requests served and the compute time required to run your code. Billing is metered in increments of 100 milliseconds, making it cost-effective and easy to scale automatically from a few requests per day to thousands per second.
- #48 Please stay within brand by using the attached template. I’d recommend being visual – use imagery, font color, bold font, etc. in your slides. Be concise – limit your number of slides and content in them. It’s always good to have a few slides with backup information in case needed. Please also make sure there’s VP/Service Leader approval in place for all the content disclosed in the slides and in your call.
- #52 Feedback: put best practices into context that it’s a fully-managed service
- #54 Feedback: put best practices into context that it’s a fully-managed service