In recent years, there has been explosive growth in the number of connected devices and real-time data sources. Data is being produced continuously and its production rate is accelerating. Businesses can no longer wait for hours or days to use this data. To gain the most valuable insights, they must use this data immediately so they can react quickly to new information. In this chalk talk, we discuss how to take advantage of streaming data sources to analyze and react in near-real time. In addition, we present different options for how to solve a real-world scenario and walk through those solutions.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A N T 3 6 0
Jeremy Zigmond
Principal Solutions Architect
AWS

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data is produced continuously
Mobile apps Web clickstream Application logs
Metering records IoT sensors Smart buildings
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/ht
docs/test

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The diminishing value of data

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis makes it easy to work with real-time
streaming data
Amazon Kinesis Data
Streams
For technical developers
Collect and stream data for
ordered, replayable, real-
time processing
Amazon Kinesis Data
Firehose
• For all developers, data
scientists
• Easily load massive volumes
of streaming data into
Amazon Simple Storage
Service (Amazon S3),
Amazon Redshift, Amazon
Elasticsearch Service
Amazon Kinesis Data
Analytics
• For all developers, data
scientists
• Easily analyze data streams
using standard SQL queries

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Streams
Reliably ingest and durably store streaming data at low cost
Build custom real-time applications to process streaming data

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Firehose
Reliably ingest and deliver batched, compressed, and encrypted data
to Amazon S3, Amazon Redshift, and Amazon ES
Point-and-click setup with zero administration and seamless elasticity

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Data Analytics
Interact with streaming data in real time using SQL
Build fully managed and elastic stream processing applications that
process data for real-time visualizations and alarms

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Producers
SDKs
• Publish directly from application code via PutRecord and PutRecords APIs
Kinesis Agent
• Tail log files and forward lines as messages to Kinesis Data Streams
Kinesis Producer Library (KPL)
• Background process aggregates and batches messages
• Producer application calls PutUserRecord method
Third party and open source
• Log4j appender
• Flume, Fluentd source libraries

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Consumers
Direct API access
• Custom application, using GetShardIterator and GetRecords APIs
• Application responsible for shard processing, check-points, reshard operations
Kinesis Client Library (KCL)
• Open source library, available in several languages
• Manages stream checkpointing
• Manages shard-worker relationships on reshard, or consumer instance scaling
AWS Lambda
• Serverless stream processing
• Lambda function is invoked only when messages exist on stream
• One Lambda function instance per shard
Third-party and open source
• Spark Streaming
• Storm Spout

11. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jeremy Zigmond
Principal Solutions Architect
AWS

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.