Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
San Francisco Loft - 2017
Introduction to Real-time, Streaming
Data and Amazon Kinesis:
Streaming Data Ingestion with
Fire...
What to Expect from the Session
• Streaming data overview
• Firehose patterns overview
• Firehose usage patterns
• Streami...
What is (Data) Streaming?
Streaming Data is data that is generated continuously by thousands of data
sources, which typically send in the data recor...
Streaming Data Overview
Most data is produced continuously
Mobile Apps Web Clickstream Application Logs
Metering Records IoT Sensors Smart Buildin...
The diminishing value of data
• Recent data is highly valuable
• Old + Recent data is more valuable
Processing real-time, streaming data
• Durable
• Continuous
• Fast
• Correct
• Reactive
• Reliable
What are the key requir...
Amazon Kinesis Platform Overview
Real-time streaming data made easy
Amazon Kinesis
Streams
• For Technical Developers
• Collect and stream data
for ordered...
Amazon Kinesis Streams
• Reliably ingest and durably store streaming data at low cost
• Build custom real-time application...
Amazon Kinesis Analytics
• Interact with streaming data in real-time using SQL
• Build fully managed and elastic stream pr...
Amazon Kinesis Firehose
• Reliably ingest and deliver batched, compressed, and
encrypted data to S3, Redshift, and Elastic...
Amazon Kinesis makes it easy to work with
real-time streaming data
Amazon Kinesis
Firehose
• For all developers, data
scie...
Amazon Kinesis
Producers Consumers
Shard 1
Shard 2
Shard n
Shard 3
…
…
Write: 1MB Read: 2MB
** A shard is a group of data ...
Amazon Kinesis Firehose
Producers Amazon S3
Amazon ES
Amazon Redshift
Shard 1
Shard 2
Shard n
Shard 3
…
…
Firehose to Amazon S3
Firehose to Amazon Redshift
Firehose to Amazon Elasticsearch
Amazon	Kinesis	Firehose	vs.	Amazon	Kinesis	Streams
Amazon Kinesis Streams is for use cases that require custom processing,...
What are common use cases for
Firehose?
IoT: Get Insights from Telemetry Data
IoT: Get Insights from Telemetry Data
Assemble a Real-time Advertising Solution
Optimize Digital Marketing with Clickstream
Analytics
Firehose Demo (IoT)
Amazon	Kinesis
Firehose
Amazon	S3 Amazon	Athena AWS	Quicksight
AWS	IoT
Sensor(s)
Firehose Demo (Clickstream)
Amazon	Kinesis
Firehose
Amazon	S3 Amazon	Athena AWS	Quicksight
Users	browse	content
Amazon Firehose:
deployments & testing
Kinesis Firehose Pricing
Thank you
Amazon Kinesis Customer Base Diversity
1 billion events/wk from
connected devices | IoT
17 PB of game data per
season | En...
Introduction to Real-time, Streaming Data and Amazon Kinesis. Streaming Data Ingestion with Firehose
Introduction to Real-time, Streaming Data and Amazon Kinesis. Streaming Data Ingestion with Firehose
Introduction to Real-time, Streaming Data and Amazon Kinesis. Streaming Data Ingestion with Firehose
Introduction to Real-time, Streaming Data and Amazon Kinesis. Streaming Data Ingestion with Firehose
Introduction to Real-time, Streaming Data and Amazon Kinesis. Streaming Data Ingestion with Firehose
Introduction to Real-time, Streaming Data and Amazon Kinesis. Streaming Data Ingestion with Firehose
Upcoming SlideShare
Loading in …5
×

Introduction to Real-time, Streaming Data and Amazon Kinesis. Streaming Data Ingestion with Firehose

1,250 views

Published on

by Adrian Hornsby, Technical Evangelist, AWS

Amazon Kinesis is a platform for streaming data on AWS, offering powerful services to make it easy to load and analyze streaming data. In this session, you’ll learn about how AWS customers are transitioning from batch to real-time processing using Amazon Kinesis, and how to get started. We will provide an overview of streaming data applications and introduce the Amazon Kinesis platform and its services. We will walk through a production use case to demonstrate how to ingest streaming data, prepare it, and analyze it to gain actionable insights in real time using Amazon Kinesis. We will also provide pointers to tutorials and other resources so you can quickly get started with your streaming data application.

Introduction to Real-time, Streaming Data and Amazon Kinesis. Streaming Data Ingestion with Firehose

  1. 1. San Francisco Loft - 2017 Introduction to Real-time, Streaming Data and Amazon Kinesis: Streaming Data Ingestion with Firehose Adrian Hornsby Technical Evangelist with AWS
  2. 2. What to Expect from the Session • Streaming data overview • Firehose patterns overview • Firehose usage patterns • Streaming data end-to-end example and walk- through
  3. 3. What is (Data) Streaming?
  4. 4. Streaming Data is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of Kilobytes). Streaming data includes a wide variety of data such as log files generated by customers using your mobile or web applications, ecommerce purchases, in-game player activity, information from social networks, financial trading floors, or geospatial services, and telemetry from connected devices or instrumentation in data centers.
  5. 5. Streaming Data Overview
  6. 6. Most data is produced continuously Mobile Apps Web Clickstream Application Logs Metering Records IoT Sensors Smart Buildings [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/h tdocs/test
  7. 7. The diminishing value of data • Recent data is highly valuable • Old + Recent data is more valuable
  8. 8. Processing real-time, streaming data • Durable • Continuous • Fast • Correct • Reactive • Reliable What are the key requirements? Ingest Transform Analyze React Persist
  9. 9. Amazon Kinesis Platform Overview
  10. 10. Real-time streaming data made easy Amazon Kinesis Streams • For Technical Developers • Collect and stream data for ordered, replayable, real-time processing Amazon Kinesis Firehose • For all developers, data scientists • Easily load massive volumes of streaming data into Amazon S3, Redshift, ElasticSearch Amazon Kinesis Analytics • For all developers, data scientists • Easily analyze data streams using standard SQL queries
  11. 11. Amazon Kinesis Streams • Reliably ingest and durably store streaming data at low cost • Build custom real-time applications to process streaming data
  12. 12. Amazon Kinesis Analytics • Interact with streaming data in real-time using SQL • Build fully managed and elastic stream processing applications that process data for real-time visualizations and alarms
  13. 13. Amazon Kinesis Firehose • Reliably ingest and deliver batched, compressed, and encrypted data to S3, Redshift, and Elasticsearch • Point and click setup with zero administration and seamless elasticity
  14. 14. Amazon Kinesis makes it easy to work with real-time streaming data Amazon Kinesis Firehose • For all developers, data scientists • Easily load massive volumes of streaming data into Amazon S3, Redshift, ElasticSearch
  15. 15. Amazon Kinesis Producers Consumers Shard 1 Shard 2 Shard n Shard 3 … … Write: 1MB Read: 2MB ** A shard is a group of data records in a stream
  16. 16. Amazon Kinesis Firehose Producers Amazon S3 Amazon ES Amazon Redshift Shard 1 Shard 2 Shard n Shard 3 … …
  17. 17. Firehose to Amazon S3
  18. 18. Firehose to Amazon Redshift
  19. 19. Firehose to Amazon Elasticsearch
  20. 20. Amazon Kinesis Firehose vs. Amazon Kinesis Streams Amazon Kinesis Streams is for use cases that require custom processing, per incoming record, with sub-1 second processing latency, and a choice of stream processing frameworks. Amazon Kinesis Firehose is for use cases that require zero administration, ability to use existing analytics tools based on Amazon S3, Amazon Redshift and Amazon Elasticsearch, and a data latency of 60 seconds or higher.
  21. 21. What are common use cases for Firehose?
  22. 22. IoT: Get Insights from Telemetry Data
  23. 23. IoT: Get Insights from Telemetry Data
  24. 24. Assemble a Real-time Advertising Solution
  25. 25. Optimize Digital Marketing with Clickstream Analytics
  26. 26. Firehose Demo (IoT)
  27. 27. Amazon Kinesis Firehose Amazon S3 Amazon Athena AWS Quicksight AWS IoT Sensor(s)
  28. 28. Firehose Demo (Clickstream)
  29. 29. Amazon Kinesis Firehose Amazon S3 Amazon Athena AWS Quicksight Users browse content
  30. 30. Amazon Firehose: deployments & testing
  31. 31. Kinesis Firehose Pricing
  32. 32. Thank you
  33. 33. Amazon Kinesis Customer Base Diversity 1 billion events/wk from connected devices | IoT 17 PB of game data per season | Entertainment 80 billion ad impressions/day, 30 ms response time | Ad Tech 100 GB/day click streams from 250+ sites | Enterprise 50 billion ad impressions/day sub-50 ms responses | Ad Tech 10 million events/day | Retail Amazon Kinesis as Databus - Migrate from Kafka to Kinesis| Enterprise Funnel all production events through Amazon Kinesis

×