Amazon Kinesis Firehose was launched in the end of 2015 and provides easy way for customers to ingest streaming data to Amazon Redshift, Amazon Elasticsearch and Amazon S3. In 2016, we introduced new features that helps customers to implement in-line processing within Kinesis firehose using AWS Lambda. Amazon Kinesis Firehose makes it easy to load real-time, streaming data into AWS without having to build custom stream processing applications. This session is designed for developers, data engineers and analysts who are looking to load, analyze and get powerful insights from real0time streaming data using existing analytics tools. We will explore key features and walk through how to use Kinesis Firehose to ingest streaming data into Amazon Kinesis Analytics, Amazon S3, Amazon Redshift and Amazon Elasticsearch service.

1. Amazon Kinesis Firehose Deep Dive
Roy Ben-Alta(RBA), Principal BDM

2. What to expect from this session
• Firehose Overview
• Data Ingestion
• Data Transformation
• Data Delivery
• Monitoring
• Pricing

3. Amazon Kinesis Firehose Overview

4. Amazon Kinesis
Streams
For Technical
Developers
Build your own custom
applications that process
or analyze streaming
data
Amazon Kinesis
Firehose
For all developers, data
scientists
Easily load massive
volumes of streaming
data into S3,Amazon
Redshift and Amazon
Elasticsearch
Amazon Kinesis
Analytics
For all developers, data
scientists
Easily analyze data
streams using standard
SQL queries
Amazon Kinesis: Streaming data made easy
Services make it easy to capture, deliver and process streams on AWS

5. Amazon Kinesis Firehose
Load massive volumes of streaming data into Amazon S3, Redshift
and Elasticsearch
Zero administration: Capture and deliver streaming data into Amazon S3, Amazon Redshift, and other
destinations without writing an application or managing infrastructure.
Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery into data
destinations in as little as 60 secs using simple configurations.
Seamless elasticity: Seamlessly scales to match data throughput w/o intervention
Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform incoming
source data.
Capture and submit
streaming data
Analyze streaming data using
your favorite BI tools
Firehose loads streaming data
continuously into Amazon S3, Redshift
and Elasticsearch

6. Delivery Stream: The underlying entity of Firehose. Use Firehose
by creating a delivery stream to a specified destination and send
data to it.
Record: The data of interest that your data producer sends to a
delivery stream. A record can be as large as 1000 KB.
Data Producers: Producers send records to a Delivery Stream. For
example, a web server sends log data to a delivery stream is a data
producer.
Key Concepts

7. Data Flow to S3

8. Data Flow to Redshift

9. Data Flow to Elasticsearch

10. Data Ingestion

11. Overview
AWS SDK
• PutRecord()
• PutRecordBatch()
Kinesis Agent
• Monitors files and sends new data records to your delivery stream
• Handles file rotation, checkpointing, and retry upon failures
• Pre-processing capabilities such as format conversion and log parsing
• Emits AWS CloudWatch metrics for monitoring and troubleshooting

12. Data Transformation

13. Overview
Kinesis Firehose AWS Lambda
• Buffer to 3MB or configured buffer size if smaller
• Asynchronous Lambda invocation
• Re-buffer in Kinesis Firehose after transformation

14. Data Model
recordId: The record ID is passed from Firehose to Lambda during the
invocation. The transformed record must contain the same record ID
result: The status of the data transformation of the record
• Ok: the record was transformed successfully
• Dropped: the record was dropped intentionally by processing logic
• ProcessingFailed: the record could not be transformed
data: The transformed data payload, after base64-encoding

15. Existing Blueprints
General Firehose Processing: Contains the data transformation and
status model described in the previous section. Use this blueprint for any
custom transformation logic
Apache Log to JSON: Parses and converts Apache log lines to JSON
objects, using predefined JSON field names
Apache Log to CSV: Parses and converts Apache log lines to CSV format
Syslog to JSON: Parses and converts Syslog lines to JSON objects,
using predefined JSON field names
Syslog to CSV: Parses and converts Syslog lines to CSV format

16. Failure and Error Handling
• Configurable invocation retry (default 3 times)
• S3 persistence for failed records
• All source records backup to S3 (optional)

17. Data Delivery

18. Delivery Format
Amazon S3 Destination
• Multiple records concatenated based on buffering options (add delimiter)
• Object path: myapp/YYYY/MM/DD/HH
• Object name: DeliveryStreamName-DeliveryStreamVersion-YYYY-MM-DD-HH-MM-
SS-RandomString, where DeliveryStreamVersion
Amazon Redshift Destination
• Generate manifest file for a group of S3 objects
• COPY command from S3 to Redshift
Amazon Elasticsearch Destination
• Multiple records index through Elasticsearch Bulk API
• Record needs to be UTF-8 encoded and flattened to a single line JSON
• Index Rotation: NoRotation, OneHour, OneDay, OneWeek, OneMonth

19. Failure and Error Handling
Amazon S3 Destination
• Pause and retry for up to 24 hours (maximum data retention period
Amazon Redshift Destination
• Configurable retry duration (0-2 hours)
• After retry, skip and load error manifest files to S3’s errors/ folder
Amazon Elasticsearch Destination
• Configurable retry duration (0-2 hours)
• After retry, skip and load failed records to S3’s elasticsearch_failed/ folder

20. Access Control
Amazon S3 Destination
• IAM role
Amazon Redshift Destination
• IAM role and Redshift user name/password
• Private VPC: allow inbound access from Firehose service
Amazon Elasticsearch Destination
• IAM role

21. Monitoring

22. CloudWatch Metrics

23. CloudWatch Logs

24. Pricing

25. Dimension Value
Per 1 GB of data ingested $0.035
Firehose Pricing
* Records that are smaller than 5KB are rounded up to 5KB for billing

26. Q & A

27. • Technical documentations
• Amazon Kinesis Agent
• Amazon Kinesis Firehose and AWS Lambda
• Building Near Real-Time Discovery Platform with Amazon Kinesis
• Public case studies
• Glu mobile – Real-Time Analytics
• Hearst Publishing – Clickstream Analytics
• How Sonos Leverages Amazon Kinesis
• Nordstorm Online Stylist
Reference – RBA Bible

28. Roy Ben-Alta
benaltar@amazon.com
Your feedback
is important to
us!