In this session, we discuss several options for performing real-time extract, transform, and load (ETL) using Amazon Kinesis, AWS Lambda, AWS Glue, and Amazon S3. We provide an overview of the different options that have distinct advantages in building real-time ETL applications before loading a data lake or warehouse.

1. S U M M I T
SA NTA CLA R A 2 0 1 9

2. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Performing real-time ETL into data
lakes
A D B 2 0 2
Joyjeet Banerjee
Enterprise Solutions Architect
Amazon Web Services

3. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

4. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

5. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

6. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
,

7. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

8. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
,

9. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
”
”
,

10. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

11. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

12. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

13. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

14. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Stream ingestion
Data from tens ofthousandsof datasources canbewritten toasingle stream
AWS IoT
Amazon CloudWatch Logs
Amazon CloudWatch Events
AWS SDK LOG4J
Flume
FluentdAWS Mobile
SDK
Kinesis
Producer
Library
Kinesis Agent
*AWS DMS includes eight on-premises databases, one Azure database, five Amazon RDS / Amazon Aurora
database types, and Amazon S3
AWS toolkits & libraries AWS service integrations Third-party offerings
AWS Database Migration
Service (AWS DMS)*

15. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

16. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Data is stored inthe order it wasreceived for aset duration. It can bereplayed indefinitely
during this time.

17. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Amazon Kinesis Data Streams
• Easy administration and low cost
• Real-time, elastic performance
• Secure, durable storage
• Available to multiple real-time analytics applications

18. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Amazon Kinesis Data Firehose
• Zero administration and seamless elasticity
• Direct-to-data store integration
• Serverless continuousdata transformations

19. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

20. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Records are read in the order they are produced, enabling real-time analytics orstreaming
ETL
Amazon EMR
AWS Lambda
Kinesis
Kinesis Client Library
AWS services
Apache Spark
Third party
SQL/
Java
Amazon Kinesis
Data Analytics

21. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Amazon Kinesis Data Analytics
• Interact with streaming data in real-time using SQL or integrated Javaapplications
• Build fully managed and elastic stream processing applications

22. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Real-time streaming on AWS
Real-time streaming and analytics
Easily collect, process, and analyze video and data streams in real time
Kinesis
Video Streams
Kinesis
Data Streams
Kinesis
Data Firehose
Kinesis
Data Analytics
Capture and store video
streamsfor analytics
Load data streamsinto
AWS data stores
Analyze data streams
with SQL or Java
Collect and store data
streamsfor analytics
Amazon Managed
Streaming for Kafka
Collect and store data
streamsfor analytics

23. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Source
.
Destination
Amazon S3IoT devices

24. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Stream processing Analytical readiness

25. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
AWS IoT CoreIoT device IoT
rule
Amazon Kinesis Data
Firehose
Lambda function
Amazon S3
AWS Glue
AWS Glue Data
Catalog
Optional convert to Parquet or
ORC

26. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Stream processing Analytical readiness

27. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Amazon S3Amazon Kinesis Data
Analytics
Amazon Kinesis Data
Streams
Elastic Load
Balancing
IoT device Amazon EC2
instances

28. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Stream processing Analytical readiness

29. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

30. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
An example architecture
Amazon S3
Amazon Redshift
Amazon Elasticsearch Service
Splunk
Real-time applications (seconds)
Streaming ETL (minutes)
Stream ingestion
[Wed Oct 11 14:32:52 2018]
[error] [client 127.0.0.1]
client denied by server
configuration:
/export/home/live/ap/htdocs
/test
Mobile device
Metering
Click streams
IoT sensors
Logs
AWS SDKsAmazon Kinesis
Agent
Amazon Kinesis Producer
Library
AmazonKinesisConsumer
LibraryAmazon Kinesis Data
Analytics
Amazon Kinesis Data
Streams
Amazon Kinesis Data
Firehose

31. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Thomson Reuters provides professionals with the
intelligence, technology, and human expertise they need to
find trusted answers.
”
“

32. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T

33. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Customers using AWS Streaming solutions
1 billion events per
week from connected
devices
Near-real-time home
valuation (Zestimates)
Live clickstream
dashboards refreshed
under 10s
100 GB/day
clickstreams from 250+
sites
50 billion daily ad
impressions, sub-50
ms responses
Online stylist processing
10 million events/day
Migrated data bus
from Self-Managed
Kafka to Kinesis
Facilitate
communications
between 100+
microservices
IoT predictive
analytics
Real-time
game events analytics

34. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Amazon Go
video analytics

35. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Thank you!
S UM M I T © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.

36. © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I TS UM M I T © 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.