Data Analytics Week at the San Francisco Loft Preparing Data for the Lake While a Data Lake can support completely unstructured data, getting performant analytics at scale requires some data preparation. We'll look at how to use Amazon Kinesis, AWS Glue, and Amazon EMR to make raw data ready to high-performance analytics. Speakers: John Mallory - Principal Business Development Manager Storage (Object), AWS Hemant Borole - Sr. Big Data Consultant, AWS

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Preparing Data for the Lake
John Mallory
johmallo@amazon.com

2. Characteristics of a data lake
Future
Proof
Flexible
Access
Dive in
Anywhere
Collect
Anything

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3 as the data lake

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Simplified architectural view
Amazon S3
Ingestion
mechanism
Data sources
Transactions
Web logs /
cookies
ERP
Connected
devices
Process Consume

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
There are lots of ingestion tools
Amazon S3
Process Consume
S3 Transfer
Acceleration
Data sources
Transactions
Web logs /
cookies
ERP
Connected
devices

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Variety of data processing tools
Amazon S3
Consume
S3 Transfer
Acceleration
Amazon AI
ML/DL Services
Amazon Athena
Interactive Query
Amazon EMR
Managed Hadoop & Spark
Amazon Redshift + Spectrum
Petabyte-scale Data
Warehousing
Amazon Elasticsearch
Real-time log analytics & search
Data sources
Transactions
Web logs /
cookies
ERP
Connected
devices

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
And multiple ways to consume the data
Amazon S3
S3 Transfer
Acceleration
Amazon AI
ML/DL Services
Amazon Athena
Interactive Query
Amazon EMR
Managed Hadoop & Spark
Amazon Redshift + Spectrum
Petabyte-scale Data
Warehousing
Amazon Elasticsearch
Real-time log analytics & search
Data sources
Transactions
Web logs /
cookies
ERP
Connected
devices
Amazon QuickSight
Fast, easy to use, cloud BI
Analytic Notebooks
Jupyter, Zeppelin, HUE
Amazon API Gateway
Programmatic Access

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Because data is not prefect
AWS Lambda
Trigger-based Code
Execution
AWS Glue
Event based Server-less ETL
engine
Amazon EMR
Spark and Hive running on
EMR
Because data is not never prefect
Clean
Transform
Concatenate
Convert to better formats
Schedule transformations
Event-driven transformations
Transformations expressed as
code

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ETL when you need it
Amazon S3
S3 Transfer
Acceleration
Amazon AI
ML/DL Services
Amazon Athena
Interactive Query
Amazon EMR
Managed Hadoop & Spark
Amazon Redshift + Spectrum
Petabyte-scale Data
Warehousing
Amazon Elasticsearch
Real-time log analytics & search
Data sources
Transactions
Web logs /
cookies
ERP
Connected
devices
Amazon QuickSight
Fast, easy to use, cloud BI
Analytic Notebooks
Jupyter, Zeppelin, HUE
API Gateway
Programmatic Access

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Metadata?
AWS Glue Data Catalog
Central Metadata Catalog for the data lake
One per account
Allows you to share metadata between
Amazon Athena, Amazon Redshift
Spectrum, EMR & JDBC sources
We added a few extensions:
§ Search over metadata for data
discovery
§ Connection info – JDBC URLs,
credentials
§ Classification for identifying and
parsing files
§ Versioning of table metadata as
schemas evolve and other metadata
are updated

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Catalog Crawlers
AWS Glue Data Catalog - Crawlers
Helping Catalog your data
Crawlers automatically build your Data
Catalog and keep it in sync
Automatically discover new data, extracts
schema definitions
• Detect schema changes and version tables
• Detect Hive style partitions on Amazon S3
Built-in classifiers for popular types; custom
classifiers using Grok expression
Run ad hoc or on a schedule; serverless – only
pay when crawler runs

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Glue Data Catalog

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Table schema
Table properties
Data statistics
Nested fields
Data Catalog – Table Details

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
List of table versionsCompare schema versions
Data Catalog: Version Control

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Automatically register available partitions
Table
partitions
Automatic Partition Detection

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A central metadata store for your lake
Amazon S3
S3 Transfer
Acceleration
Amazon AI
ML/DL Services
Amazon Athena
Interactive Query
Amazon EMR
Managed Hadoop & Spark
Amazon Redshift + Spectrum
Petabyte-scale Data
Warehousing
Amazon Elasticsearch
Real-time log analytics & search
Data sources
Transactions
Web logs /
cookies
ERP
Connected
devices
Amazon QuickSight
Fast, easy to use, cloud BI
Analytic Notebooks
Jupyter, Zeppelin, HUE
API Gateway
Programmatic Access
AWS Glue Data Catalog
Hive-compatible Metastore

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-time (instream processing)
Amazon S3
S3 Transfer
Acceleration
Amazon AI
ML/DL Services
Amazon Athena
Interactive Query
Amazon EMR
Managed Hadoop & Spark
Amazon Redshift + Spectrum
Petabyte-scale Data
Warehousing
Amazon Elasticsearch
Real-time log analytics & search
Data sources
Transactions
Web logs /
cookies
ERP
Connected
devices
Amazon QuickSight
Fast, easy to use, cloud BI
Analytic Notebooks
Jupyter, Zeppelin, HUE
API Gateway
Programmatic Access
AWS Glue Data Catalog
Hive-compatible Metastore
Spark Streaming
& Flink on EMR
Amazon Kinesis
Analytics

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
W r i t e o n c e , c a t a l o g o n c e , r e a d m u l t i p l e , E T L A n y w h e r e
Amazon S3
S3 Transfer
Acceleration
Amazon AI
ML/DL Services
Amazon Athena
Interactive Query
Amazon EMR
Managed Hadoop & Spark
Amazon Redshift + Spectrum
Petabyte-scale Data
Warehousing
Amazon Elasticsearch
Real-time log analytics & search
Data sources
Transactions
Web logs /
cookies
ERP
Connected
devices
Amazon QuickSight
Fast, easy to use, cloud BI
Analytic Notebooks
Jupyter, Zeppelin, HUE
API Gateway
Programmatic Access
AWS Glue Data Catalog
Hive-compatible Metastore
Spark Streaming
& Flink on EMR
Amazon Kinesis
Analytics

19. Characteristics of a data lake
Future
Proof
Flexible
Access
Dive in
Anywhere
Collect
Anything

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s take an example
1. What is going on with a specific
sensor
2. Daily Aggregations (device,
inefficiencies, average temperature)
3. A real-time view of how many
sensors are showing inefficiencies
1. Scale
2. Highly availability
3. Less management overhead
4. Pay what I need
Business Questions
Operations
Record-level dataSensor/IOT device

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s push this data into a Kinesis
Kinesis Firehose
Amazon S3
Amazon S3 Amazon Athena

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Querying it in Amazon Athena
Either Create a Crawler to
auto-generate schema
OR
Write a DDL on the Athena
console/API/ JDBC/ODBC
driver
Start Querying Data

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Query daily aggregates in Amazon Athena
Kinesis Firehose
Amazon S3
Amazon S3
Amazon Athena
“raw-time-series”
Amazon S3
“daily-average”

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Glue Job
Serverless, event-driven execution
Data is written out to S3
Output table is automatically
created in Amazon Athena

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Query daily aggregates in Amazon Athena
Kinesis Firehose
Amazon S3
Amazon S3
Amazon Athena
“raw-time-series”
Amazon S3
“daily-average”

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis Analytics for in-stream analytics
Kinesis Firehose
Amazon S3
Amazon S3 Amazon Athena
“raw-time-series”
Amazon S3
“daily-average”
Amazon S3
Kinesis Analytics Kinesis Firehose
“results”

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
KPI - Overall device daily inefficiency"
SELECT ( SUM(daily_avg_inefficiency)/COUNT(*) )
AS all_device_avg_inefficiency, date
FROM awsblogsgluedemo.daily_avg_inefficiency
GROUP BY date;
Top 10 most inefficient devices - event-level granularity
SELECT col0 AS "uuid", col1 AS "deviceid", col2 AS
"devicets", col3 AS "temp", col4 AS "settemp", col5 AS
"pct_inefficiency" FROM awsblogsgluedemo.results ORDER BY
pct_inefficiency DESC limit 10;
“raw” table with raw data
Top 20 most active devices
SELECT
deviceid, COUNT(*) AS num_events
FROM awsblogsgluedemo."raw"
GROUP BY deviceid
ORDER BY num_events DESC
Events by Device ID
SELECT uuid, devicets,deviceid,
temp
FROM awsblogsgluedemo."raw"
WHERE deviceid = 1
ORDER BY devicets DESC;
“daily-agg” table with daily
aggregation
“result” table

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Overall architecture
Kinesis Firehose
Amazon S3
Amazon S3 Amazon Athena
“raw-time-series”
Amazon S3
“daily-average”
Amazon S3
Kinesis Analytics Kinesis Firehose
“results”

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
C h a r a c t e r i s t i c s
ü Scale to hundreds of thousands of data sources
ü Virtually infinite storage scalability
ü Real-time and batch processing layers
ü Interactive queries
ü Highly available and durable
ü Pay only for what you use
No servers to manage

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
aws.amazon.com/activate
Everything and Anything Startups
Need to Get Started on AWS