Immersion Day Presentation

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Daniel Bento, Solutions Architect
Consuming The DataLake
Data Lake, Reporting, Analytics, Machine
Learning

2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Anti-Pattern
Everything
Query

3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Also an Anti-Pattern
Everything
Query

4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
One tool to
rule them all

5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Where do I start?
• Understand your data
• Data Structure, Access patterns & characteristics,
Temperature, Cost, Size
• Know your audience
• Business Users, Data Scientists, Developers
• Select the right service

6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Archival
In-memory Warehouse
NoSQL
Hot data Warm data Cold data
Data
Structure
Low
High
Object
Search
Understand your Data
Latency
Data volume
HighLow
Request rate
Cost / GB
High Low

7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
ElastiCache
Amazon ES
Amazon
DynamoDB
Amazon S3 Amazon Glacier
Hot data Warm data Cold data
Data
Structure
Low
High
Understand your Data
Latency
Data volume
HighLow
Request rate
Cost / GB
High Low
NoSQL
Object
Archival
Search
In-Memory
Warehouse
Amazon Redshift

8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Who is your audience?

9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PRIORITIES NEEDS
Creating engaging visual and narrative journeys
for analytical solutions
Data Visualizer
Manages data as a product. Ensures freshness
and consistency of data; understands lineage and
compliance needs; treats DS as customers
Data Product
Manager
Monitoring for reliability, quickly diagnose
deployment or availability issues
DevOps
Engineer
ROLE
Visualization
Dashboards
Reporting
Reports – data quality, errors
Ad hoc querying
Dashboards
Makes sense of data, generates and communicates
insights to improve or create business processes,
creates predictive ML models to support them
Data Scientist
Ad hoc querying
Robust ML tools
Builds scalable pipelines, transforms and loads data
into structures complete with metadata that can be
readily consumed by DS
Data
Engineer
Ad hoc querying
Quick visualization
Vetting the priortization and ROI, funding projects,
providing ongoing feedback
Business
Sponsor
Reporting
Dashboards

10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enabling your Consumers
Dashboards – Reports – Ad-Hoc Analysis – Machine Learning

11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dashboards
Visual Representation of key metrics that change over time
• Data structure - Low
• Usage - Near real-time visualization
• Data temperature - Hot
Available Services:
AWS Lambda Amazon DynamoDB Amazon Kinesis
Data Streams
Amazon Elasticsearch
Service

12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dashboards – Near Real-time
Amazon
EMR
AWS Glue
OR
ETL
Data Lake
Amazon
S3
Raw Bucket Transformed Data
Bucket
DynamoDB
Users
EC2
Containers
Serverless
OR
OR
Web serving layer

13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dashboards + Search
Amazon
EMR
AWS Glue
OR
ETL
Data Lake
Amazon
S3
Raw Bucket Transformed Data
Bucket
DynamoDB
Users
DynamoDB
Streams
Amazon Kinesis
Firehose
AWS
Lambda
Amazon
Elasticsearch

14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ad Hoc Analysis
Information sought on an as-needed basis
• Usage - Dynamic Data Querying
• Data structure - Case based
• Data temperature - Medium - cold
Available Services:
Amazon Redshift
Spectrum
Athena Amazon
EMR
Amazon
ElasticSearch

15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reports and Ad-Hoc Analysis
Amazon
QuickSight
OR
Amazon Redshift
Athena
Amazon
EMR
AWS Glue
OR
ETL
Data Lake
Amazon
S3
Raw Bucket Transformed Data
Bucket
Amazon Redshift
Spectrum

16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine Learning
Data labeled with outcomes to train predication models
• Usage - Machine learning data preparation
• Data structure - Case based
• Data temperature - Medium - cold
Available Services:
Amazon
EMR
Amazon
SageMaker

17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine Learning
Amazon EMR
Users
Amazon
EMR
AWS Glue
OR
ETL
Data Lake
Amazon
S3
Raw Bucket Transformed Data
Bucket
Amazon
SageMaker

18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reports
Static representations of data rendered at a point in time
• Usage - Point in time data extraction
• Data structure - High
• Data temperature – Medium - cold
Available Services:
Amazon Redshift Amazon Athena Amazon QuickSight

19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift
Data Scientists & Developers
Business
UsersAmazon Redshift
Amazon
QuickSight
OR
Amazon
EMR
AWS Glue
OR
ETL
Data Lake
Amazon
S3
Raw Bucket Transformed Data
Bucket
Amazon EMR
Data
Scientist
BI/BA
Engineer

20. Data Lake e pipeline de dados end-to-end

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Speed (Processamento em Real-time)
Ingestão Camada de entregaData
sources
Scale (Processamento em Batch)
Midias sociais
Near-Zero Latency
Amazon DynamoDB
Staged Data
(Data Lake)
Amazon S3
Raw Data
(Data Lake)
Amazon S3
ETL/Processamento
Amazon EMR
Advanced Analytics
Amazon EMR
Captura de
Eventos
Amazon Kinesis
Firehose
Data Lake e pipeline de dados end-to-end
Cientistas de Dados
Modelagem analítica/AI
Amazon EMR
Outras fontes
Crawlers
Camada de APIs
API Gateway
AWS Lambda

22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Obrigado!