From Data To Insights - How to build data lake in practice? Follow the architectural guidelines for building your data architecture.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
From Data To Insights
Orit Alul
Solutions Architect, Amazon Web Services

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect from this session?
• The data architecture challenges
• Architectural principles
• Applying the architectural principles in practice
• Combining data and artificial intelligence
• Lake Formation Demo
• Summary

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Business Monitoring
Business Insights
New Business Opportunity
Business Optimization
Business Transformation
Evolving Tools and Methods
AI/MLSQL Query

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Data Architecture Challenges
• Discovering the data
• Maintaining a short time-to-insight
• Analyzing the data by different personas
• Being cost efficient

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architectural Principles
• Build decoupled systems
• Data → Store → Process → Store → Analyze → Insights
• Use the right tool for the job
• Data structure, latency, throughput, access patterns
• Leverage managed and serverless services
• Scalable/elastic, available, reliable, secure, no/low admin
• Use log-centric design patterns
• Immutable logs (data lake), materialized views
• Be cost-conscious
• Big data ≠ big cost
• AI/ML enable your applications

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sounds good!
But, How do I practically apply those principles…?

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s build together!
Use case: smart analyzer for tweets

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use case: smart analyzer for tweets
• Our goal is to get smart insights on a stream of tweets related to a specific topic
• Get the general sentiment around a topic
• Get the highlights of a topic
• Enable data scientists to run queries
• Present the highlights in a simple graphical way
• Short time-to-insight
• Cost efficiency

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Smart analyzer for tweets: accessories
• Tweepy - An easy-to-use Python library for accessing the Twitter API.
• How to scale sentiment analysis using Amazon Comprehend, AWS Glue and Amazon
Athena – blog post by Roy Hasson

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The proposed architecture
Data → Store → Process → Store → Analyze → Answers

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Firehose
• Easily load streaming data into AWS
• Seamless elasticity
• Direct-to-data store integration
AMAZON S3
AMAZON REDSHIFT

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The proposed architecture
Data → Store → Process → Store → Analyze → Answers

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Glue - ETL Service
Data Catalog ETL Job authoring
Discover data and extract
schema
Auto-generates
customizable ETL code in
Python and Spark

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Glue Data Catalog
Central Metadata Catalog for the data lake
• Unified metadata repository
across relational databases,
Amazon RDS, Amazon Redshift,
and Amazon S3.
AWS Glue Data Catalog
Central Metadata Catalog for the data lake

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Entities
Key Phrases
Language
Sentiment
Amazon
Comprehend
AMAZON COMPREHEND
Discover valuable insights from text

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AI Services
ML Services
ML Frameworks + Infrastructure
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D
& C O M P R E H E N D
M E D I C A L
L E X F O R E C A S TR E K O G N I T I O N
I M A G E
R E K O G N I T I O N
V I D E O
T E X T R A C T P E R S O N A L I Z E
Amazon SageMaker
F P G A SE C 2 P 3
& P 3 D N
E C 2 G 4
E C 2 C 5
I N F E R E N T I AG R E E N G R A S S E L A S T I C
I N F E R E N C E
D L
C O N T A I N E R S
& A M I s
E L A S T I C
K U B E R N E T E S
S E R V I C E
E L A S T I C
C O N T A I N E R
S E R V I C E
T H E A W S M L S T A C K
Broadest and deepest set of capabilities

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The proposed architecture
Data → Store → Process → Store → Analyze → Answers

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Athena - Interactive Analysis
Interactive query service to analyze data in Amazon S3 using standard SQL
SQL
Query Instantly Pay per query Open Easy
$

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Visualize with
Amazon QuickSight

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon QuickSight
Fast, easy to use, serverless analytics at 1/10th the cost of traditional BI
Empower
everyone
Seamless
connectivity
Fast analysis Serverless

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Summary
üBuild decoupled systems
üUse the right tool for the job
üLeverage managed and serverless services
üUse log-centric design patterns
üBe cost-conscious
üAI/ML enable your applications

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank You!