AWS Dev Day Kyiv 2019 Track: Analytics & Machine Learning Session: "Building a Modern Data platform in the Cloud" Speaker: Alex Casalboni, AWS Technical Evangelist Level: 300 AWS Dev Day is a free, full-day technical event where new developers will learn about some of the hottest topics in cloud computing, and experienced developers can dive deep on newer AWS services. Provectus has organized AWS Dev Day Kyiv in close collaboration with Amazon Web Services: 800+ participants, 18 sessions, 3 tracks, a really AWSome Day!  Now, together with Zeo Alliance, we're building and nurturing AWS User Group Ukraine — join us on Facebook to stay updated about cloud technologies and AWS services: https://www.facebook.com/groups/AWSUserGroupUkraine Video: https://youtu.be/HIDnAG9AxZo

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
K Y I V
06.11.19
Building a Modern Data
Platform in the Cloud
Alex Casalboni
Sr. Technical Evangelist
Amazon Web Services
@alex_casalboni

2. About me
• Software Engineer & Web Developer
• Worked in a startup for 4.5 years
• ServerlessDays Organizer
• AWS Customer since 2013

3. S U M M I T
bit.ly/AWSDataLakeDemo

4. Organizations that successfully
generate business value from their
data, will outperform their peers. An
Aberdeen survey saw organizations
who implemented a Data Lake
outperforming similar companies by
9% in organic revenue growth.*
24%
15%
Leaders Followers
Organic revenue growth
*Aberdeen: Angling for Insight in Today’s Data Lake, Michael Lock, SVP Analytics and Business Intelligence
To Become a Leader, Data is Your Differentiator

5. Data variety and data volumes are increasing rapidly
Multiple Consumers and Applications
Ingest
Discover
Catalog
Understand
Curate
Find insights

6. Purpose-built
engines
Right tool for the job

7. Collect Store Analyze
Amazon Kinesis
Firehose
AWS Direct
Connect
Amazon
Snowball
Amazon Kinesis
Analytics
Amazon Kinesis
Streams
Amazon S3 Amazon Glacier
Amazon
CloudSearch
Amazon RDS,
Amazon Aurora
Amazon
Dynamo DB
Amazon
Elasticsearch
Amazon EMR
Amazon
Redshift
Amazon
QuickSight
AWS Database
Migration Service AWS Glue
Amazon
Athena
Amazon
SageMaker

8. Traditionally, Analytics Used to Look Like This
OLTP ERP CRM LOB
Data Warehouse
Business Intelligence • Relational data
• TBs–PBs scale
• Schema defined prior to data load
• Operational reporting and ad hoc
• Large initial CAPEX + $10K–$50K/TB/Year

10. “A data lake is a centralized repository that
allows you to store all your structured and
unstructured data at any scale”

11. Collect analyze
semi-structured unstructured
Decoupled
ingestion
on-read
warehouses

12. exabyte scale
once
many tools
Open formats

13. S3
ElasticsearchGlueDynamoDB
Catalog & search
Cognito
API
Gateway
API/UI
Athena QuickSight
Redshift
Spectrum
Analytics & processing
LambdaKinesis
Streams
Kinesis
Firehose
Direct
Connect
Ingest
AWS
IoT
KMS CloudTrailIAM Macie
Security & auditing

14. CHALLENGE
Need to create constant feedback loop
for designers
Gain up-to-the-minute understanding
of gamer satisfaction to guarantee
gamers are engaged, thus resulting in
the most popular game played in the
world
Fortnite | 125+ million players

16. time
Capture, process, and
store video streams for
analytics
Load data streams into
AWS data stores
Analyze data streams with
SQL
Build custom applications
that analyze data streams
Kinesis Video Streams Kinesis Data Streams Kinesis Data Firehose Kinesis Data Analytics

17. Amazon S3:
Buffered files
Kinesis
Agent
Record
producers Amazon Redshift:
Table loads
Amazon Elasticsearch Service:
Domain loads
Amazon S3:
Source record backup
Transformed recordsPut Records
Kinesis Firehose:
Delivery stream

18. Amazon S3:
Buffered files
Kinesis
Agent
Record
producers Amazon Redshift:
Table loads
Amazon Elasticsearch Service:
Domain loads
Amazon S3:
Source record backup
Transformed recordsPut Records
Kinesis Firehose:
Delivery stream
AWS Lambda:
Transformations &
enrichment
Raw Transformed

19. Open-source standards (Apache)
Parquet, ORC, etc.
Optimize Performance
Optimize Costs
Analytical queries

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

22. Storing is Not Enough, Data Needs to Be Discoverable
Dark data are the information
assets organizations collect,
process, and store during
regular business activities,
but generally fail to use for other
purposes (for example, analytics,
business relationships and
direct monetizing).
CRM ERP Data warehouse Mainframe
data
Web Social Log
files
Machine
data
Semi-
structured
Unstructured
“
”Gartner IT Glossary, 2018
https://www.gartner.com/it-glossary/dark-data

23. Building training sets
Cleaning and organizing data
Collecting data sets
Mining data for patterns
Refining algorithms
Other
80%

24. &
Data Catalog
ETL Job
authoring
Discover data and
extract schema
Auto-generates
customizable ETL code
in Python and Spark
Data & schema automatic discovery
Generates customizable code for ETL
Schedule and run ETL jobs periodically
Serverless model

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Crawlers automatically build your
data catalog and keep it in sync
Automatically discover new data & extract
schema definitions
Detect schema changes and version tables
Detect Hive style partitions on Amazon S3
Built-in classifiers for popular types; custom
classifiers using Grok expression
Run ad hoc or on a schedule; serverless – only
pay when crawler runs
AWS Glue Crawlers
Crawlers
Automatically catalog your data

26. AWS Lake Formation (join the preview)
Build, secure, and manage a data lake in days
Build a data lake in days,
not months
Build and deploy a fully
managed data lake with a few
clicks
Enforce security policies
across multiple services
Centrally define security,
governance, and auditing policies in
one place and enforce those policies
for all users and all applications
Combine different
analytics approaches
Empower analyst and data scientist
productivity, giving them self-
service discovery and safe access to
all data from a single catalog

28. User-Defined Functions
• Bring your own functions & code
• Execute without provisioning servers
Processing and Querying In Place
Fully Managed Process & Query
AWS
Glue
Amazon
Athena
Amazon
Redshift
Amazon
SageMaker
AWS
Lambda

29. Query S3 using standard SQL (Presto as distributed engine)
Serverless - No infrastructure to set up or manage
Multiple data format support – Define Schema on Demand
$
Query Instantly Pay per query Open Easy

31. Data scanned: 169.53GB (of 2.2TB)
Query duration: 44.66 seconds
Cost: $0.85
($5/TB or $0.005/GB)
SELECT gram, year, sum(count)
FROM ngram
WHERE gram = 'just say no'
GROUP BY gram, year
ORDER BY year ASC;
registry.opendata.aws/google-ngrams

32. year 2018 month 11 day 25

33. Amazon QuickSight
easy
Empower
everyone
Seamless
connectivity
Fast analysis Serverless

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. S U M M I T
bit.ly/AWSDataLakeDemo

36. JSON Payload Example for each event
{
"r": 255,
"g": 0,
"b": 0,
"c": "Red",
"device": {
"id": "4992157",
"browser": "Chrome",
"browserVersion": "72.0.3626.109",
"os": "Mac OS",
"isMobile": false,
"isMobileIOS": false,
"isMobileAndroid": false
},
"dt": {
"year": 2019,
"month": 4,
"day": 17,
"hour": 16,
"minutes": 30,
"seconds": 47,
"millis": 725
},
"id": 1551116627725,
"region": "Europe",
"awsExperience": "1-3 Years",
"awsServiceArea": "Management Tools"
}

37. Demo Architecture
Amazon CloudFront
Amazon Cognito
Amazon S3
Web App
Users Amazon Kinesis
Data Firehose
Amazon AthenaAWS Glue Amazon
QuickSight
Client
Mobile
client
AWS SDK
S3 Bucket
AWS Cloud
Region

38. Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Alex Casalboni
@alex_casalboni