Building Data Lakes for Analytics on AWS

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Building data lakes for analytics
on AWS
Gautam Srinivasan
AWS solutions architect
A D B 2 0 1

Agenda
Trends in analytics
The AWS analytics portfolio
Services & customer results

More than 125 million players
Data provides a constant feedback loop for
game designers
Up-to-the-minute analysis of gamer
satisfaction to drive gamer engagement
Resulting in the most popular game played
in the world
Fortnite

Customers want more value from their data
Growing
exponentially
From new
sources
Increasingly
diverse
Used by
many people
Analyzed by
many applications

Companies want more value from their data
Complications
Siloed approaches don’t work anymore
It’s too expensive and limiting
to store data on-premises
Implication
A new approach is needed to
extract insights and value

Cloud data lakes are the future
Data Lake
Customers want:
To move to a single store; i.e., a data lake in the cloud
To store data securely in standard formats
To grow to any scale, with low costs
To analyze their data in a variety of ways
To democratize data access and analysis

Why choose AWS for data lakes and analytics?
Most
comprehensive
Most
secure
Easiest
to build
Most
cost effective
Most
customers
& partners

Most comprehensive
Broadestand deepestportfolio, purpose-built forbuilders
Migration & Streaming Services
Infrastructure Data Catalog
& ETL
Security &
Management
Dashboards Predictive Analytics
Data
Warehousing
Big Data
Processing
Interactive
Query
Operational
Analytics
Real time
Analytics
Serverless
Data Processing
Visualization & Machine Learning
Data Movement
Analytics
Data Lake Infrastructure & Management

Data Movement
Analytics
Most comprehensive
Broadestand deepestportfolio, purpose-built forbuilders
+ 10 more
Redshift
EMR (Spark &
Hadoop)
Athena
Elasticsearch
Service
Kinesis Data
Analytics
AWS Glue
(Spark &
Python)
S3/Glacier AWS GlueLake
Formation
QuickSight SageMaker Comprehend Lex Polly Rekognition Translate Transcribe
Deep Learning
AMIs
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Managed Streaming for Kafka
Data Lake Infrastructure & Management

Most secure
Servicesforsecurity and governance
Compliance
AWS Artifact
Amazon Inspector
AWS CloudHSM
Amazon Cognito
AWS CloudTrail
Security
Amazon GuardDuty
AWS Shield
AWS WAF
Amazon Macie
Amazon Virtual Private Cloud
(Amazon VPC)
Encryption
AWS Certificate Manager
AWS Key Management Service
Encryption at rest
Encryption in transit
Bring your own keys, CloudHSM
support
Identity
AWS IAM
AWS SSO
Amazon Cloud Directory
AWS Directory Service
AWS Organizations
Customers need to have multiple levels of security, identity and access management, encryption,
and compliance to secure their data lake

Most secure – Certifications
CSA
Cloud Security
Alliance Controls
ISO 9001
Global Quality
Standard
ISO 27001
Security Management
Controls
ISO 27017
Cloud Specific
Controls
ISO 27018
Personal Data
Protection
PCI DSS Level 1
Payment Card
Standards
SOC 1
Audit Controls
Report
SOC 2
Security, Availability, &
Confidentiality Report
SOC 3
General Controls
Report
Global United States
CJIS
Criminal Justice
Information Services
DoD SRG
DoD Data
Processing
FedRAMP
Government Data
Standards
FERPA
Educational
Privacy Act
FIPS
Government Security
Standards
FISMA
Federal Information
Security Management
GxP
Quality Guidelines
and Regulations
ISO FFIEC
Financial Institutions
Regulation
HIPPA
Protected Health
Information
ITAR
International Arms
Regulations
MPAA
Protected Media
Content
NIST
National Institute of
Standards and Technology
SEC Rule 17a-4(f)
Financial Data
Standards
VPAT/Section 508
Accountability
Standards
Asia Pacific
FISC [Japan]
Financial Industry
Information Systems
IRAP [Australia]
Australian Security
Standards
K-ISMS [Korea]
Korean Information
Security
MTCS Tier 3 [Singapore]
Multi-Tier Cloud
Security Standard
My Number Act [Japan]
Personal Information
Protection
Europe
C5 [Germany]
Operational Security
Attestation
Cyber Essentials
Plus [UK]
Cyber Threat
Protection
G-Cloud [UK]
UK Government
Standards
IT-Grundschutz
[Germany]
Baseline Protection
Methodology
X P
G

Most cost effective
Decouplecompute and storage,choiceofPAYG analytics services
Storage
Amazon S3 tiers &
intelligent tiering
From $0.023 per
GB/mo to as low as
$0.004 per GB/mo
Compute
Spot & reserved
instances
Save up to 90% off
on-demand prices
EMR
Autoscaling
57% less than
on-premises
per IDC report
Redshift
less than a tenth
of the cost of
traditional solutions.
Athena &
QuickSight
Serverless, pay
only for what is used

More data lakes and analytics than anywhere else
Morethan 10,000 data lakes on AWS

Most partners to complement AWS offerings

Data movement solutions
Migration & Streaming Services
Data Movement

Most ways to move data to the data lake
Data movement from
on-premises datacenters
Dedicated network connection
Secure appliances
Ruggedized shipping containers
Database migration
Gateway that lets applications write to the cloud
Data movement from real-time sources
Connect devices to AWS
Real-time data streams
Real-time video streams
Data movement from
real-time sources
Data movement from your
on-premises datacenters
Amazon S3
Amazon S3 Glacier
AWS Glue
Synchronizing data
across environments
Professional services and partners
to help migration
Data
movement

Data lake infrastructure
& management solutions
Infrastructure Data Catalog
& ETL
Security &
Management
Data lake infrastructure & management

Amazon S3
Lake Formation & Glue
Snowball Kinesis
Data Streams
Snowmobile Kinesis
Data Firehose
Amazon
Redshift
Amazon
EMR
Athena
Kinesis
Amazon
Elasticsearch
Service
Robust data lake infrastructure
Amazon
SageMaker
Amazon
Comprehend
Amazon
Rekognition
Durable and available; exabyte scale
Secure, compliant, auditable
Object-level controls for fine-grain access
Fast performance by retrieving subsets of data
Decoupling of compute and storage
On-demand resources, tiering, cost choices
& management

“Zestimates are more up-to-date
and accurate, because they’re
built with the absolute latest
data. That’s a huge benefit for
our users, who depend on this
information to influence their
buying or selling decisions.”
— Jasjeet Thind, vice president of data science and
engineering, Zillow Group
& management

Set up a catalog, ETL, and data prep
with AWSGlue
Serverless provisioning, configuration, and
scaling to run your ETL jobs on Apache Spark
Pay only for the resources used for jobs
Crawl your data sources, identify data
formats and suggest schemas and
transformations
Automates the effort in building, maintaining
and running ETL jobs
& management

“Beeswax uses Amazon S3 and AWS
Glue Data Catalog to build a highly
reliable data lake that is fully
managed by AWS. Our platform
leverages the AWS Glue Data Catalog
integration with Amazon EMR in
Hive and SparkSQL applications to
deliver reporting and optimization
features to our customers.”
— Ram Kumar Rengaswamy, CTO, Beeswax
& management

Challenges to making a secure data lake
Typical steps in building a data lake
Move data2 Cleanse, prep, and
catalog data
3
Configure and enforce security
and compliance policies
4
Make data available
for analytics5
Set up storage1
& management

Build a secure data lake in days
with AWSLake Formation
Move, store, catalog, and
clean your data faster
Move, store, catalog,
and clean your data faster
with Machine Learning
Enforce security policies
across multiple services
Enforce security policies across
multiple services
Gain and manage
new insights
Empower analyst and data scientist to
gain and manage new insights
& management

& management
“With an enterprise-ready option
like Lake Formation, we will be
able to spend more time deriving
value from our data rather than
doing the heavy lifting involved in
manually setting up and managing
our data lake.”
— Joshua Couch, VP engineering at Fender Digital

Analytics solutions
Data
Warehousing
Big Data
Processing
Interactive
Query
Operational
Analytics
Real time
Analytics
Serverless
Data Processing

Big data processing with Apache Spark & Hadoop
with Amazon EMR
Easy to use notebooks
Low cost versus on-premises
Elastic autoscaling
Reliable 99.9% SLA
Secure with encryption and keys
Flexible, open-source choice
Analytics
Enterprise grade Easy Lowest cost

Analytics
FINRA’s legacy system did not scale
to handle 75 billion events per day. It
needed to run complex surveillance
queries on more than 20 PB of data
FINRA moved its big data appliance
to an Amazon S3 data lake and uses
Amazon EMR for ingestion and
processing

The Forrester Wave
Cloud Hadoop/Spark Platforms
Q12019
“The 11 Providers That Matter Most
and How They Stack Up”
by Noel Yuhanna and Mike Gualtieri
February 13, 2019
The Forrester Wave is copyrighted by Forrester Research, Inc. Forrester and Forrester Wave are
trademarks of Forrester Research, Inc. The Forrester Wave is a graphical representation of
Forrester's call on a market and is plotted using a detailed spreadsheet with exposed scores,
weightings, and comments. Forrester does not endorse any vendor, product, or service
depicted in the Forrester Wave. Information is based on best available resources. Opinions
reflect judgment at the time and are subject to change.

Data warehouse for business reporting
with Amazon RedShift
Fast-up to 10x faster than
traditional data warehouses
Easy to set up, deploy, and
manage
Cost effective
Scale on demand for large data
volume and high query
concurrency
Query data in open formats
directly from the data lake
Analytics

Analytics
“Twenty percent of our queries
now complete in less than one
second. Best of all, we didn’t have
to change anything to get this
speed-up with Redshift, which
supports our mission-critical
workloads.”
— Greg Rokita, executive director of technology, Edmunds

Real-time analytics for timely insights
with Amazon Kinesis
Make streaming data available to
multiple real-time analytics
applications
Run streaming applications without
managing any infrastructure
Durable to reduce the probability
of data loss
Scalable to process data from hundreds
of thousands of sources with low
latencies
Analytics

Analytics
“Amazon Kinesis makes it simple to
scale our solution end to end,
including the capture, processing,
and delivery of actionable insights.
This empowers our customers to
better understand their user base.”
—Indu Narayan, director of data, Yieldmo

Operational analytics for logs and search
with Amazon Elasticsearch Service
Fully managed; deploy
production-ready cluster
in minutes
Direct access to Elasticsearch
open-source APIs, Logstash
and Kibana
Amazon VPC support; at-rest
and in-transit encryption
Easily scale up and down
Analytics

Interactive analysis
with Amazon Athena
Interactive query service to analyze data in
Amazon S3 using standard SQL
No infrastructure to set up or manage, and no
data to load
Ability to run SQL queries on data archived in
Amazon Glacier (coming soon)
Analytics

Analytics
“One of the big attractions of Amazon Athena
is that it’s serverless and purely
consumption based.”
“We only pay when we’re actually querying
the data, and we don’t have to keep a
cluster running all the time. Using Amazon
Athena, we’re able to query seven years’
worth of data—adding up to hundreds of
terabytes—get results at least 50% faster,
and save nearly $15,000 per month.”
— Matt Chesler, director of DevOps at Movable Ink

Serverless analytics
Deliveron-demand analytics onthe data lake
Amazon S3
Data lake
AWS Glue
(ETL &
Data Catalog)
Athena
QuickSight
Serverless; zero infrastructure,
zero administration
Never pay for
idle resources
$
Availability and fault
tolerance
built in
Automatically scales
resources with usage
AWS IoT
AI/ML
Devices Web Sensors Social
Analytics

Visualization & machine learning
solutions
Dashboards Predictive Analytics

Visual insights for everyone
with Amazon QuickSight
Pay only for what you use
Scale to tens of thousands of users
Embedded analytics
Build end-to-end BI solutions
Visualization &
machine learning

Visual insights for everyone
With AWSML & AI services
Frameworks and interfaces for
machine learning practitioners
Platform services that make it easy
for any developer to get started and
get deep with ML
Application services that enable
developers to plug in pre-built
AI functionality into their apps
Visualization &
machine learning
Amazon S3
Raw Data Initial training data
is annotated by
human labelers
Active learning model
is trained from human
labeled data
Ambiguous data is sent to human
labelers for annotation
Human labeled data is then sent
back to retrain and improve the
machine learning model
Training data the
model understands is
labeled automatically
An accurate training data
set is ready for use in
Amazon SageMaker

Visualization &
machine learning
Using Amazon Translate,
Lionbridge is able to scale
machine translation in order to
localize content faster and in
more languages.
Using Amazon Translate,
Lionbridge was able to reduce
translation costs by 20%.

Next steps…
Dive deeper into
specific AWS services
Set up a proof-of-concept
Talk about how professional
services can help
Sign up for an
AWS account
Instantly get access
to the AWS Free Tier
Learn with
10-minute tutorials
Explore and learn with
simple tutorials
Start building
with AWS
Begin building with step-by-step
guide to help you launch
your AWS project.

Building Data Lakes for Analytics on AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building Data Lakes for Analytics on AWS

Similar to Building Data Lakes for Analytics on AWS (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Building Data Lakes for Analytics on AWS