Building Data Lakes and Analytics on AWS

P U B L I C S E C T O R
S U M M I T
NEW DELHI

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Building Data Lake with AWS
Sachin Arora
Head BD, Big Data and Analytics
AWS
Sachin Punyani
Solutions Architect
AWS

S U M M I T
Challenge
Needed to analyze data to find insights, identify
opportunities, and evaluate business performance
The Oracle DW was costly, difficult to maintain, and
did not scale
Solution
Deployed a data lake with Amazon S3 and ran
analytics with Amazon Redshift, Amazon Redshift
Spectrum, and Amazon EMR
Result: They doubled the data stored (100PB),
lowered costs, and gained insights faster
50 PB of data
600,000 analytics jobs/day

S U M M I T
“20% of our queries now
complete in less than one
second. Best of all, we didn’t
have to change anything to
get this speed-up with
Amazon Redshift, which
supports our mission-critical
workloads.”
— Greg Rokita, Executive Director
of Technology, Edmunds

S U M M I T
Analytics
FINRA’s legacy system did not
scale to handle 75 billion events
per day. They needed to run
complex surveillance queries
over 20+ PB of data.
FINRA migrated their big data
appliance to an Amazon S3 data
lake and uses Amazon EMR for
ingestion and processing.

S U M M I T
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stop
throwing
data away
Make it
available to
more users
Arm users
with more
data processing
technologies
Organizations get more value from data
By thinking about data as an asset not a cost

S U M M I T
… but realizing value from data is challenging
What’s holding you back from using data?
Unable to link data together
Data collected too
infrequently Data difficult to access
96%
of organizations
state data is not used
39%
of organizations do not
regularly collect data
66%
of organizations find data
is difficult to access
Without the right platform, data insights remain elusive

S U M M I T
… but realizing value from data is challenging
What’s holding you back from using data?
Unable to link data together
Data collected too
infrequently Data difficult to access
96%
of Organizations
state data is not used
39%
of Organizations do not
regularly collect data
66%
of Organizations find data
is difficult to access
Without the right platform, data insights remain elusive
Siloed approaches don’t work anymore
A new approach is needed to extract insights and value

S U M M I T
Cloud data lakes are the future
Customers want:
A single data store that is scalable and cost effective
To store data securely in standard formats
To analyze their data in a variety of ways
Cloud Data Lake
Infrastructure
Decoupled Storage
& Compute Resources
Security & Governance
Data
Migration
Streaming
Services
Data
Warehouse
Big Data
Processing
Serverless Data
Processing
Real-time
Analytics
Operational
Analytics
Predictive
AnalyticsETL & Catalog
Data Management

S U M M I T
Most secure – the most certifications – virtually every agency
CSA
Cloud Security
Alliance Controls
ISO 9001
Global Quality
Standard
ISO 27001
Security Management
Controls
ISO 27017
Cloud Specific
Controls
ISO 27018
Personal Data
Protection
PCI DSS Level 1
Payment Card
Standards
SOC 1
Audit Controls
Report
SOC 2
Security, Availability, &
Confidentiality Report
SOC 3
General Controls
Report
Global United States
CJIS
Criminal Justice
Information Services
DoD SRG
DoD Data
Processing
FedRAMP
Government Data
Standards
FERPA
Educational
Privacy Act
FIPS
Government Security
Standards
FISMA
Federal Information
Security Management
GxP
Quality Guidelines
and Regulations
ISO FFIEC
Financial Institutions
Regulation
HIPPA
Protected Health
Information
ITAR
International Arms
Regulations
MPAA
Protected Media
Content
NIST
National Institute of
Standards and Technology
SEC Rule 17a-4(f)
Financial Data
Standards
VPAT/Section 508
Accountability
Standards
Asia Pacific
FISC [Japan]
Financial Industry
Information Systems
IRAP [Australia]
Australian Security
Standards
K-ISMS [Korea]
Korean Information
Security
MTCS Tier 3 [Singapore]
Multi-Tier Cloud
Security Standard
My Number Act [Japan]
Personal Information
Protection
Europe
C5 [Germany]
Operational Security
Attestation
Cyber Essentials
Plus [UK]
Cyber Threat
Protection
G-Cloud [UK]
UK Government
Standards
IT-Grundschutz
[Germany]
Baseline Protection
Methodology
X P
G

S U M M I T
More data lakes and analytics than anywhere else
More than 10,000 data lakes on AWS

S U M M I T
Most partners to complement AWS offerings

S U M M I T

S U M M I T
AI/ML
Algorithms
Insights
Data
Storage
Analyze
Alert
Store
S3
Data Lake Conceptual Architecture

S U M M I T
Data lake on AWS
Amazon
DynamoDB
Amazon Elasticsearch
Service
AWS
AppSync
Amazon
API Gateway
Amazon
Cognito
AWS
KMS
AWS
CloudTrail
AWS
IAM
Amazon
CloudWatch
AWS
Snowball
AWS Storage
Gateway
Amazon
Kinesis Data
Firehose
AWS Direct
Connect
AWS Database
Migration
Service
Amazon
Athena
Amazon
EMR
AWS
Glue
Amazon
Redshift
Amazon
DynamoDB
Amazon
QuickSight
Amazon
Kinesis
Amazon
Elasticsearch
Service
Amazon
Neptune
Amazon
RDS
AWS
Glue

S U M M I T
Data Movement
Analytics
The AWS analytics portfolio
Amazon
Redshift
Amazon EMR
(Spark & Hadoop)
Athena
Amazon ES Kinesis Data
Analytics
AWS Glue
(Spark & Python)
Amazon S3/
Amazon S3 Glacier
AWS GlueLake
Formation
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Managed Streaming for Kafka
Data Lake Infrastructure & Management
+ 11 more
Visualization, Engagement, & Machine Learning
Amazon
QuickSight
Amazon
SageMaker
Amazon
Comprehend Amazon
Lex
Amazon
Polly
Amazon
Rekognition
Amazon
Translate
Amazon
Transcribe
Amazon
Pinpoint

S U M M I T
Predictive insights
With AWS ML services and AI
With AWS services developers
can plug-in pre-built AI
functionality into their apps
ML platform services that make
it easy for any developer to get
started and get deep with ML
ML frameworks and interfaces
for machine learning
practitioners
Amazon S3
Raw Data Initial training data
is annotated by
human labelers
Active learning model
is trained from human
labeled data
Ambiguous data is sent to human
labelers for annotation
Human labeled data is then sent
back to retrain and improve the
machine learning model
Training data the
model understands is
labeled automatically
An accurate training data
set is ready for use in
Amazon SageMaker
Visualization,
Engagement, & ML

S U M M I T
Next steps…
Dive deeper into
specific AWS services
Set up a proof-of-concept
Talk about how
professional services can
help
Sign up for an
AWS account
Instantly get access
to the AWS Free Tier
Learn with
10-minute tutorials
Explore and learn with
simple tutorials
Start building
with AWS
Begin building with step-by-step
guide to help you launch
your AWS project

S U M M I T
Thank you!
S U M M I T
Sachin Arora
Head BD, Big Data and Analytics
AWS
Sachin Punyani
Solutions Architect
AWS

S U M M I T
Download our app to enhance
your Summit experience
Access the agenda, build your own
schedule, provide feedback easily
and more

S U M M I T
Three ways to get started:
• Scan the QR code on the screen or at
the back of your attendee badge
• Search “AWS Global Summits” in
Apple Store or Google Play
• Visit guidebook.com/app/aws

S U M M I T
Tap on the
featured guide
Tap “Download
Guide”
Tap
“Open”

S U M M I T
Summit Session Feedback
Take a quick five question survey – let us
know how we can improve.
Three ways to take the survey:
• Access the Summit app - session survey tab
• Scan the QR code
• Visit https://amzn.to/summit-session
Ballroom 1 & 2

S U M M I T
S U M M I T

Building Data Lakes and Analytics on AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building Data Lakes and Analytics on AWS

Similar to Building Data Lakes and Analytics on AWS (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Building Data Lakes and Analytics on AWS