Accelerate Data Analytics at Scale with Amazon EMR In this session you will learn the best practices and various use cases for performing data analytics at scale with Amazon EMR. We will introduce you to Amazon EMR design patterns and share how to use big data analytics to provide business insights. Jonathan Fritz, Principal Product Manager, Amazon Web Services

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jonathan Fritz
Principal Product Manager – Amazon Web Services
Accelerate Analytics At Scale With
Amazon EMR

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Intro and architecture
Using Amazon EC2 Spot and Auto Scaling
Security overview
Ad hoc and advanced workflows
Customer use cases

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Is Amazon EMR?
Low cost
Pay per-second
Open-source variety
Latest versions of software
Secure
Easy to enable options
Flexible
Full customisation and control
Easy to use
Launch a cluster in minutes
Managed
Spend less time monitoring
Flink

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Open-Source Applications
Amazon EMR
service

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use The AWS Glue Data Catalog
• Support for Spark,
Hive and Presto
• Auto-generate
schema and
partitions
• Managed table
updates

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hbase For Random Access At Massive Scale
HDFS
(data node)
Local Disk
HBase region
server
YARN node
manager

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tips To Lower Your Costs

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transient Or Long-Running Clusters
• Shut your clusters down when you don’t need them
• Use Amazon Linux AMI with preinstalled customisations for faster startup
• Use Auto Scaling to minimise costs for long-running clusters

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EC2 Spot And Instance Fleets
• EMR will select optimal EC2 AZ
• Provision across instance types
• Switch to on-demand

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use Auto Scaling
Scaling options
Threshold
CloudWatch or custom metric

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto Scaling
• EMR scales-in at YARN task completion
• Selectively removes nodes with no running tasks
• yarn.resourcemanager.decommissioning.timeout
• Default timeout is one hour
• Spark scale-in contributions
• Spark specific blacklisting of tasks
• Unregistering cached data and shuffle blocks
• Advanced error handling

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tips To Secure Your Cluster

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Encryption
• Spark
• Tez
• MapReduce
• Presto
• HBase
• Hive
• Pig

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Authentication
LDAP
HiveServer2
Presto Coordinator
Spark Thrift Server
Hue Server
Zeppelin Server
AWS credentials
EMR Step (EMR API)
EC2 key pair
SSH as “hadoop”

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New – Authentication With Kerberos
Microsoft
Active Directory
KDC
Users
YARN RMDoAs
Service principals for
all cluster nodes
Master Node

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Authorisation
• Storage-based
• EMRFS/S3
• HDFS
• HiveServer2 and Presto (SQL-based)
• HBase
• YARN queues
• Fine-grained access control by cluster tag (IAM)
• Apache Ranger on edge node (using CloudFormation)

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configure IAM Roles For EMRFS Requests
Context
User: aduser
Group: analyst
IAM role: analytics_prod
Can map IAM roles to user, group, or S3 prefix
Context
User: aduser2
Group: dev
IAM role: analytics_dev

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tips To Submit Workflows

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use Livy As An Ad-hoc Spark Job Server
Custom
Application
Livy

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Oozie And Airflow For Dags Of Jobs

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customer Use Cases

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fraud Detection
FINRA uses Amazon EMR
and Amazon S3 to process
up to 75 billion trading
events per day and securely
store over 5 petabytes of
data, attaining savings of
$10-20mm per year.

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
FINRA Saved 60% And Reduced Operational Load
100 x m3.2xlarge
1 x c3.4xlarge
HFiles on S3
App
Servers
HBase Client
HBase API
Calls
Query Cluster ETL Cluster
HBase
Bulk Load
60 x m3.2xlarge
1 x c3.4xlarge
On Spot
Lookups ETL

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2 Petabytes Processed Per Day
Learn
Models
ModelsImpressions
Clicks
Activities
Calibrate
Evaluate
Real
Time
Bidding
S3
ETL Attribution
Machine
Learning
S3Amazon Kinesis

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Consolidated ETL And ML Pipeline
CDN
Real Time
Bidding
Retargeting
Platform
Amazon Kinesis Attribution & ML
S3
Reporting
Data Visualisation
Data
Pipeline
ETL(Spark SQL)
Event Data
• Impressions
• Activities
• Attributions
• (Facts)
Reference Data
(Dimensions)
Application Logs
Exceptions Data
Reporting Data
Zeppelin notebooks

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Machine Learning
RECOMMENDATION API
(Python, R, Flask)
Zillow Group
Data Lake
(S3 / Kinesis)
Property Featurisation
(Spark EMR)
User Profiles
(Spark EMR)
Ranking
(Spark EMR)
Wedge Counting
Collaborative Filtering
(Spark EMR)
Property Aggregate Features
(Spark EMR)
Data Collection Systems
(Java/Python/SQL)

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Training And Scoring
Collect user behavior and real-estate data, train the various
models, generate the candidate set, and make predictions.
User
Behavior
(Kinesis
/S3)
Public
Record
(Kinesis /
S3)
Event API
(Java)
Producer
(Python)
Filter
(Spark)
User Store
(Hive / S3)
Active
Listings
(Kinesis /
S3)
Producer
(Python)
Training Data
(Spark) Training Set
(Hive / S3)
Models
(Python)
Train Models
(Spark)
Score
(Spark)
Recommendations
Hashmap
(Redis)
Spark job creates
Hive table with user
events (uid, pid)
partitioned by date
Property Data
pid -> uid reverse
index
Past and current
user events
Wedge features or
property features
(user profile)
Collaborative
Filtering / User
Profile Models

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ad Hoc Environment
Scale cluster to accommodate more users

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank You
jonfritz@amazon.com
https://aws.amazon.com/emr/