Want to gain agility, reduce costs, and reduce risk by migrating your on-premises Apache Spark and Hive workloads to Amazon EMR? Learn how to re-architect vs. just doing a “lift and shift.” Discover how to separate compute and storage, build a catalog, check data quality, estimate and optimize costs, and secure everything. Get advice for Apache HBase, Presto, and Impala applications too!

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Migrating on-premises Apache Spark
and Hive to Amazon EMR
Radhika Ravirala
Specialist Solutions Architect
Data and Analytics
AWS
A D B 3 0 4

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Hadoop/Spark deployments on-premises
Tightly coupled

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Compute and storage grow together
Storage grows along with compute
Compute requirements vary
Tightly coupled

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Underutilized or scarce resources
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Underutilized or scarce resources
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Re-processingWeekly peaks
Steady state

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Contention for same resources
Compute
bound
Memory
bound

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Separation of resources creates data silos
Team A

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Replication adds to cost
Single data center
3x

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Application point of view
Large scale transformation: Map/Reduce, Hive, Pig, Spark
Interactive queries: Impala, Spark SQL, Presto
Machine learning: Spark ML, MxNet, TensorFlow
Interactive notebooks: Jupyter, Zeppelin
NoSQL: HBase

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Swim lane of jobs
Type of Job /Time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
EDR
CDR
CDR JavaMR
LSR Load Load Load Load Load Load Load Load Load Load Load Load
Aggregation
LSR Rotation
Performance Performance
RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC
BQR BQR
Others
10EDR DailyAggregations10EDR DailyAggregations
58Hive Extracts- CDR
Others
Over utilized
Under utilized

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Decouple storage and compute

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Simple Storage Service (Amazon S3) is your persistent
data store
11 nines of durability
Low cost
Life cycle policies
Versioning
Distributed by default
EMR FS
Amazon S3

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Benefit 1: Switch off clusters
Amazon S3Amazon S3
Amazon S3

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Separate them to run on persistent or long-running clusters
Type of Job /Time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
EDR
CDR
CDR JavaMR
LSR Load Load Load Load Load Load Load Load Load Load Load Load
Aggregation
LSR Rotation
Performance Performance
RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC RCC
BQR BQR
Others
10EDR DailyAggregations10EDR DailyAggregations
58Hive Extracts- CDR
Others

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Run a long-running cluster
Amazon EMR Cluster
Benefit 2: Autoscale persistent clusters to save costs

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Benefit 3: Logical separation of jobs
Hive, Pig,
Cascading
Prod
Presto Ad-Hoc
Amazon S3

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Benefit 4: Disaster recovery built in
Cluster 1 Cluster 2
Cluster 3 Cluster 4
Amazon S3
Availability Zone
Availability Zone