Amazon EMR was built for agility, enabling you to spin up and down resources for big data processing and analytics on demand, and realize the flexible potential of cloud. In this chalk talk, we discuss in detail how to efficiently start, stop, and resize your clusters for Apache Spark and Hadoop, reducing your costs, and accelerating your time to completion (TTC) for jobs. Join us to hear expert advice on how to optimize your "one-and-done" workloads.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EMR: Optimize Transient
Clusters for Data Processing & ETL
Anthony Virtuoso
Principal Engineer
AWS
A N T 3 4 1
Eric Mills
Senior Software Engineer
AWS EMR
Esther Kundin
Senior Software Engineer
AWS EMR

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• Stateless clusters
• Scaling clusters
• Reducing costs
• Cluster orchestration

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EMR basics
• Master node
• Manages cluster
• NameNode and JobTracker
• Core nodes
• Task tracker (compute)
• DataNode (HDFS)
• Task nodes
• Task tracker only
• No HDFS
Amazon EMR cluster
Task instance groupCore instance group
HDFS HDFS
Master instance group

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example cluster
• Mixed workloads
• Capacity exhausted at peak
• Paying for idle time

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Making our cluster stateless
• Maintain metastores off
cluster
• Faster startup time lowers
cost
Amazon RDS
Amazon
Redshift
Amazon
Athena
AWS Glue
AWS Glue
Data Catalog
EMR cluster EMR cluster EMR cluster

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Making our cluster stateless
Master node
CPU
Memory
HDFS
storage
CPU
Memory
HDFS
storage
CPU
Memory
HDFS
storage
CPU
Memory
HDFS
storage
Master node
CPU
Memory
CPU
Memory
CPU
Memory
CPU
Memory
Amazon S3 as streaming HDFS through EMRFS
Old clustering/Localized model Amazon EMR decoupled model
HDFS has 3x replication
500 TB dataset equals 1.5 PB cluster with replication

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling our cluster
• Scale out or in cluster task instances using automatic scaling
• Control through policies which monitor Amazon CloudWatch metrics
• Popular metrics include ’YARNMemoryAvailablePercentage’ and ‘ContainerPendingRatio’

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example cluster with Auto Scaling
• Clusters adapt to demand needs
• Peak throughput: finish faster
• Pay less for idle time

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Turning on Spot Instances
Master node Core instances Task instances
• Master node and at least one core node should be on-demand
• Launch clusters in the optimal availability zone based on capacity
and price

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Instance fleets for advanced Spot provisioning
Master node Core instances Task instances
• Mix and match instance types and Spot versus on-demand

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example cluster with Spot Instances

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Options for orchestrating a cluster
Amazon EMR JobFlow/Step API
Configure a cluster and launch a
job using Step API
Amazon EMR
AWS Data Pipeline
Airflow, Luigi, or other schedulers on
Amazon Elastic Compute Cloud (Amazon EC2)
Create a pipeline
to schedule cluster creation and
job scheduling
AWS Lambda
Use AWS Lambda to
launch clusters using the
Amazon EMR Step API
Use Oozie on your
cluster to build
DAGs of jobs
Use an external orchestration
system to launch clusters and jobs

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example cluster with orchestration

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways
• Use Amazon Simple Storage Service (Amazon S3) for storage
• Use Amazon Aurora or AWS Glue for remote metastore
• Auto Scaling for task instances
• Use Spot Instances for lower costs
• Interact with clusters and submit steps with JobFlow/Step API, AWS
Data Pipeline, or Airflow on Amazon EC2

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Anthony Virtuoso
Eric Mills
Esther Kundin

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Appendix: Job optimization
Hadoop
Compress input and output data
Adjust number of mappers and reducers
Skewed joins
Tez
Spark
Dynamic allocation settings
RDD reuse
Correct join type—broadcast join
https://spark.apache.org/docs/latest/tuning.html