Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Operationalizing YARN Based Hadoop
Clusters in the Cloud
Abhishek Modi
Lead Developer,
Yarn and Hadoop Team,
Qubole
Hadoop at Qubole
● Over 300 Petabytes data processed per month.
● More than 100 customers with more than 1000 active users...
Qubole Architecture
Qubole UI
Qubole
SaaS
Hadoop
Cluster
Hadoop
Cluster
Hadoop
Cluster
Hadoop
Cluster
Cloud
Storage
Prod
N...
Ephemeral Hadoop Clusters
Bring up Cluster Perform Jobs Terminate Cluster
Scale
Up
Scale
Down
• Use cloud storage for job output and input.
• Needs to auto-scale as per work-load.
• Store job history and logs at pers...
YARN Auto-scaling
Up-scaling for MR jobs
Resource
Manager
Node 1
Node 2
User
Submit Job
Launches
MR AM
NodeManager
MR AppMaster
Container
Re...
Generic Up-scaling
Resource
Manager
Cluster
Manager
MR
AppMaster
Spark
AppMaster
Tez
AppMaster
Up Scale
Request
Add
Node
Node 2
Down-scaling
Resource
Manager
NodeManager
C1 C2
C3 C4
NodeManager
C1 C2
C3 C4
NodeManager
C1 C2
C4C3
Status
Update
...
Re-commissioning
NodeManager
C2C1
NodeManager
C1 C2
C4C3
C4C3
NodeManager
C4
C2C1
Resource
Manager
Graceful Shutdown
User
...
• Containers contains output of Map tasks.
• Can not be terminated until Map output is consumed.
• Upload Map output to cl...
• DFS used and incoming data rate is monitored periodically.
• Upscale if free DFS goes below an absolute threshold.
• Ups...
Cost Benefits of Auto-scaling
• AWS and Google Cloud provide volatile nodes termed as “Spot Nodes” or “Pre-
emptible Nodes”
• Available at very low pric...
• While starting cluster, percentage of volatile nodes can be specified.
• A maximum ‘bid’ price for volatile nodes is als...
• While up-scaling, RM tries to maintain volatile node percentage.
• If volatile node are not available, fall back to stab...
• Show job history for terminated clusters.
• Multi-tenant job history server.
• Clusters are generally running in isolate...
Job History – Running cluster
Qubole
UI
Cluster Proxy
Hadoop
Cluster
Hadoop
Cluster
Hadoop
Cluster
Hadoop
Cluster
User
Cli...
Job History – Terminated Cluster
Qubole
UI
User
Cluster
Proxy
Job
History
Server
Clicks
UI link
Authenticates
the request
...
• Writing output directly to cloud without storing at temporary location.
• Optimizations in getting file status for large...
• Issues with newer version of JetS3t (0.9.4)​
– Seek performance degraded around 10X.​
– Empty files.​
• Deadlock when nu...
Thank You
Upcoming SlideShare
Loading in …5
×

Operationalizing YARN based Hadoop Clusters in the Cloud

702 views

Published on

Operationalizing YARN based Hadoop Clusters in the Cloud

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Operationalizing YARN based Hadoop Clusters in the Cloud

  1. 1. Operationalizing YARN Based Hadoop Clusters in the Cloud Abhishek Modi Lead Developer, Yarn and Hadoop Team, Qubole
  2. 2. Hadoop at Qubole ● Over 300 Petabytes data processed per month. ● More than 100 customers with more than 1000 active users. ● Over 1 million Hadoop jobs completed per month. ● More than 8,000 Hadoop clusters brought up per month.
  3. 3. Qubole Architecture Qubole UI Qubole SaaS Hadoop Cluster Hadoop Cluster Hadoop Cluster Hadoop Cluster Cloud Storage Prod New Qubole REST API
  4. 4. Ephemeral Hadoop Clusters Bring up Cluster Perform Jobs Terminate Cluster Scale Up Scale Down
  5. 5. • Use cloud storage for job output and input. • Needs to auto-scale as per work-load. • Store job history and logs at persistent location. • Adapting YARN/HDFS to take into account ephemeral cloud nodes. Challenges: Ephemeral Hadoop Clusters
  6. 6. YARN Auto-scaling
  7. 7. Up-scaling for MR jobs Resource Manager Node 1 Node 2 User Submit Job Launches MR AM NodeManager MR AppMaster Container Request Allocate Resources NodeManager C1 C2 Task Progress Up Scale Request Cluster Manager Add Node NodeManager C3 C4 Node 3
  8. 8. Generic Up-scaling Resource Manager Cluster Manager MR AppMaster Spark AppMaster Tez AppMaster Up Scale Request Add Node
  9. 9. Node 2 Down-scaling Resource Manager NodeManager C1 C2 C3 C4 NodeManager C1 C2 C3 C4 NodeManager C1 C2 C4C3 Status Update Evaluates cluster is being underutilized and can be down scaled Selects node whose estimated task completion time is lowest Graceful Shutdown User Submits Job Allocates container Job1 Completes Cluster Manager Remove Node Job 1 Job 2 Job 3 Decommission Node Node 1 Node 3
  10. 10. Re-commissioning NodeManager C2C1 NodeManager C1 C2 C4C3 C4C3 NodeManager C4 C2C1 Resource Manager Graceful Shutdown User Submit Job Allocates Containers C3 Upscale Request Re-commission
  11. 11. • Containers contains output of Map tasks. • Can not be terminated until Map output is consumed. • Upload Map output to cloud. • Reducers access Map output directly from cloud. Further Optimizations in Down-scaling
  12. 12. • DFS used and incoming data rate is monitored periodically. • Upscale if free DFS goes below an absolute threshold. • Upscale if free DFS is projected to go below absolute threshold in next few minutes. HDFS Based Up-Scaling
  13. 13. Cost Benefits of Auto-scaling
  14. 14. • AWS and Google Cloud provide volatile nodes termed as “Spot Nodes” or “Pre- emptible Nodes” • Available at very low price as compared to stable nodes. • Can be lost at any point of time without any prior notification. • Hadoop’s failure resilience makes these nodes good candidates for Hadoop. • Approx. 77% of all Qubole clusters make use of volatile nodes. Volatile Nodes
  15. 15. • While starting cluster, percentage of volatile nodes can be specified. • A maximum ‘bid’ price for volatile nodes is also specified. • Qubole Placement Policy: – Ensures at least one replica of each HDFS block is present on Stable Node. – No Application Master is scheduled on volatile nodes. Volatile Nodes at Qubole
  16. 16. • While up-scaling, RM tries to maintain volatile node percentage. • If volatile node are not available, fall back to stable nodes. • Periodically tries to re-balance the volatile node percentage. Rebalancing – Volatile Nodes
  17. 17. • Show job history for terminated clusters. • Multi-tenant job history server. • Clusters are generally running in isolated networks – need a proxy. • Job History files needs to be stored at cloud storage. Job History
  18. 18. Job History – Running cluster Qubole UI Cluster Proxy Hadoop Cluster Hadoop Cluster Hadoop Cluster Hadoop Cluster User Clicks UI link Authenticates the request Find cluster corresponding to the request Proxifies link in html and js Sends Request
  19. 19. Job History – Terminated Cluster Qubole UI User Cluster Proxy Job History Server Clicks UI link Authenticates the request Finds cluster is down Fetches jhist file from cloud Jhist file Rendered JobHist Proxifies Link
  20. 20. • Writing output directly to cloud without storing at temporary location. • Optimizations in getting file status for large number of files with common prefix. • Added streaming upload support in NativeS3FileSystem. • Added bulk delete and move support in NativeS3FileSystem. Cloud Read/Write Optimizations
  21. 21. • Issues with newer version of JetS3t (0.9.4)​ – Seek performance degraded around 10X.​ – Empty files.​ • Deadlock when number of threads reading from S3 exceeds JetS3t’s max number of connections (HADOOP-12739).​ • Too many queues causes a deadlock in cluster.(YARN-3633)​ • Support for Socks Proxy was missing from HA.​ Open Source Issues
  22. 22. Thank You

×