Hadoop2

562 views

Published on

Published in: Technology, Education
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
562
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
22
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Hadoop2

  1. 1. Hadoop2 Overview - Gagan Agrawal
  2. 2. Agenda ● HDFS ● ● ● HDFS Federation HDFS High Availability Map Reduce ● YARN Architecture
  3. 3. HDFS
  4. 4. "Classical" HDFS
  5. 5. HDFS Main Limitations ● Single NameNode that ● ● ● Performs all metadata Operations ● ● Keeps all metadata in RAM Becomes a Single Point Of Failure (SPOF) Number of files limited by the amount of RAM of one machine More RAM means also heavier GC
  6. 6. HDFS Main Limitations ● Stats based on Yahoo! Cluster ● ● An average file = 1.5 blocks (block size = 128 MB) An average file = 600 bytes of metadata (1 file and 2 blocks objects) ● 100M files = 60GB of metadata ● 1 GB of metadata = 1 PB physical storage
  7. 7. HDFS Main Limitations ● Read/Write operations throughput limited by one machine ● MapReduce tasks are also HDFS clients ● Internal load increases as the cluster grows ● ● Block Reports and heartbeats from DataNodes Bigger snapshots transferred from/to Secondary NameNode
  8. 8. HDFS Main Improvement ● Introduce multiple NameNodes ● HDFS Federation ● HDFS High Availability (HA)
  9. 9. HDFS Federation
  10. 10. HDFS Federation
  11. 11. HDFS Federation
  12. 12. HDFS Federation
  13. 13. The Client View of Cluster ● ● ● There are multiple namespaces and corresponding NameNodes Client can use any of them to compose its own view of HDFS Idea is much like Linux /etc/fstab file
  14. 14. Client Side Mount Tables
  15. 15. core-site.xml
  16. 16. Deploying HDFS Federation ● Deploying HDFS Federation ● ● NameNodes can be added/removed at any time ● ● Use for small(+isolation) and large(+scalability) cluster Cluster does have to be restarted Add Federation to the existing cluster ● Do not format existing NameNode ● Newly added NameNode will be "empty"
  17. 17. HDFS Federation Benefits ● Namespace scalability ● Performance ● Isolation
  18. 18. HDFS High Availability
  19. 19. HDFS High Availability ● Introduces a pair of redundant NameNodes ● ● One Active and one Standby If the Active NameNode crashes or is intentionally stopped, ● Then the Standby takes over quickly
  20. 20. NameNode Responsibilities ● ● The Active NameNode is responsible for all client operations The Standby NameNode is watchful ● Maintains enough state to provide a fast failover ● Does also checkpointing (disappearance of Secondary NN)
  21. 21. Synchronizing the state of metadata ● ● ● The Standby must keep its state as up-to-date as possible Edit logs - two alternative ways of sharing them ● Shared storage using NFS ● Quorum-based storage (recommended) Block locations ● DataNodes send block reports to both NameNodes
  22. 22. Shared Storage Using NFS
  23. 23. hdfs-site.xml <property> <name>dfs.nameservices</name> <value>mycluster</value> </property> <property> <name>dfs.ha.namenodes.mycluster</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.mycluster.nn1</name> <value>machine1.example.com:8020</value> </property> <property> <name>dfs.namenode.rpc-address.mycluster.nn2</name> <value>machine2.example.com:8020</value> </property>
  24. 24. Supported Failover Modes ● ● Manual fail-over ● Initiated by administrator by a dedicated command Automatic fail-over ● Detected by a fault in the active NN ● Implemented via Zookeeper
  25. 25. Manual Failover ● Initiate a failover from the first NN to the second one hdfs haadmin -failover <tobestandbyNN> <tobeactiveNN>
  26. 26. Automatic Failover
  27. 27. The Split Brain Scenario ● A potential scenario when two NameNodes ● ● ● Both think they are active Both make conflicting changes to the namespace Example: The ZKFC crashes, but its NN is still running
  28. 28. Fencing ● Ensure that the previous Active NameNode is no longer able to make any changes to the system metadata
  29. 29. Fencing with Shared Edit File ● sshfence ● ● ssh into the active namenode host and kill the process listening on the service port shell ● Alows to perform arbitrarily complex fencing using – – – IPMI (Intelligent Platform Management Interface) NFS filer level fencing or other vendor specific functions
  30. 30. Fencing with Quorum-based storage ● Only a single NN can be a writer to JournalNodes at a time ● ● ● Whenever NN becomes active, it generated an “epoch number” NN that has a higher “epoch number” can be a writer This prevents from corrupting the file system metadata
  31. 31. HDFS Highly-Available Federated Cluster ● Any combination is possible ● Federation without HA ● HA without Federation ● HA with Federation ● e.g. NameNode HA for HBase, but not for the others
  32. 32. HDFS Federation + HA
  33. 33. YARN (YET ANOTHER RESOURCE NEGOTIATOR)
  34. 34. WHY YARN?
  35. 35. Hadoop Map Reduce Classic ● Job Tracker ● ● Manages cluster resources and job scheduling Task Tracker ● Per-node agent ● Manage tasks
  36. 36. Hadoop Map Reduce Classic
  37. 37. Job Tracker Resposibilities ● ● Manages the computational resources (map and reduce slots) Schedules all user jobs ● Schedules all tasks that belongs to a job ● Monitors tasks executions ● Restarts failed and speculatively runs slow tasks ● Calculates job counters totals
  38. 38. MapReduce Classic: Limitations ● Scalability ● ● ● Maximum Cluster size – 4,000 nodes Maximum concurrent tasks – 40,000 Availability ● Job Tracker failure kills all queued and running jobs
  39. 39. MapReduce Classic: Limitations ● Hard partition of resources into map and reduce slots ● ● Low resource utilization Lacks support for alternate paradigms and services
  40. 40. Poor Cluster Resources Utilization
  41. 41. Poor Cluster Resources Utilization
  42. 42. Job Tracker Redesign Idea ● ● Reduce responsibilities of JobTracker ● Separate cluster resource management from job coordination ● Use slaves (many of them!) to manage jobs life-cycle Scale to (at least) 10K nodes, 10K jobs, 100K tasks
  43. 43. YARN : Hadoop as Next-Gen Platform
  44. 44. YARN : Taking Hadoop Beyond Batch ● ● ● Store ALL DATA in one place Interact with that data in MULTIPLE WAYS With Predictable Performance and Quality of Service
  45. 45. YARN : Concepts ● Application ● ● ● Application is a job submitted to the framework Example – Map Reduce Job Container ● ● Basic unit of allocation Fine-grained resource allocation across multiple resource type (memory, cpu, disk, network, etc.) – – ● container_0 = 2GB, 1 CPU container_1 = 1GB, 6 CPU Replaces the fixed map/reduce slots
  46. 46. YARN Architecture
  47. 47. YARN Architectue ● Resource Manager ● ● ● Global Resource Scheduler Hierarchial queues Node Manager ● ● Manages the life-cycle of container ● ● Per-machine agent Container resource monitoring Application Master ● Per-application ● Manages application scheduling and task execution ● E.g MapReduce Application Master
  48. 48. YARN Architecture
  49. 49. YARN Design Consideration ● Split up the two major functions of JobTracker ● Cluster resource management ● Application life-cycle management
  50. 50. 5 Key Benefits of YARN ● ● ● ● ● Scale New Programming Models and Services Improved cluster utilization Agility Beyond Java
  51. 51. Thank You

×