High availability hadoop november 2010

3,126 views
3,035 views

Published on

Nov 2010 version of Johannes' talk on running Hadoop MapReduce against the Ibrix systemically.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,126
On SlideShare
0
From Embeds
0
Number of Embeds
1,165
Actions
Shares
0
Downloads
54
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

High availability hadoop november 2010

  1. 1. © Copyright 2010 Hewlett-Packard Development Company, L.P.1© Copyright 2010 Hewlett-Packard Development Company, L.P. Johannes Kirschnick, Steve Loughran November 2010 MAKING HADOOP HIGHLY AVAILABLE Using an alternative File system – HP IBRIX
  2. 2. © Copyright 2010 Hewlett-Packard Development Company, L.P.2 © Copyright 2010 Hewlett-Packard Development Company, L.P.2 SOMETHING ABOUT ME – I work at HP Labs, Bristol, UK • Degree in computer science, TU Munich – Automated Infrastructure Lab • Automated, secure, dynamic instantiation and management of cloud computing infrastructure and services – Personal interest • Cloud Services • Automated service deployment • Storage Service
  3. 3. © Copyright 2010 Hewlett-Packard Development Company, L.P.3 © Copyright 2010 Hewlett-Packard Development Company, L.P.3 WHAT DO I WANT TO TALK ABOUT – Motivate High Availability – Overview about Hadoop • Word count example – Failure Modes in Hadoop – Introduce HP IBRIX – Performance Results – Summary
  4. 4. © Copyright 2010 Hewlett-Packard Development Company, L.P.4 © Copyright 2010 Hewlett-Packard Development Company, L.P.4 WHY AN ALTERNATIVE FILE SYSTEM – High availability • Continued availability in times of failures for − Hadoop Map Reduce Layer Service − Data stored in the file system – Fault tolerant operation • What happens if a node dies – Reduce time to restart – Alternative File system • security out of the box • Generic file system
  5. 5. © Copyright 2010 Hewlett-Packard Development Company, L.P.5 © Copyright 2010 Hewlett-Packard Development Company, L.P.5 HADOOP IN A NUTSHELL Job sd Dearest creature in creation, Study English pronunciation. I will teach you in my verse Sounds like corpse, corps, horse, and worse. I will keep you, Suzy, busy, Make your head with heat grow dizzy. Tear in eye, your dress will tear. So shall I! Oh hear my prayer. I HAVE, alas!Philosophy,Medicine,Jurisprudencetoo,And to mycost Theology,With ardentlabour,studied through.And hereIstand,with allmy lore,Poor fool,no wiser than before.Magister,doctor styled,indeed, Alreadythese ten years Ilead,Up,down,across,andto andfro,My pupils by the nose,--and learn,Thatwe intruth can nothing know!Thatinmy heartlike fire doth burn. 'Tis true I've more cunningthanallyour dull tribe, Magister and doctor,priest,parson,andscribe;Scrupleor doubtcomes notto enthrallme,Neither can devilnor hellnow appalme-- Hencealsomy heartmustallpleasure forego!Imaynotpretend,aughtrightlyto know,I maynotpretend,throughteaching,to findAmeans to improve or convert mankind.ThenIhave neither goods nor treasure,No worldlyhonour,rank, or pleasure;No dogin such fashionwould longer live!Therefore myselfto magic Igive,In hope,throughspirit-voiceandmight,Secrets nowveiled to bring to light,ThatIno more,with achingbrow,Need speak ofwhatI nothing know;ThatIthe force mayrecogniseThatbinds creation's inmost energies;Her vital powers,her embryo seeds survey,And fling the trade in emptywords away.O full-orb'dmoon,didbutthyrays Their lastuponmine anguish gaze!Besidethis desk,atdeadofnight,Oft have I watched to hail thy light:Then,pensive friend!o'er book andscroll,With soothingpower, thy radiance stole!In thydear light,ah,mightIclimb,Freely,some mountainheightsublime,Round mountaincaves with spirits ride,In thymild haze o'er meadows glide,And,purged from knowledge-fumes,renewMy spirit,in thyhealingdew!Woe's me!stillprison'd inthe gloomOfthis abhorr'd and mustyroom!Where heaven's dear lightitselfdoth pass,But dimlythrough the paintedglass!Hemmedin bybook-heaps,piledaround, Worm-eaten,hid'neath dustand mould,Which to the high vault's topmost bound,Asmoke-stainedpaper dothenfold;With boxes roundtheepiled, and glass,And manya useless instrument,With old ancestral lumber blent-- This is thyworld!a world!alas!And dostthou ask whyheaves thy heart,With tighten'dpressurein thybreast? Whythe dull ache willnot depart,Bywhich thylife-pulseis oppress'd?Instead ofnature's living sphere,Createdfor mankindofold,Bruteskeletons surroundtheehere, And dead men's bones insmoke and mould. Ham. To be, or not to be, that is the Question: Whether 'tis Nobler in the minde to suffer The Slings and Arrowes of outragious Fortune, Or to take Armes against a Sea of troubles, And by opposing end them: to dye, to sleepe No more; and by a sleepe, to say we end The Heart-ake, and the thousand Naturall shockes That Flesh is heyre too? 'Tis a consummation Deuoutly to be wish'd. To dye to sleepe, To sleepe, perchance to Dreame; I, there's the rub, For in that sleepe of death, what dreames may come, When we haue shuffel'd off this mortall coile, Must giue vs pawse. There's the respect That makes Calamity of so long life: For who would beare the Whips and Scornes of time, The Oppressors wrong, the poore mans Contumely, The pangs of dispriz'd Loue, the Lawes delay, The insolence of Office, and the Spurnes That patient merit of the vnworthy takes, When he himselfe might his Quietus make With a bare Bodkin? Who would these Fardles beare To grunt and sweat vnder a weary life, But that the dread of something after death, The vndiscouered Countrey, from whose Borne No Traueller returnes, Puzels the will, And makes vs rather beare those illes we haue, Then flye to others that we know not of. Thus Conscience does make Cowards of vs all, And thus the Natiue hew of Resolution Is sicklied o're, with the pale cast of Thought, And enterprizes of great pith and moment, With this regard their Currants turne away, And loose the name of Action. Soft you now, The faire Ophelia? Nimph, in thy Orizons Be all my sinnes remembred ReduceTaskMapTask <Word>,1 where1 what,1 and,1 the,1 "(Lo)cra" 1 "1490 1 "1498," 1 "35" 1 "40," 1 "A 2 "AS-IS". 2 "A_ " 1 "Absoluti " 1 "Alack " 1 the,1 the,1 the,1 and,1 and,1 and,1 ReduceMap Sort reduce(word, values ...) { count = 0 for each value v in values: count+=v emit(word, count) } map(name, document) { for each word w in document: emitIntermediate(w,1) } Input Output Copy Example: Word count across a number of documents
  6. 6. © Copyright 2010 Hewlett-Packard Development Company, L.P.6 © Copyright 2010 Hewlett-Packard Development Company, L.P.6 HADOOP COMPONENTS – Map Reduce Layer • Provides the map and reduce programming framework • Breaks up Jobs into tasks • Keeps track of execution status – File system Layer • Pluggable file system • Support for location aware file systems • Access through an API Layer • Default is HDFS (Hadoop Distributed File system) • HDFS − Provides fault high availability by replicating individual files − Consists of a central metadata server – NameNode − And a number of Data nodes, which store copies of files (or parts of them)
  7. 7. © Copyright 2010 Hewlett-Packard Development Company, L.P.7 © Copyright 2010 Hewlett-Packard Development Company, L.P.7 HADOOP OPERATION (WITH HDFS) JobTracker MapReduce Layer File system Layer ... Master Disk Data Node Slave Node Disk Data Node Slave Node NameNode TaskTracker TaskTracker
  8. 8. © Copyright 2010 Hewlett-Packard Development Company, L.P.8 © Copyright 2010 Hewlett-Packard Development Company, L.P.8 HADOOP OPERATION (WITH HDFS) JobTracker MapReduce Layer File system Layer ... Master Disk Data Node TaskTracker Slave Node Disk Data Node TaskTracker Slave Node Job Scheduler NameNode
  9. 9. © Copyright 2010 Hewlett-Packard Development Company, L.P.9 © Copyright 2010 Hewlett-Packard Development Company, L.P.9 HADOOP OPERATION (WITH HDFS) JobTracker MapReduce Layer File system Layer ... Master Disk Data Node TaskTracker Slave Node Disk Data Node TaskTracker Slave Node Job Scheduler Location Information NameNode
  10. 10. © Copyright 2010 Hewlett-Packard Development Company, L.P.10 © Copyright 2010 Hewlett-Packard Development Company, L.P.10 TaskTask Task HADOOP OPERATION (WITH HDFS) JobTracker MapReduce Layer File system Layer ... Master Disk Data Node TaskTracker Slave Node Disk Data Node TaskTracker Slave Node Job Scheduler Location Information NameNode
  11. 11. © Copyright 2010 Hewlett-Packard Development Company, L.P.11 © Copyright 2010 Hewlett-Packard Development Company, L.P.11 FAILURE SCENARIOS Failure in Map Reduce components – TaskTracker • Sends heartbeat to JobTracker • If unresponsive for x seconds, JobTracker marks TaskTracker as dead and stop assigning work to it - blacklisting • Scheduler reschedules tasks running on that TaskTracker – JobTracker • No build in heartbeat mechanism • Checkpoints to file system • Can be restarted and resumes operation – Individual Tasks • TaskTracker monitors progress • Can restart failed Tasks • Complex failure handling − E.g. skip parts of input data which produces failure
  12. 12. © Copyright 2010 Hewlett-Packard Development Company, L.P.12 © Copyright 2010 Hewlett-Packard Development Company, L.P.12 FAILURE SCENARIOS (2) Failure of Data storage – Pluggable file system implementation needs to detect and remedy error scenarios – HDFS • Failure of Data Node − Keeps track of replication count for files (chunks/parts of files) − Can re-replicate missing pieces − Tries to place copies of individual physically apart from each other • Same rack vs. different racks • Failure of NameNode − Operations are written to logs, makes restart possible • During restart the file system is in read only mode − A secondary NameNode can periodically read these logs, to speed up time to become available − BUT If secondary NameNode takes over, restart of the whole cluster is needed, since assigned hostnames have changed.
  13. 13. © Copyright 2010 Hewlett-Packard Development Company, L.P.13 © Copyright 2010 Hewlett-Packard Development Company, L.P.13 AVAILABILITY TAKEAWAY – Map reduce Layer • Checkpoints to the persisting file system to resume work • TaskTracker − Can be restarted • JobTracker − Can be restarted – HDFS • Single point of failure is the NameNode − Restarts can take a long time, depending on amount of data stored and number of operations in the log itself − In the regions of hours
  14. 14. © Copyright 2010 Hewlett-Packard Development Company, L.P.14 © Copyright 2010 Hewlett-Packard Development Company, L.P.14 A DIFFERENT FILE SYSTEM – HP IBRIX – Software solution which runs on top of storage configurations – Fault tolerant, high availability file system • Assumes RAIDed storage for fault tolerance – Segmented File system • Disks (Luns) are treated as Segments – Segments are managed by Segment servers • Aggregated into global file system(s) • File system provide single namespace • Each file system supports up to 16 Petabyte
  15. 15. © Copyright 2010 Hewlett-Packard Development Company, L.P.15 © Copyright 2010 Hewlett-Packard Development Company, L.P.15 IBRIX IN A NUTSHELL Fusion Manager Client Client Client Client NFS, CIFS or native client Disk Disk Segment Server ... ... Performance Capacity increase No single metadata server / segmented file system Disk Disk Segment Server ...
  16. 16. © Copyright 2010 Hewlett-Packard Development Company, L.P.16 © Copyright 2010 Hewlett-Packard Development Company, L.P.16 HOW DOES IT LOOK LIKE – Fusion Manager Web Console • Also command line interface for programmatic configuration – Global management view of the installation • Here segments correspond to disks attached to servers
  17. 17. © Copyright 2010 Hewlett-Packard Development Company, L.P.17 © Copyright 2010 Hewlett-Packard Development Company, L.P.17 HOW DOES IT LOOK LIKE (2) – A client mounts the file system via: • NFS • CIFS / Samba • Native Client • Each segment server is automatically a client – Mount points and exports need to be created firsts • on the fusion manager • Access control enforced on a per file system level – Clients access file system via “normal” file system calls
  18. 18. © Copyright 2010 Hewlett-Packard Development Company, L.P.18 © Copyright 2010 Hewlett-Packard Development Company, L.P.18 HARDWARE CONFIGURATION – Supports failover – Different hardware topologies configurations – Couplet configuration • Best suited for hybrid of performance and capacity RAID server server RAID server server RAID server server Single Namespace ...
  19. 19. © Copyright 2010 Hewlett-Packard Development Company, L.P.19 © Copyright 2010 Hewlett-Packard Development Company, L.P.19 Location aware Hadoop on IBRIX
  20. 20. © Copyright 2010 Hewlett-Packard Development Company, L.P.20 © Copyright 2010 Hewlett-Packard Development Company, L.P.20 TaskTask Task HADOOP INTERNALS – WITH IBRIX JobTracker MapReduce Layer File system Layer ... Master Disk Segment Server TaskTracker Slave Node Disk Segment Server TaskTracker Slave Node Job Scheduler Location Information IBRIX Client
  21. 21. © Copyright 2010 Hewlett-Packard Development Company, L.P.21 © Copyright 2010 Hewlett-Packard Development Company, L.P.21 PERFORMANCE TEST – Small test with • 1 GB of randomly generated data, spread across 10 input files • In a virtualized environment – Test • Sort the records • measure job execution - Includes mapping, sorting and reducing time – Vary the number of slave nodes – File access test – no real performance test • Actual computation on each TaskTracker is low • Governing factors for execution time are − Time to read and write files − Time to distribute data to the reducers
  22. 22. © Copyright 2010 Hewlett-Packard Development Company, L.P.22 © Copyright 2010 Hewlett-Packard Development Company, L.P.22 PERFORMANCE RESULTS 0 50 100 150 200 250 300 350 400 450 500 2 3 4 5 6 7 8 9 10 Number of Slaves HDFS Plain IBRIX Location aware IBRIX executionTimeofsortjob(sec)
  23. 23. © Copyright 2010 Hewlett-Packard Development Company, L.P.23 © Copyright 2010 Hewlett-Packard Development Company, L.P.23 PERFORMANCE RESULTS – Comparable performance to native HDFS system • For smaller workload even increased performance – due to no replication • Goal is not to be faster, but to provide the same capabilities as HDFS and add − Security − Fault Tolerance – Can take advantage of location information – Is dependent on distribution and type of input data • Files need to be evenly spread across the segment servers • Prefers many small(er) file, since they can be distributed better
  24. 24. © Copyright 2010 Hewlett-Packard Development Company, L.P.24 © Copyright 2010 Hewlett-Packard Development Company, L.P.24 FEATURE HDFS IBRIX FURTHER FEATURE COMPARISON Single Point of Failure Needs RAID Can expose location information Individual file replication Respond to node failure Homogenous file system Split each file across data nodes Yes, NameNode No, replicates Yes Yes Re-Replication mark as dead Yes Yes - files are split into chunks which are distributed individually No Yes Yes No, only configured by file systems path Failover mark as dead, can fallback No, can define Tiers Only if a segment is full
  25. 25. © Copyright 2010 Hewlett-Packard Development Company, L.P.25 © Copyright 2010 Hewlett-Packard Development Company, L.P.25 SUMMARY
  26. 26. © Copyright 2010 Hewlett-Packard Development Company, L.P.26 © Copyright 2010 Hewlett-Packard Development Company, L.P.26 SUMMARY – Hadoop provides a number of failure handling methods • Dependent on persistent file system – IBRIX as alternative file system • Not specifically build for Hadoop − Integration using a light weight file system plug-in for Hadoop • Location aware design enables computation close to the data • Comparable performance while gaining on fault tolerance − no single point of failure • Reduced storage requirement • Storage not exclusive to Hadoop • Build in POSIX security model • Fault tolerant handled by hardware – Future work • Short lived Hadoop Cluster
  27. 27. © Copyright 2010 Hewlett-Packard Development Company, L.P.27 © Copyright 2010 Hewlett-Packard Development Company, L.P.27 Q&A
  28. 28. © Copyright 2010 Hewlett-Packard Development Company, L.P.28 © Copyright 2010 Hewlett-Packard Development Company, L.P.28 BACKUP
  29. 29. © Copyright 2010 Hewlett-Packard Development Company, L.P.29 © Copyright 2010 Hewlett-Packard Development Company, L.P.29 IBRIX DETAILS – IBRIX uses iNodes as backend store – Extends them by a file-based globally unique identifier – Each Segment server is responsible for a fixed number of iNodes) • Determined by blocksize within that segment and overall size • Example − 4 GB segment size, 4kb block size  1,048,576 iNodes (1M) • Simplified calculation example − Where is iNode 1,800,000 − divide by 1M ≈ 1.71  lives on segment server 1 – iNodes do not store the data but have a reference to the actual data • Backend storage for iBrix is ext3 filesystem
  30. 30. © Copyright 2010 Hewlett-Packard Development Company, L.P.30 © Copyright 2010 Hewlett-Packard Development Company, L.P.30 MORE DETAILS – Based on distributed iNodes Segment 1st iNode Nth iNode Disk local file system
  31. 31. © Copyright 2010 Hewlett-Packard Development Company, L.P.31 © Copyright 2010 Hewlett-Packard Development Company, L.P.31 SECURITY – File system respects POSIX like interface • Files belong to user/group and have read/write/execute flags – Native Client • Needs to be bound to a Fusion Manager • Export control can be enforced − Mounting only possible from the Fusion manager console – CIFS / Samba • Requires Active Directory to translate windows ids to Linux id • Export only sub path of the file system (e.g. /filesystem/sambadirectory) – NFS • Create exports on Segment server • Limit clients by IP Mask • Export only sub path of the file system (e.g. /filesystem/nfsdirectory) • Normal NFS properties (read/write/root squash)
  32. 32. © Copyright 2010 Hewlett-Packard Development Company, L.P.32 © Copyright 2010 Hewlett-Packard Development Company, L.P.32 FEATURES – Multiple logical file systems • Select different segments as base for them – Task Manager / Policy • Rebalancing between different segment servers • Tiering of data − Some segments could be better/worse than others − Move data to from them based on policy/rule • Replicate complete logical file systems - Replicate to remote cluster – Failover • Buddy system of two (or more) segment servers (active/active standby) • Native clients will automatically failover – Growing • Segment servers register with Fusion Manager • New segments (discs) need to be programmatically discovered − Can be added to logical file systems – Is location aware • By nature of design • For each file, the segment server(s) holding the file can be determined
  33. 33. © Copyright 2010 Hewlett-Packard Development Company, L.P.33 © Copyright 2010 Hewlett-Packard Development Company, L.P.33 FEATURES (2) – De-duplication – Caching • On segment server owning a particular file – Distributed Metadata • No single point of failure – Supports snapshotting of whole file systems • Creates a new virtual file system – Policy for storing new files • Distribute them randomly across segment servers • Assign them to the “local” segment server – Speeds up Reduce Tasks – Separate data network • Allows to configure the network interface to use for storage communication

×