Next Generation Hadoop Operations
Upcoming SlideShare
Loading in...5
×
 

Next Generation Hadoop Operations

on

  • 5,459 views

Andrew Ryan describes how Facebook operates Hadoop to provide access as a shared resource between groups. ...

Andrew Ryan describes how Facebook operates Hadoop to provide access as a shared resource between groups.

More information and video at:
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/hug-feb-2011-recap/

Statistics

Views

Total Views
5,459
Views on SlideShare
5,410
Embed Views
49

Actions

Likes
11
Downloads
145
Comments
0

21 Embeds 49

http://www.mongodb.org 12
http://static.slidesharecdn.com 7
http://www.slideshare.net 5
http://paper.li 4
http://www.onlydoo.com 3
http://www.saujana.sg 2
http://pibatef.com.br 2
http://www.ewavemd.com 1
http://gbs-moodle.glenbrook225.org 1
http://existentialpsychotherapy.net 1
http://translate.googleusercontent.com 1
http://www.cs.nott.ac.uk 1
http://www.magnolia-cms.com 1
http://notch8.com 1
http://users.sch.gr 1
http://elhumordellicenciado-tomo1.blogspot.com 1
http://www.2g-cenergy.com 1
http://www.waterandstone.com 1
http://repetidoralabreguara.blogspot.com 1
http://twitter.com 1
http://preacherspen.org 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Next Generation Hadoop Operations Next Generation Hadoop Operations Presentation Transcript

  • ext-Generation Hadoopperationshat’s ahead in the next 12 monthsr Hadoop cluster administrationw Ryanps Engineer6, 2011
  • genda1 Hadoop operations @Facebook: an overview2 Existing operational best practices3 The challenges ahead: new directions in Hadoop4 Emerging operational best practices5 Conclusions and next steps
  • adoop Operations @Facebookan staffing, fast moving, highly leveraged sic oncall structure: evel 1: 24x7 sysadmin team (“SRO”) for whole site evel 2: 2 people (“AppOps”) trading 1-week oncall shifts evel 3: 4 different Hadoop dev subteams with 1-week rotationsPlus oncalls from other adjunct teams: SiteOps for machine repairNetEng for network, etc.ery engineer @FB is issued a cell phone and expected to beailable in emergencies and/or if they make a change to a productstem or code.
  • perational gaps in Hadoopr best practices address all these gaps rdware selection, preparation, and configurationstallation/packaging gradestostart/start/stop/restart/status as correct UNIX user de level application and system monitoringuster-level and job-level monitoringegrated log viewing/tailing/greppingst, reliable, centrally logged cluster-level shell ( != slaves.sh)
  • isting operational best practices (1)sadmin the stuff you would do for a large distributed system but especia iled/failing hardware is your biggest enemy. FIND IT AND FIX IT,ET IT OUT OF YOUR CLUSTERS! (the ‘excludes’ file is your frien gularly run every possible diagnostic to safely scan for bad hardwentify and remove “repeat offender” hardwareil fast, recover quickly, small things add up in big clusters:RHEL/Centos kickstart steals your disk space (1.5%-3%+ per diskNo swap + vm.panic_on_oom=1 + kernel.kdb=0 for “fast auto reboOOM”
  • sadmin examplentifying your “America’s Most Wanted” pays off
  • isting operational best practices (2)olingaintain a central registry of clusters, nodes, and each node’s role ster, integrated with your service/asset management platform ild centrally maintained tools to:Start/stop/restart/autostart daemons on hosts (hadoopctl)View/grep/tail daemon logs on hosts (hadooplog)Start/stop, or execute commands on entire clusters (clusterctl)Manage excludes files based on repair status (excluderator)Deploy any arbitrary version of software to clustersMonitor daemon health and collect statistics
  • oling exampleploy & upgrade clusterseploy an HDFS/MapReduce cluster pair: 2 to 4000 nodes via torreploy-hadoop-release.py --clusterdeploy=DFS1,SILVER branch@usterctl restart DFS1 SILVER efresh deploy” on 10 clusters, and then restart just the datanodeeploy-hadoop-release.py –poddeploy=DFSSCRIBE-ALL redeployusterctl restart DFSSCRIBE-ALL:datanode
  • isting operational best practices (3)cesscument everythinggregate different classes of users on different clusters, with approrvice levels and capacitiesaph user-visible metrics like HDFS and job latencyild “least destructive” procedures for getting hardware back in sevelopers and Ops should use the same procedures and tools
  • ocess exampleaphing our users’ experience on the cluster
  • Hadoop cluster admin’s worst enemies“X-Files”: machines which fail in strange ways, undetected by yoitoring systemset your basics under control, then you’ll have more time for theseerica’s Most Wanted”: machines which keep failing, again and agur data: 1% of our machines accounted for 30% of our repair ticke
  • ew directions for Hadoopase (Facebook Messages, real-time click logs)o-downtime upgrades (AvatarNode, rolling upgrades) gadatanodes” and Hadoop RAIDFS as an “appliance”e also: ://www.facebook.com/notes/facebook-engineering/looking-at-the e-behind-our-three-uses-of-apache-hadoop/468211193919
  • base and Hadoopy new technology with emerging operational characteristicsplications using Hbase are also new, with their own usage quirks ing for large number of small clusters (~100 nodes)w/dead nodes are a big problem: these are real-time, user facinggion failover slow ; no speculative execution-downtime restarts must be avoided the Messages tech talk here: http://fb.me/95OQ8YaD2rkb3r
  • ro-downtime upgradesFS upgrades are 1-2 hours of downtimetracker upgrades are quick (5 min), but kill all currently running joing upgrades work today, but are too slow for large clustersst be able to be strict and lenient about multiple versions of clientver software installed and running in the cluster
  • egadatanodes” and Hadoop RAIDrage requirements continue to increase rapidly, as does CPU/RA ncrease in datanode density from 2009-2011 (4TB->36TB)doop RAID with XOR and Reed-Solomon bring tremendous cost ings along with management challenges:osing one node is a big deal (200k-600k blocks/node?). A rack?uch!ools and admin capabilities are not ready yet HDFS administration in 2012 be “like administering a cluster of 4apps”?st/rack level network will be a bottleneck
  • DFS as an “appliance”e HDFS cluster instead of commercial storage applianceequires commercial-grade support & featuresust be price-competitive vs.
  • merging operational best practicese careful selection of hardware and network designs toommodate new uses of Hadoopd and deal with slowness at a node/rack/segment levelo-healing at granularity better than “reboot” or “restart”de-level version detection and installation ing, zero-downtime upgrades (AvatarNode + new JobTracker)d do all this without making Hadoop any harder to set up and run
  • ext steps we trying to do too much?acebook needs an enormous data warehouseacebook needs a large distributed filesystemacebook needs a database alternative to MySQLacebook always looking to spend less money and all that other stuff tooure is not an optionver a dull moment!
  • (c) 2009 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0