Hadoop Operations at LinkedIn
Upcoming SlideShare
Loading in...5
×
 

Hadoop Operations at LinkedIn

on

  • 8,035 views

This is the slide deck for the talk I gave at SouthWest Big Data (UK) and Hadoop Summit EMEA 2013.

This is the slide deck for the talk I gave at SouthWest Big Data (UK) and Hadoop Summit EMEA 2013.

Video is at: http://www.youtube.com/watch?v=Hw-V7-T3GmE

Statistics

Views

Total Views
8,035
Views on SlideShare
6,870
Embed Views
1,165

Actions

Likes
25
Downloads
127
Comments
0

11 Embeds 1,165

http://eventifier.co 458
http://www.scoop.it 321
https://twitter.com 196
http://eventifier.com 155
http://www.linkedin.com 15
https://www.linkedin.com 11
http://tweetedtimes.com 5
http://webcache.googleusercontent.com 1
http://flask.radcool.co 1
https://www.google.com 1
http://www.google.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Goals: - Fix the performance - Make the system operationally sound
  • Goals: - Corporate decision to switch to Linux - Start prep for security
  • we use cobbler to control our kickstart installs. key features: * template engine * snippet system * RPM repo sync * both command line and programmable APIs * and, most importantly, great support for a “netboot always” environment. This means that we always have our hosts boot from the network and, if that fails, local disk. We generally always re-install the machine after a disk failure so that we can start it from a clean slate, cleaning any excess cruft and restoring any host specific parts like Kerberos keytabs. What may be surprising is that our kickstart environment serves primarily to do three things: * partition disks * get enough of the OS installed to troubleshoot a broken kickstart * bootstrap our configuration management tool
  • Born out of the HPC community in 2004 Python BSD License Love the community Works with everything, not just the Hadoop ecosystem Services based methodology with conflict resolution Awesome reporting engine
  • Goals: - Deploy secure Hadoop - Reduce user friction
  • A talk in and of itself Highlights: - another cultural shift - finding many bugs in what was considered stable code - forking the kerberos web filter due to poor code quality
  • Goals: - What do we need for the next 4 years?

Hadoop Operations at LinkedIn Hadoop Operations at LinkedIn Presentation Transcript

  • Grid Operations Hadoop Operations at LinkedIn Allen Wittenauer Grid Computing Architect ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • “Hadoop is not a developer problem; it’s an operations problem.” -- Hadoop vendor ex-employee ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • § August 2009 – 20 Nodes in 1 grid – Apache Hadoop 0.20.0 – No configuration management – No monitoring – No security – Free for all, including random mafia hits on running jobs – FIFO Scheduling – ~20 users – 20 tasks per node – Solaris – No operational support ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • How We Fixed This (In Chronological Order) ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • Year One ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • § Dropped task count – 10 mappers => 7 mappers – 10 reducers => 5 reducers § Reworked ETL – hourlies => dailies – Re-ordered to take advantage of compression § 10x storage improvement – Sample impact on one job (not workflow!): § 80,000 map tasks => 2,000 map tasks § Run time cut in half § Optimize work flows/culture shift § More task time, less tasks § Production review to reinforce good behavio(u)r ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • § Switched to Capacity Scheduler 5% ETL Tasks – FIFO is terrible 15% Fast Queue: – Fair Share only viable for small tasks - Task Time < 15 Minutes - Job Time < 1 Hour – Enforced SLAs via custom patch - Slot stealing from "Slow" Queue § Submitted Jar Size Limit 80% Slow Queue: – Encourage distributed cache usage - Job Time < 24 Hours – Enforced limit via custom patch - Up to 80% of slots ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • § Benchmarking – Use production code not TeraSort! Old Node: New Node: - 2 Rack Units - 1 Rack Unit - 2 CPUs - 2 CPUs - 16 GB - 24 or 32 GB - 8 x 1 TB SATA - 6 x 2 TB SATA - 1 x 2 gb NIC - 1 x 1 gb NIC § Cut cost per unit in half § 2x nodes per rack § Extra RAM – buffering – bus speed ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • Year Two ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • § DataNode disk partitioning – Separate file systems for different purposes 20 GB 200 GB HDFS /, ... MR ... 5GB 200 GB HDFS Swap MR – Mount options: noatime, commit=30, data=writeback § NN, JT, etc – No “special hardware” == use SW RAID ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • LDAP Master Multi LDAP Master + Master + Replication KDC Master KDC LDAP/KDC LDAP/KDC Slaves Slaves username, uid username, uid group name, gid group name, gid netgroup, sudoers netgroup, sudoers nscd nscd Client Node Client Node ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • Host bcfg2 Server Group1, Group2, ... Group1 -> Svc1, Svc2, ... bcfg2 Group2 -> Svc1, Svc3, ... client Svc1+ Group3 -> Svc4, Svc5, ... Svc2+ Svc3 Content § Service Bundle – RPMs, config files, etc – Conflict resolution ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • § Different RPM names + different install locations = pre-deploy-ability: Object RPM Name File Path Hadoop 1.0.4-p3 Binaries hadoop-1043-bin-1.0.4-3 /dir/hadoop-1.0.4-p3 Grid Config for 1.0.4-p3 gridname-1043- /dir/grid-conf-1.0.4-p3 hadoopconf-1.0.4.3-1 Hadoop 1.1.2-p1 Binaries hadoop-1121-bin-1.1.2.1-1 /dir/hadoop-1.1.2-p1 Grid Config for 1.1.2-p1 gridname-1043- /dir/grid-conf-1.1.2-p1 hadoopconf-1.0.4.3-1 ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • Year Three+ ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • Corp IT Grid Realm Active Directory krbtgt/GRID@CORP @GRID @CORP Password krbtgt/host@GRID krbtgt/service@GRID krbtgt/user@CORP Hadoop krbtgt/GRID@CORP Services ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • Many months moving to secure Apache Hadoop... ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • § March 2013 – 5000 Nodes in ~10 grids – Apache Hadoop 1.0.4 + custom patches – Full configuration management – Full monitoring – Security – Capacity scheduler with SLA – ~700 users – 12 tasks per node – Linux – Five dedicated operations staff members ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • Future Work ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • Is ‘pure Hadoop’ the right tool for all of our workloads? ©2013 LinkedIn Corporation. All Rights Reserved.Wednesday, March 20, 13
  • YARN PBS H D F S C E P H ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13
  • ©2013 LinkedIn Corporation. All Rights Reserved. BUSINESS OPERATIONSWednesday, March 20, 13
  • § More on LinkedIn Hadoop Performance: – http://www.slideshare.net/allenwittenauer/2012-lihadoopperf § LinkedIn Data Analytics: – http://data.linkedin.com/ ©2013 LinkedIn Corporation. All Rights Reserved. GRID OPERATIONSWednesday, March 20, 13