Yahoo! has been utilizing and contributing to the Apache Hadoop project since 2006, establishing itself as the largest user, tester, and contributor of Hadoop. The platform supports massive data processing needs for Yahoo!'s services, leveraging commodity hardware and a robust testing infrastructure to ensure stability and performance. Yahoo! continues to invest aggressively in Hadoop developments while maintaining an open-source commitment.

1. Hadoop at Yahoo! Eric Baldeschwieler VP Hadoop Software Development Yahoo!

2. Agenda Hadoop 101 Hadoop and Yahoo! Yahoo! usage of Hadoop Hadoop Improvements

3. Introduction to Hadoop Hadoop Apache.org Open Source project In use at Yahoo! since 2006 Hadoop Core includes Distributed File System Distributed Processing Framework Map-Reduce paradigm Runs on Linux, Mac OS/X, Solaris and Windows Written in Java Large clusters of commodity hardware

4. MapReduce cat input | grep | sort | uniq -c | cat

5. Hadoop and Yahoo!

6. What we want you to know about Yahoo! and Hadoop Yahoo! is: The largest user of Hadoop The largest tester of Hadoop The largest contributor to Hadoop Also: We release “The Yahoo! Distribution of Hadoop” We contribute all of our Hadoop work to the Apache Foundation as open source We continue to aggressively invest in Hadoop We do not sell Hadoop service or support! We use Hadoop services to run Yahoo!

7. The Largest Hadoop User 2006 now Hardware Internal Hadoop Users building new datacenter PB Disk, >82PB Today Nodes, >25,000 Today

8. The Largest Hadoop Tester Every release of The Yahoo! Distribution of Hadoop goes through multiple levels of testing before we declare it stable 4 Tiers of Hadoop clusters Development, Testing and QA (~10% of our hardware) Continuous integration / testing of new code Proof of Concepts and Ad-Hoc work (~10% of our hardware) Runs the latest version of Hadoop Science and Research (~60% of our hardware) Runs more stable versions Production (~20% of our hardware) The most stable version of Hadoop We continue to grow our Quality Engineering team We are hiring!!

9. The majority of all patches to Hadoop have come from Yahoo!s 72% of core patches Core = HDFS, Map-reduce, Common Metric is an underestimate Some Yahoo!s use apache.org accounts Full time contributors submit larger patches Yahoo! is the largest employer of Hadoop Contributors, by far! We contribute ALL of our Hadoop development work back to Apache Team: Sunnyvale, Bangalore, Beijing http://careers.yahoo.com The Largest Hadoop Contributor Core Patches

10. Usage of Hadoop

11. Why Hadoop @ Yahoo! ? Massive scale 500M+ unique users per month Billions of “transactions” per day Many petabytes of data Analysis and data processing key to our business Need to process all of that data in a timely manner Lots of ad hoc investigation to look for patterns, run reports… Need to do this cost effectively Use low cost commodity hardware Share resources among multiple projects Try new things quickly at huge scale Handle the failure of some of that hardware every day The Hadoop infrastructure provides these capabilities

12. Yahoo! front page - Case Study

13. Yahoo! front page - Case Study Ads Optimization Search Index

14. Yahoo! front page - Case Study Ads Optimization Content Optimization Search Index Machine Learned Spam filters Content Management Content Optimization

15. Large Applications 2008 2009 Webmap ~70 hours runtime ~300 TB shuffling ~200 TB output ~73 hours runtime ~490 TB shuffling ~280 TB output +55% Hardware Sort benchmarks (Jim Gray contest) 1 Terabyte sorted 209 seconds 900 nodes 1 Terabyte sorted 62 seconds, 1500 nodes 1 Petabyte sorted 16.25 hours, 3700 nodes Largest cluster 2000 nodes 6PB raw disk 16TB of RAM 16K Cores 4000 nodes 16PB raw disk 64TB of RAM 32K Cores (40% faster too!)

16. Makes Developers & Scientists more productive Research questions answered in days, not months Projects move from research to production easily Easy to learn! “Even our rocket scientists use it!” The major factors You don’t need to find new hardware to experiment You can work with all your data! Production and research based on same framework No need for R&D to do I.T., the clusters just work Develop in high level languages (java, python…, Pig) Tremendous Impact on Productivity

17. Example: Search Assist TM Database for Search Assist™ is built using Hadoop. 3 years of log-data 20-steps of map-reduce Before Hadoop After Hadoop Time 26 days 20 minutes Language C++ Python Development Time 2-3 weeks 2-3 days

18. Pig – Making Hadoop Easy! 1/20 the lines of code 1/16 the development time Performance within 2x

19. Pig – Making Hadoop Easy! Users = load ‘users’ as (name, age); Fltrd = filter Users by age >= 18 and age <= 25; Pages = load ‘pages’ as (user, url); Jnd = join Fltrd by name, Pages by user; Grpd = group Jnd by url; Smmd = foreach Grpd generate group, COUNT(Jnd) as clicks; Srtd = order Smmd by clicks desc; Top5 = limit Srtd 5; store Top5 into ‘top5sites’ ;

20. Hadoop Improvements

21. Current Yahoo! Development Hadoop Backwards compatibility New APIs in 21, Avro Append, Sync, Flush Improved Scheduling Capacity Scheduler, Hierarchical Queues in 21 Security - Kerberos authentication Performance - GridMix3 / Scheduler Simulator / Better trace & log analysis PIG SQL and Metadata Column oriented storage access layer - Zebra Multi-query, lots of other optimizations Oozie New workflow and scheduling system Quality Engineering – Many more tests Stress tests, Fault injection, Functional test coverage, performance regression…

22. Questions? Eric Baldeschwieler VP Hadoop Software Development Yahoo! For more information: http://hadoop.apache.org/ http://hadoop.yahoo.com/ (including job openings)