Hadoop @ Yahoo! - Internet Scale Data Processing

Hadoop at Yahoo! Eric Baldeschwieler VP Hadoop Software Development Yahoo!

Agenda
Hadoop 101
Hadoop and Yahoo!
Yahoo! usage of Hadoop
Hadoop Improvements

Introduction to Hadoop
Hadoop
Apache.org Open Source project
In use at Yahoo! since 2006
Hadoop Core includes
Distributed File System
Distributed Processing Framework
Map-Reduce paradigm
Runs on
Linux, Mac OS/X, Solaris and Windows
Written in Java
Large clusters of commodity hardware

MapReduce
cat input | grep | sort | uniq -c | cat

Hadoop and Yahoo!

What we want you to know about Yahoo! and Hadoop
Yahoo! is:
The largest user of Hadoop
The largest tester of Hadoop
The largest contributor to Hadoop
Also:
We release “The Yahoo! Distribution of Hadoop”
We contribute all of our Hadoop work to the Apache Foundation as open source
We continue to aggressively invest in Hadoop
We do not sell Hadoop service or support!
We use Hadoop services to run Yahoo!

The Largest Hadoop User 2006 now Hardware Internal Hadoop Users building new datacenter PB Disk, >82PB Today Nodes, >25,000 Today

The Largest Hadoop Tester
Every release of The Yahoo! Distribution of Hadoop goes through multiple levels of testing before we declare it stable
4 Tiers of Hadoop clusters
Development, Testing and QA (~10% of our hardware)
Continuous integration / testing of new code
Proof of Concepts and Ad-Hoc work (~10% of our hardware)
Runs the latest version of Hadoop
Science and Research (~60% of our hardware)
Runs more stable versions
Production (~20% of our hardware)
The most stable version of Hadoop
We continue to grow our Quality Engineering team
We are hiring!!

The majority of all patches to Hadoop have come from Yahoo!s
72% of core patches
Core = HDFS, Map-reduce, Common
Metric is an underestimate
Some Yahoo!s use apache.org accounts
Full time contributors submit larger patches
Yahoo! is the largest employer of Hadoop Contributors, by far!
We contribute ALL of our Hadoop development work back to Apache
Team: Sunnyvale, Bangalore, Beijing
http://careers.yahoo.com
The Largest Hadoop Contributor Core Patches

Usage of Hadoop

Why Hadoop @ Yahoo! ?
Massive scale
500M+ unique users per month
Billions of “transactions” per day
Many petabytes of data
Analysis and data processing key to our business
Need to process all of that data in a timely manner
Lots of ad hoc investigation to look for patterns, run reports…
Need to do this cost effectively
Use low cost commodity hardware
Share resources among multiple projects
Try new things quickly at huge scale
Handle the failure of some of that hardware every day
The Hadoop infrastructure provides these capabilities

Yahoo! front page - Case Study

Yahoo! front page - Case Study Ads Optimization Search Index

Yahoo! front page - Case Study Ads Optimization Content Optimization Search Index Machine Learned Spam filters Content Management Content Optimization

Large Applications 2008 2009 Webmap ~70 hours runtime ~300 TB shuffling ~200 TB output ~73 hours runtime ~490 TB shuffling ~280 TB output +55% Hardware Sort benchmarks (Jim Gray contest)
1 Terabyte sorted
209 seconds
900 nodes
1 Terabyte sorted
62 seconds, 1500 nodes 1 Petabyte sorted
16.25 hours, 3700 nodes
Largest cluster
2000 nodes
6PB raw disk
16TB of RAM
16K Cores
4000 nodes
16PB raw disk
64TB of RAM
32K Cores
(40% faster too!)

Makes Developers & Scientists more productive
Research questions answered in days, not months
Projects move from research to production easily
Easy to learn! “Even our rocket scientists use it!”
The major factors
You don’t need to find new hardware to experiment
You can work with all your data!
Production and research based on same framework
No need for R&D to do I.T., the clusters just work
Develop in high level languages (java, python…, Pig)
Tremendous Impact on Productivity

Example: Search Assist TM
Database for Search Assist™ is built using Hadoop.
3 years of log-data
20-steps of map-reduce
Before Hadoop After Hadoop Time 26 days 20 minutes Language C++ Python Development Time 2-3 weeks 2-3 days

Pig – Making Hadoop Easy! 1/20 the lines of code 1/16 the development time Performance within 2x

Pig – Making Hadoop Easy! Users = load ‘users’ as (name, age); Fltrd = filter Users by age >= 18 and age <= 25; Pages = load ‘pages’ as (user, url); Jnd = join Fltrd by name, Pages by user; Grpd = group Jnd by url; Smmd = foreach Grpd generate group, COUNT(Jnd) as clicks; Srtd = order Smmd by clicks desc; Top5 = limit Srtd 5; store Top5 into ‘top5sites’ ;

Hadoop Improvements

Current Yahoo! Development
Hadoop
Backwards compatibility
New APIs in 21, Avro
Append, Sync, Flush
Improved Scheduling
Capacity Scheduler, Hierarchical Queues in 21
Security - Kerberos authentication
Performance - GridMix3 / Scheduler Simulator / Better trace & log analysis
PIG
SQL and Metadata
Column oriented storage access layer - Zebra
Multi-query, lots of other optimizations
Oozie
New workflow and scheduling system
Quality Engineering – Many more tests
Stress tests, Fault injection, Functional test coverage, performance regression…

Questions? Eric Baldeschwieler VP Hadoop Software Development Yahoo! For more information: http://hadoop.apache.org/ http://hadoop.yahoo.com/ (including job openings)

Hadoop @ Yahoo! - Internet Scale Data Processing

by Yahoo! Developer Network on Nov 05, 2009

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 36

Statistics

Hadoop @ Yahoo! - Internet Scale Data Processing Presentation Transcript

http://www.slideshare.net	34
http://www.schoox.com	2