Apache Hadoop India Summit 2011 talk "Pig - Making Hadoop Easy" by Alan Gate

Alan F. Gates
Yahoo!
Pig, Making Hadoop Easy

Who Am I?
Pig committer and PMC Member
An architect in Yahoo! grid team
Photo credit: Steven Guarnaccia, The Three Little Pigs

Motivation By Example
You have web server logs of purchases on your site. You want to find the 10 users who bought the most and the cities they live in. You also want to know what percentage of purchases they account for in those cities.
Load Logs
Find top 10 users
Sum purchases by city
Join by city
Store top 10 users
Calculate percentage
Store results

In Pig Latin
raw = load 'logs' as (name, city, purchase);
-- Find top 10 users
usrgrp = group raw by (name, city);
byusr = foreach usrgrp generate group as k1,
SUM(raw.purchase) as utotal;
srtusr = order byusr by usrtotal desc;
topusrs = limit srtusr 10;
store topusrs into 'top_users';
-- Count purchases per city
citygrp = group raw by city;
bycity = foreach citygrp generate group as k2,
SUM(raw.purchase) as ctotal;
-- Join top users back to city
jnd = join topusrs by k1.city, bycity by k2;
pct = foreach jnd generate k1.name, k1.city, utotal/ctotal;
store pct into 'top_users_pct_of_city';

Translates to Four MapReduce Jobs

Performance

Where Do Pigs Live?
Data Factory
Pig
Pipelines
Iterative Processing
Research
Data Warehouse
BI Tools
Analysis
Data Collection

Pig Highlights
Language designed to enable efficient description of data flow
Standard relational operators built in
User defined functions (UDFs) can be written for column transformation (TOUPPER), or aggregation (SUM)
UDFs can be written to take advantage of the combiner
Four join implementations built in: hash, fragment-replicate, merge, skewed
Multi-query: Pig will combine certain types of operations together in a single pipeline to reduce the number of times data is scanned
Order by provides total ordering across reducers in a balanced way
Writing load and store functions is easy once an InputFormat and OutputFormat exist
Piggybank, a collection of user contributed UDFs

Multi-store script
A = load ‘users’ as (name, age, gender, city, state);
B = filter A by name is not null;
C1 = group B by age, gender;
D1 = foreach C1 generate group, COUNT(B);
store D into ‘bydemo’;
C2= group B by state;
D2 = foreach C2 generate group, COUNT(B);
store D2 into ‘bystate’;
group by age, gender
store into ‘bydemo’
apply UDFs
load users
filter nulls
group by state
store into ‘bystate’
apply UDFs

Multi-Store Map-Reduce Plan
map
filter
split
local rearrange
local rearrange
reduce
multiplex
package
package
foreach
foreach

Hash Join
Users = load‘users’as (name, age);Pages = load ‘pages’ as (user, url);Jnd = join Users by name, Pages by user;
Map 1
Reducer 1
(1, user)
Pages
block n
(1, fred)
(2, fred)
(2, fred)
Pages
Users
Map 2
Reducer 2
Users
block m
(1, jane)
(2, jane)
(2, jane)
(2, name)

Fragment Replicate Join
Users = load‘users’as (name, age);Pages = load ‘pages’ as (user, url);Jnd = join Pages by user, Users by name using “replicated”;
Map 1
Users
Pages
block 1
Pages
Users
Map 2
Users
Pages
block 2

Skew Join
Users = load‘users’as (name, age);Pages = load ‘pages’ as (user, url);Jnd = join Pages by user, Users by name using “skewed”;
Map 1
Reducer 1
SP
(1, user)
Pages
block n
(1, fred, p1)
(1, fred, p2)
(2, fred)
Pages
Users
SP
Map 2
Reducer 2
Users
block m
(1, fred, p3)
(1, fred, p4)
(2, fred)
(2, name)

Merge Join
Users = load‘users’as (name, age);Pages = load ‘pages’ as (user, url);Jnd = join Pages by user, Users by name using “merge”;
Map 1
Users
Pages
Pages
Users
aaron…
amr
aaron
…
aaron
.
.
.
.
.
.
.
.
zach
aaron
.
.
.
.
.
.
.
.
zach
Map 2
Users
Pages
amy…
barb
amy
…

Who uses Pig for What?
70% of production grid jobs at Yahoo (10ks per day)
Also used by Twitter, LinkedIn, Ebay, AOL, …
Used to
Process web logs
Build user behavior models
Process images
Build maps of the web
Do research on raw data sets

Components
Accessing Pig:
Submit a script directly
Grunt, the pig shell
PigServer Java class, a JDBC like interface
Job executes on cluster
Hadoop Cluster
Pig resides on user machine
User machine
No need to install anything extra on your Hadoop cluster.

How It Works
Pig Latin
A = LOAD ‘myfile’
AS (x, y, z);
B = FILTER A by x > 0;
C = GROUP B BY x;
D = FOREACH A GENERATE
x, COUNT(B);
STORE D INTO ‘output’;
pig.jar:
parses
checks
optimizes
plans execution
submits jar to Hadoop
monitors job progress
Execution Plan
Map:
Filter
Count
Combine/Reduce:
Sum

New in 0.8
UDFs can be Jython
Improved and expanded statistics
Performance Improvements
Automatic merging of small files
Compression of intermediate results
PigUnit for unit testing your Pig Latin scripts
Access to static Java functions as UDFs
Improved HBase integration
Custom PartitionersB = group A by $0 partition by YourPartitioner parallel2;
Greatly expanded string and math built in UDFs

What’s Next?
Preview of Pig 0.9
Integrate Pig with scripting languages for control flow
Add macros to Pig Latin
Revive ILLUSTRATE
Fix runtime type errors
Rewrite parser to give more useful error messages
Programming Pig from O’Reilly Press

Learn More
Online documentation: http://pig.apache.org/
Hadoop, The Definitive Guide 2nd edition has an up to date chapter on Pig, search at your favorite bookstore
Join the mailing lists:
user@pig.apache.org for user questions
dev@pig.apache.com for developer issues
Follow me on Twitter, @alanfgates

UDFs in Scripting Languages
Evaluation functions can now be written in scripting languages that compile down to the JVM
Reference implementation provided in Jython
Jruby, others, could be added with minimal code
JavaScript implementation in progress
Jython sold separately

Example Python UDF
test.py:
@outputSchema(”sqr:long”)
def square(num): return ((num)*(num))
test.pig:
register 'test.py' using jython as myfuncs;
A = load ‘input’ as (i:int);
B = foreach A generate myfuncs.square(i);
dump B;

Better statistics
Statistics printed out at end of job run
Pig information stored in Hadoop’s job history files so you can mine the information and analyze your Pig usage
Loader for reading job history files included in Piggybank
New PigRunner interface that allows users to invoke Pig and get back a statistics object that contains stats information
Can also pass listener to track Pig jobs as they run
Done for Oozie so it can show users Pig statistics

Sample stats info
Job Stats (time in seconds):
JobId Maps Reduces MxMTMnMT AMT MxRTMnRT ART Alias
job_0 2 1 15 3 9 27 27 27 a,b,c,d,e
job_1 1 1 3 3 3 12 12 12 g,h
job_2 1 1 3 3 3 12 12 12 i
job_3 1 1 3 3 3 12 12 12 i
Input(s):
Successfully read 10000 records from: “studenttab10k"
Successfully read 10000 records from: “votertab10k"
Output(s):
Successfully stored 6 records (150 bytes) in: ”outfile"
Counters:
Total records written : 6
Total bytes written : 150

Invoke Static Java Functions as UDFs
Often UDF you need already exists as Java function, e.g. Java’s URLDecoder.decode() for decoding URLs
define UrlDecode InvokeForString('java.net.URLDecoder.decode', 'String String');A = load 'encoded.txt' as (e:chararray);B = foreach A generate UrlDecode(e, 'UTF-8');
Currently only works with simple types and static functions

Improved HBase Integration
Can now read records as bytes instead of auto converting to strings
Filters can be pushed down
Can store data in HBase as well as load from it
Works with HBase 0.20 but not 0.89 or 0.90. Patch in PIG-1680 addresses this but has not been committed yet.

Apache Hadoop India Summit 2011 talk "Pig - Making Hadoop Easy" by Alan Gate

by Yahoo! Developer Network

Accessibility

Upload Details

Usage Rights

1 Embed 160

Statistics

Apache Hadoop India Summit 2011 talk "Pig - Making Hadoop Easy" by Alan Gate Presentation Transcript