This document provides an introduction and overview of Pig UDFs (User Defined Functions). It discusses what UDFs are, why they are used, and how to write and use them. The main types of UDFs covered are EvalFunc and LoadFunc. EvalFunc basics are explained using the example of the UPPER EvalFunc from the Piggybank. More advanced EvalFunc topics like returning Tuples are also covered. Finally, the document mentions the Piggybank as a source of common UDFs.

1. Introduction to
Pig UDFs
Chris Wilkes
cwilkes@seattlehadoop.org

2. Agenda
1 What, Why, How
2 EvalFunc basics
3 More EvalFunc
4 LoadFunc
5 Piggybank

3. Agenda Point 1
1 What, Why, How
2 EvalFunc basics
3 More EvalFunc
4 LoadFunc
5 Piggybank

4. What is a UDF?
User Defined Function
• Way to do an operation on a field or fields
• Note: not on the group
• Called from within a pig script
• b = FOREACH a GENERATE foo(color)
• Currently all done in java

5. Why use a UDF?
• You need to do more than grouping or
filtering
• Actually filtering is a UDF
• Probably using them already
• Maybe more comfortable in java land
than in SQL / Pig Latin

6. How to write an use?
• Just extend / implement an
interface
• No need for administrator
rights, just call your script
• Very simple java, just think
about your small problem
Magical Powers not required

7. Moving right along
Now to the informative part of the talk

8. Agenda
1 What, Why, How
2 EvalFunc basics
3 More EvalFunc
4 LoadFunc
5 Piggybank

9. EvalFunc : probably what you need to do
•Easiest to understand: takes one or more
fields and spits back a generic object
•Extend the EvalFunc interface and it
practically writes itself
•Let’s look at the UPPER example from the
piggybank

10. The UPPER EvalFunc
public class UPPER extends EvalFunc<String> {
@Override
public String exec(Tuple input) throws IOException {
if (input == null||input.size() == 0||input.get(0) == null)
return null;
try {
return ((String)input.get(0)).toUpperCase();
} catch (ClassCastException e) {
warn(“error msg”, PigWarning.UDF_WARNING_1);
} catch(Exception e){
warn("Error”, PigWarning.UDF_WARNING_1);
}
return null;
}
}
modified version from the piggybank SVN

11. The UPPER EvalFunc
public class UPPER extends EvalFunc<String> {
@Override
public String exec(Tuple input) throws IOException {
if (input == null||input.size() == 0||input.get(0) == null)
return null;
try {
return ((String)input.get(0)).toUpperCase();
} catch (ClassCastException e) {
warn(“error msg”, PigWarning.UDF_WARNING_1);
} catch(Exception e){
warn("Error”, PigWarning.UDF_WARNING_1);
}
return null;
}
}
The generic <String> tells Pig what class will be
returned from this method

12. The UPPER EvalFunc
public class UPPER extends EvalFunc<String> {
@Override
public String exec(Tuple input) throws IOException {
if (input == null||input.size() == 0||input.get(0) == null)
return null;
try {
return ((String)input.get(0)).toUpperCase();
} catch (ClassCastException e) {
warn(“error msg”, PigWarning.UDF_WARNING_1);
} catch(Exception e){
warn("Error”, PigWarning.UDF_WARNING_1);
}
return null;
}
}
The Tuple input contains the fields within the script ()

13. The UPPER EvalFunc
public class UPPER extends EvalFunc<String> {
public String exec(Tuple input) throws IOException {
if (input == null||input.size() == 0||input.get(0) == null)
return null;
try {
return ((String)input.get(0)).toUpperCase();
} catch (ClassCastException e) {
warn(“error msg”, PigWarning.UDF_WARNING_1);
} catch(Exception e){
warn("Error”, PigWarning.UDF_WARNING_1);
}
return null;
}
}
Check your inputs for empties or nulls

14. The UPPER EvalFunc
public class UPPER extends EvalFunc<String> {
public String exec(Tuple input) throws IOException {
if (input == null||input.size() == 0||input.get(0) == null)
return null;
try {
return ((String)input.get(0)).toUpperCase();
} catch (ClassCastException e) {
warn(“error msg”, PigWarning.UDF_WARNING_1);
} catch(Exception e){
warn("Error”, PigWarning.UDF_WARNING_1);
}
return null;
}
}
You have to know that the 1st parameter inside the
tuple is a String

15. The UPPER EvalFunc
public class UPPER extends EvalFunc<String> {
public String exec(Tuple input) throws IOException {
if (input == null||input.size() == 0||input.get(0) == null)
return null;
try {
return ((String)input.get(0)).toUpperCase();
} catch (ClassCastException e) {
warn(“error msg”, PigWarning.UDF_WARNING_1);
} catch(Exception e){
warn("Error”, PigWarning.UDF_WARNING_1);
}
return null;
}
}
Catch errors that are acceptable and return null so
can be skipped over

16. The UPPER EvalFunc
public class UPPER extends EvalFunc<String> {
public List<FuncSpec> getArgToFuncMapping() {
List<FuncSpec> funcList = new ArrayList<FuncSpec>();
funcList.add(new FuncSpec(this.getClass().getName(),
new Schema(new Schema.FieldSchema(null,
DataType.CHARARRAY))));
return funcList;
}
}
Tells Pig what parameters this function takes

17. Recap of UPPER
• Generics outlines contract for return type
• Schemas are preserved (chararray / String)
• Check inputs for empty or null
• Return null if item should be skipped
• Throw an exception if deadly
• Name “UPPER” can be used if known to
PigContext’s packageImportList, otherwise need
full classname
• Cast items inside of the Tuple parameter

18. Another simple EvalFunc: AstroDist
• Two input files: planet names with coordinates
and pairs of planets
• Goal: find the distance between the pairs
• Loading is slightly different: coords in a tuple
• Input to EvalFunc is a Tuple that contains a Tuple

19. AstroDist input files
$ cat src/test/resources/cosmo
aaa bbb
aaa ccc
ddd aaa
$ cat src/test/resources/planets
aaa (1,0,10)
bbb (2,-5,15)
ccc (-7,12,48) image from xkcd.com
ddd (3,3,8)

20. AstroDist pig script
REGISTER target/pig-demo-1.0-SNAPSHOT.jar;
planets = load '$dir/planets' as (name : chararray,
l:tuple(x : int, y : int, z : int));
cosmo = load '$dir/cosmo' as (planet1 : chararray, planet2 : chararray);
A = JOIN cosmo BY planet1, planets BY name;
B = JOIN A by planet2, planets BY name;
locations = FOREACH B GENERATE
$1 AS p1name:chararray,
$2 AS p2name : chararray,
AstroDist($3,$5) as distance;
dump locations;

21. AstroDist output
$ pig -x local -f src/test/resources/distances.pig
-param dir=src/test/resources/
What B looks like:
(ddd,aaa,ddd,(3,3,8),aaa,(1,0,10))
(aaa,bbb,aaa,(1,0,10),bbb,(2,-5,15))
(aaa,ccc,aaa,(1,0,10),ccc,(-7,12,48))
Output:
(aaa,ddd,4.123105625617661)
(bbb,aaa,7.14142842854285)
(ccc,aaa,40.64480286580315)

22. AstroDist program
public class AstroDist extends EvalFunc<Double> {
@Override
public Double exec(Tuple input) throws IOException {
Point3D astroPos1 = new Point3D((Tuple) input.get(0));
Point3D astroPos2 = new Point3D((Tuple) input.get(1));
return astroPos1.distance(astroPos2);
}
@Override
public List<FuncSpec> getArgToFuncMapping() {
Schema s = new Schema();
s.add(new Schema.FieldSchema(null, DataType.TUPLE));
s.add(new Schema.FieldSchema(null, DataType.TUPLE));
return Arrays.asList(
new FuncSpec(this.getClass().getName(), s));
}
}

23. AstroDist program (cont)
private static class Point3D {
private final int x, y, z;
private Point3D(Tuple tuple) throws ExecException {
if (tuple.size() != 3) {
throw new ExecException("Received " + tuple.size() +
" points in 3D tuple", ERROR_CODE_BAD_TUPLE, PigException.BUG);
}
x = (Integer) tuple.get(0);
y = (Integer) tuple.get(1);
z = (Integer) tuple.get(2);
}
private double distance(Point3D other) {
return Math.sqrt(Math.pow(x - other.x, 2) +
Math.pow(y - other.y, 2) + Math.pow(z - other.z, 2));
}
}

24. Fun times when running this script
• Looking through PigContext and Main found
that /pig.properties in the classpath is parsed for
the key/value “udf.import.list”
• Put this into my jar (src/main/resources with
maven) but it didn’t appear to load
• Debug log should show what’s going on, except
debug isn’t turned on till after this load
• Ended up putting into ~/.pigrc but Pig warns that
it should go into conf/pig.properties, a file that
isn’t read
• Schemas and UDFs are picky, use trial and error

25. Agenda Point 3
1 What, Why, How
2 EvalFunc basics
3 More EvalFunc
4 LoadFunc
5 Piggybank

26. Returning a Tuple from a UDF
• Sometimes you want to return more than one
thing from a function
• For example an expensive calculation was done
and its results can be reused
• But what should be returned?
• Of course a Tuple
• “tuple” is the answer 92% of the time
http://tuplemusic.org/
Tuple is dedicated to exploring and expanding
the contemporary repertoire for two bassoons

27. BestBook: returns the highest scored book
$ cat src/test/resources/bookscores
book1 aaa 1
book1 bbb 3
Want output of that for
book1 ccc 12
book3 reviewer bbb was
book2 aaa 4
the highest at 5
book2 bbb 1
book3 ccc 1
book3 bbb 5

28. BestBook EvalFunc
public class BestBook extends EvalFunc<Tuple> {
@Override
public Tuple exec(Tuple p_input) throws IOException {
Iterator<Tuple> bagReviewers =
((DataBag) p_input.get(0)).iterator();
Iterator<Tuple> bagScores =
((DataBag) p_input.get(1)).iterator();
int bestScore = -1;
String bestReviewer = null;
while (bagReviewers.hasNext() && bagScores.hasNext()) {
String reviewerName = (String) bagReviewers.next().get(0);
Integer score = (Integer) bagScores.next().get(0);
if (score.intValue() > bestScore) {
bestScore = score;
bestReviewer = reviewerName;
}
}
return TupleFactory.getInstance().newTuple(
Arrays.asList(bestReviewer, (Integer) bestScore));
}

29. BestBook EvalFunc
public class BestBook extends EvalFunc<Tuple> {
@Override
public Tuple exec(Tuple p_input) throws IOException {
Iterator<Tuple> bagReviewers =
((DataBag) p_input.get(0)).iterator();
Iterator<Tuple> bagScores =
((DataBag) p_input.get(1)).iterator();
int bestScore = -1;
String bestReviewer = null;
while (bagReviewers.hasNext() && bagScores.hasNext()) {
String reviewerName = (String) bagReviewers.next().get(0);
Integer score = (Integer) bagScores.next().get(0);
if (score.intValue() > bestScore) {
bestScore = score;
bestReviewer = reviewerName;
}
}
return TupleFactory.getInstance().newTuple(
Arrays.asList(bestReviewer, (Integer) bestScore));
}
The inputs are bag “columns”

30. BestBook EvalFunc
public class BestBook extends EvalFunc<Tuple> {
@Override
public Tuple exec(Tuple p_input) throws IOException {
Iterator<Tuple> bagReviewers =
((DataBag) p_input.get(0)).iterator();
Iterator<Tuple> bagScores =
((DataBag) p_input.get(1)).iterator();
int bestScore = -1;
String bestReviewer = null;
while (bagReviewers.hasNext() && bagScores.hasNext()) {
String reviewerName = (String) bagReviewers.next().get(0);
Integer score = (Integer) bagScores.next().get(0);
if (score.intValue() > bestScore) {
bestScore = score;
bestReviewer = reviewerName;
}
}
return TupleFactory.getInstance().newTuple(
Arrays.asList(bestReviewer, (Integer) bestScore));
}
return a Tuple that’s just like the inputs

31. BestBook EvalFunc
public class BestBook extends EvalFunc<Tuple> {
@Override
public Schema outputSchema(Schema p_input) {
try {
return Schema.generateNestedSchema(DataType.TUPLE,
DataType.CHARARRAY, DataType.INTEGER);
} catch (FrontendException e) {
throw new IllegalStateException(e);
}
}
}
How to define the outbound
schema inside the Tuple

32. BestBook: returns the highest scored book
REGISTER target/demo-pig-udf-1.0-SNAPSHOT.jar;
A = LOAD '$dir/bookscores' as (name : chararray,
reviewer : chararray, score : int);
B = group A by name;
describe B;
dump B;
C = FOREACH B GENERATE group,
BestBook(A.reviewer, A.score) as reviewandscore;
describe C;
dump C;

33. BestBook: returns the highest scored book
B: {group: chararray,A: {name: chararray,reviewer: chararray,score: int}}
(book1,{(book1,aaa,1),(book1,bbb,3),(book1,ccc,12)})
(book2,{(book2,aaa,4),(book2,bbb,1)})
(book3,{(book3,ccc,1),(book3,bbb,5)})
C: {group: chararray,reviewandscore: (chararray,int)}
(book1,(ccc,12))
(book2,(aaa,4))
(book3,(bbb,5))

34. BestBook: improve by implementing Algebraic
•If EvalFunc can be run in stages and summed up consider
implementing Algebraic
•Three methods to override:
•String getInitial();
•String getIntermed();
•String getFinal()
•See COUNT and DoubleAvg

35. FilterFunc: a filter that’s an EvalFunc
• For keeping and disgarding entries write a filter
• FilterFunc extends EvalFunc<Boolean>
• Adds a method “void finish()” for cleanup
• Example: only wants dates that are within 10
minutes of one another

36. FilterFunc: DateWithinFilter
public class DateWithinFilter extends FilterFunc {
@Override
public Boolean exec(Tuple input) throws IOException {
if (input.size() != 3) {
throw new IOException(“error msg”);
}
Date[] startAndTryDates = getColumnDates(input);
if (startAndTryDates == null)
return false;
long dateDiff = startAndTryDates[1].getTime() -
startAndTryDates[0].getTime();
if (dateDiff < 0) {
return false; // maybe make optional
}
int maxDateDiff = (Integer) input.get(2);
return dateDiff <= maxDateDiff;
}

37. FilterFunc: DateWithinFilter
private Date[] getColumnDates(Tuple input) throws ExecException {
String strDate1 = (String) input.get(0);
String strDate2 = (String) input.get(1);
if (strDate1 == null || strDate2 == null) {
return null;
}
Date date1 = null;
try {
date1 = df.parse(strDate1);
} catch (ParseException e) {
warn(“date format err”, PigWarning.UDF_WARNING_1);
return null;
}
Date date2 = null;
try {
date2 = df.parse(strDate2);
} catch (ParseException e) {
warn(“date format err”, PigWarning.UDF_WARNING_1);
return null;
}
return new Date[] { date1, date2 };
}

38. FilterFunc: DateWithinFilter
@Override
public List<FuncSpec> getArgToFuncMapping() throws
FrontendException {
List<FuncSpec> funcList = new ArrayList<FuncSpec>();
Schema s = new Schema();
s.add(new Schema.FieldSchema(null, DataType.CHARARRAY));
s.add(new Schema.FieldSchema(null, DataType.CHARARRAY));
s.add(new Schema.FieldSchema(null, DataType.INTEGER));
funcList.add(new FuncSpec(this.getClass().getName(), s));
return funcList;
}
Defining what inputs we accept
stay tuned for what happens when violated

39. FilterFunc: DateWithinFilter
$ cat src/test/resources/purchasetimes
1234 2010-06-01 10:31:22 2010-06-01 10:32:22
7121 2010-06-01 10:30:18 2010-06-01 11:02:59
1234 2010-06-01 10:40:18 2010-06-01 10:45:32
7681 lol wut
4532 2010-06-01 11:37:18 2010-06-01 11:42:59
$ cat src/test/resources/purchasetimes.pig
REGISTER target/demo-pig-udf-1.0-SNAPSHOT.jar;
purchasetimes = LOAD '$dir/purchasetimes' AS
(userid: int, datein: chararray, dateout: chararray);
quickybuyers = FILTER purchasetimes BY
DateWithinFilter(datein, dateout, 600000);
DUMP quickybuyers; $ pig -x local -f src/test/resources/purchasetimes.pig
-param dir=src/test/resources/
(1234,2010-06-01 10:31:22,2010-06-01 10:32:22)
(1234,2010-06-01 10:40:18,2010-06-01 10:45:32)
(4532,2010-06-01 11:37:18,2010-06-01 11:42:59)

40. EvalFunc: not passing in correct number args
$ cat src/test/resources/purchasetimes.pig
quickybuyers = FILTER purchasetimes BY
DateWithinFilter(datein, dateout);
$ pig -x local -f src/test/resources/purchasetimes.pig -param dir=src/test/resources/
2010-06-17 17:25:43,440 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
1045: Could not infer the matching function for
org.seattlehadoop.demo.pig.udf.DateWithinFilter as multiple or none of them fit.
Please use an explicit cast.
Details at logfile: /Users/cwilkes/Documents/workspace5/SeattleHadoop-
demo-code/pig_1276820742917.log
log file has:
at
org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVi
sitor.java:1197)
so error caught before loading data

41. Agenda
1 What, Why, How
2 EvalFunc basics
3 More EvalFunc
4 LoadFunc
5 Piggybank

42. LoadFunc: definition
• How does something get loaded into Pig?
• A = load ‘B’;
• But what is actually going on?
• A = load ‘B’ using PigStorage();
• PigStorage is a LoadFunc that reads off of disk
and splits on tab to create a Tuple

43. LoadFunc: definition
• LoadFunc is an interface with a number of
methods, the most interesting being
• bindTo(fileName,inputStream,offset,end)
• Tuple getNext()
• Extend from UTF8StorageConverter like
PigStorage to get defaults
• Overview: PigStorage’s getNext() creates an array
of objects after splitting on a tab and puts those
into a Tuple

44. LoadFunc: make our own
• Have a lot of log files, some just contain a URL
• http://example.com?use=mind+bullets&target=yak
• Want to load URLs and do analysis
• Write your own LoadFunc to do this that takes a
URL and returns a Map of the query parameters
• Know what parameters you care about, only look
for those

45. LoadFunc: make our own
• Have a lot of log files, some just contain a URL
• http://example.com?use=mind+bullets&target=yak
• Want to load URLs and do analysis
• Write your own LoadFunc to do this that takes a
URL and returns a Map of the query parameters
• Know what parameters you care about, only look
for those
• Goal:
• A = LOAD 'urls' USING
QuerystringLoader('query', 'userid') AS (query:
chararray, userid : int);

46. LoadFunc: QuerystringLoader
• Passing in constructor arguments from the pig
script is easy:
• public QuerystringLoader(String... fieldNames)
• bindTo is almost exactly the same as the
PigStorage one, using the PigLineRecordReader
to parse the InputStream
• Tuple getTuple() is where the action happens
• parse the querystring into a Map
• loop through the fields given in the constructor
• return a Tuple of a list of those objects

47. LoadFunc: QuerystringLoader getTuple()
@Override
public Tuple getNext() throws IOException {
if (in == null || in.getPosition() > end) {
return null;
}
Text value = new Text();
boolean notDone = in.next(value);
if (!notDone) {
return null;
}
Map<String, Object> parameters = getParameterMap(value.toString());
List<String> output = new ArrayList<String>();
for (String fieldName : m_fieldsInOrder) {
Object object = parameters.get(fieldName);
if (object == null) {
output.add(null);
continue;
}
if (object instanceof String) {
output.add((String) object);
} else {
List<String> objectVal = (List<String>) object;
output.add(objectVal.get(0));
}
}
return mTupleFactory.newTupleNoCopy(output);
}

48. LoadFunc: notes
• boolean okay=in.next(tuple) is how to get the next
parsed line
• getParameterMap(url) splits querystring into a
Map<String,Object>
• Pig handles type conversion for you, just hand back
a Tuple.
• In this case the Tuple can be made up of anything so
user specifies the schema in the script
• AS (query:chararray, userid:int)

49. RegexLoader
Same concept, pass in a Pattern for the constructor
and have getTuple() return only the matched parts
@Override
public Tuple getNext() throws IOException {
Matcher m = m_linePattern.matcher(value.toString());
if (!m.matches()) {
return EmptyTuple.getInstance();
}
List<String> regexMatches = new ArrayList<String>();
for (int i = 1; i <= m.groupCount(); i++) {
regexMatches.add(m.group(i));
}
return mTupleFactory.newTupleNoCopy(regexMatches);
}

50. Agenda
1 What, Why, How
2 EvalFunc basics
3 More EvalFunc
4 LoadFunc
5 Piggybank

51. Piggybank
• CVS repository of common UDFs
• Excited about it at first, doesn’t appear to be
used that much
• Needs to be an easier way of doing this
• CPAN (Perl) for Pig would be great
• register pigpan://Math::FFT
• brings down the jars from a maven-like
repository and tells pig where to load from
• any takers? Looking into it

52. Bonus section: unit testing
@Test
public void testRepeatQueryParams() throws IOException {
String url = "http://localhost/foo?a=123&a=456nx=y
nhttp://localhost/bar?a=761&b=hi";
QuerystringLoader loader = new QuerystringLoader("a", "b");
InputStream in = new ByteArrayInputStream(url.getBytes());
loader.bindTo(null,
new BufferedPositionedInputStream(in), 0, url.length());
Tuple tuple = loader.getNext();
assertEquals("123", (String) tuple.get(0));
assertNull(tuple.get(1));
tuple = loader.getNext();
assertEquals(2, tuple.size());
assertNull(tuple.get(0));
assertNull(tuple.get(1));
tuple = loader.getNext();
assertEquals("761", (String) tuple.get(0));
assertEquals("hi", (String) tuple.get(1));
}

53. Resources
UDF reference:
http://hadoop.apache.org/pig/docs/r0.5.0/
piglatin_reference.html
Code samples:
http://github.com/seattlehadoop
Presentation:
http://www.slideshare.net/seattlehadoop