Testing Hadoop jobs with MRUnit

Eric Wendelin Hadooper at Return Path Blog: eriwen.com Twitter: @eriwen

What is MRUnit? • Testing library for MapReduce • Developed by Cloudera • Easy integration between MapReduce and standard testing tools (e.g. JUnit)

Why do I need that?

Testing without MRUnit • Write tests that create JobConf or Configuration objects • conf.set(‘mapred.job.tracker’, ‘local’) • Developing new test input files stored alongside MapReduce test code • Lots of work to validate output files • External file I/O makes tests slooooow

MRUnit makes testing Hadoop jobs easier

Testing with MRUnit • No external test input or output ﬁles • Programmatically speciﬁed • Less test harness code (but also perhaps less control) • Concise, fast tests

class ExampleTest() { private Example.MyMapper mapper private Example.MyReducer reducer private MapReduceDriver driver @Before void setUp() { mapper = new Example.MyMapper() reducer = new Example.MyReducer() driver = new MapReduceDriver(mapper, reducer) } @Test void testMapReduce() { driver.withInput(new Text(‘a’), new Text(‘b’)) driver.withOutput(new Text(‘c’), new Text(‘d’)) driver.runTest() } }

class ExampleTest() { private Example.MyMapper mapper private Example.MyReducer reducer private MapReduceDriver driver @Before void setUp() { mapper = new Example.MyMapper() reducer = new Example.MyReducer() driver = new MapReduceDriver(mapper, reducer) } @Test void testMapReduce() { driver.withInput(new Text(‘a’), new Text(‘b’)) .withOutput(new Text(‘c’), new Text(‘d’)) .runTest() } }

Test map and reduce separately

class ExampleTest() { private Example.MyMapper mapper private MapDriver mDriver @Before void setUp() { mapper = new Example.MyMapper() driver = new MapDriver(mapper) } @Test void testMap() { mDriver.withInput(new Text(‘a’), new Text(‘b’)) mDriver.withOutput(new Text(‘c’), new Text(‘d’)) mDriver.runTest() } }

class ExampleTest() { private Example.MyReducer reducer private ReduceDriver rDriver @Before void setUp() { rDriver = new Example.MyReducer() driver = new ReduceDriver(reducer) } @Test void testReduce() { rDriver.withInput(new Text(‘a’), [new Text(‘foo’), new Text(‘bar’)]) rDriver.withOutput(new Text(‘c’), new Text(‘d’)) rDriver.runTest() } }

Counters!

driver.withInput(...) driver.run() def counters = driver.getCounters() assertEquals(1, counters.findCounter (‘foo’, ‘bar’).getValue())

Verifying logging

def messages = [] def appender = [ append: { messages.add(it) }, requiresLayout: { false } ] as AppenderSkeleton Logger.getRootLogger().addAppender(appender) driver.runTest() assertTrue messages.find { it.getLevel.toString() == ‘WARN’ && it.getMessage().contains(‘My err’) } Logger.getRootLogger().removeAppender(appender)

Cool stuff I haven’t tried... • The PipelineMapReduceDriver - allows testing a series of MapReduce passes • Just call addMapReduce(mapper, reducer) • Mock objects - MockReporter, MockInputSplit, and MockOutputCollector • Test combiners with myMapReduceDriver.setCombiner(myCombiner)

Problems with MRUnit

runTest() does not give meaningful information on failure

Better to use run() and then assert

driver.setInput(new Text(‘foo’), new Text(‘bar’)) def output = driver.run() assertEquals ‘baz’, output[0].first assertEquals ‘jy’, output[0].second

Documentation is severely lacking

runXxx() calls setup() called for new Hadoop API, but not old API

Tests are not executed in a distributed way

In Summary, MRUnit... • Makes testing your Hadoop jobs easier • Abstracts away a lot of the boilerplate test setup you need • Has it’s problems • but they are outweighed by the beneﬁts

http://eriwen.com	4334
http://www.eriwen.com	693
http://feeds2.feedburner.com	428
http://kernel-panik.blogspot.com	172
http://therichwebexperience.com	135
http://feeds.feedburner.com	92
http://nourlcn.ownlinux.net	74
http://www.nofluffjuststuff.com	48
http://kernel-panik.blogspot.in	38
http://www.slideshare.net	23
http://www.therichwebexperience.com	22
http://kernel-panik.blogspot.ru	10
http://kernel-panik.blogspot.de	9
http://dashboard.bloglines.com	7
http://kernel-panik.blogspot.fr	7
http://feedplanets.com	7
http://www.newsblur.com	6
http://kernel-panik.blogspot.co.uk	6
http://127.0.0.1	6
http://www.linkedin.com	5
http://localhost:4000	5
http://www.nfjsone.com	5
http://kernel-panik.blogspot.com.es	4
http://www.rxx.co.il	4
http://www.springone2gx.com	4
http://kernel-panik.blogspot.com.br	3
http://polygynia2.rssing.com	3
http://kernel-panik.blogspot.co.il	3
http://projectautomationexperience.com	2
http://kernel-panik.blogspot.ch	2
http://www.hanrss.com	2
http://kernel-panik.blogspot.it	2
http://prlog.ru	2
http://kernel-panik.blogspot.kr	2
http://kernel-panik.blogspot.jp	2
http://blog.ownlinux.net	2
http://xss.yandex.net	2
http://webcache.googleusercontent.com	2
http://translate.googleusercontent.com	2
http://kernel-panik.blogspot.nl	2
http://www.twylah.com	2
http://newsblur.com	2
http://kernel-panik.blogspot.com.ar	2
http://grails.resourcezen.com	1
http://kernel-panik.blogspot.dk	1
http://blogs.sun.com	1
http://kernel-panik.blogspot.fi	1
http://uberconf.com	1
http://kernel-panik.blogspot.se	1
http://www.zhuaxia.com	1
http://kernel-panik.blogspot.pt	1

Testing Hadoop jobs with MRUnit

by Eric Wendelin

Accessibility

Categories

Upload Details

Usage Rights

54 Embeds 6,194

Statistics

Testing Hadoop jobs with MRUnit Presentation Transcript