Test Driven Elephants
Upcoming SlideShare
Loading in...5
×
 

Test Driven Elephants

on

  • 1,869 views

Test Driven Elephants: ETL Validation with Cascading on Elastic MapReduce

Test Driven Elephants: ETL Validation with Cascading on Elastic MapReduce

Presentation from Data Week 2013, Vijay Ramesh, Change.org

Statistics

Views

Total Views
1,869
Views on SlideShare
304
Embed Views
1,565

Actions

Likes
1
Downloads
4
Comments
0

12 Embeds 1,565

http://making.change.org 1497
http://assets.txmblr.com 48
http://www.bing.com 4
https://www.linkedin.com 3
http://www.soso.com 2
http://search.yahoo.com 2
http://cloud.feedly.com 2
http://translate.googleusercontent.com 2
http://www.linkedin.com 2
http://yandex.ru 1
http://www.google.com 1
http://newsblur.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Test Driven Elephants Test Driven Elephants Presentation Transcript

  • Test Driven Elephants: ETL Validation with Cascading on Elastic MapReduce
  • The problem Difficulty with TDD on Hadoop: · Slow feedback cycle · End-to-end smoke tests only · Requires Hadoop install in your CI environment
  • Cascading’s solution Organize map/reduce jobs into flows, connecting: · taps: source and sink - where your data is coming from, where it is going to (e.g., on HDFS or S3) · pipe assemblies: what types of transformations/operations to run against it, specified independently of the data sources they process from http://docs.cascading.org/impatient/
  • And this buys us? Lots of stuff: · Topological scheduling · Reusable/parameterizable components · Debug facilities (traps, intermediate sinks, logging, etc) · LocalFlowConnector platform to run in-memory flows · and... Testable Components!
  • An example application Verify* extract data from a variety of production sources (MySQL, MongoDB, etc) *meaning data creation/update patterns fall within acceptable (experimentally determined) thresholds “Medium” data: · 60M+ users, 200M+ signatures, 1.7B+ emails · 100M+ rows a week for more active sources · Scale with number of sources & traffic patterns
  • Cascading Flow Initial source standardization: Pipe source = new Rename( new Retain( new Pipe(sourceName + "/continuity"), sourceField ), sourceField, new Fields("timestamp") );
  • Cascading Flow Custom Pipe Assemblies: @VisibleForTesting Pipe timestampContinuityAggregate(Pipe source) { return new Every( new GroupBy( source, new Fields("time_bucket"), new Fields("timestamp") ), new Fields("timestamp", "time_bucket"), new TimestampContinuityAggregator(), new Fields("time_bucket", "average_delta", "max_delta") ); }
  • Pipe assembly: timestampContinuityAggregate
  • Simple unit tests @Test public void testContinuityAggregate() throws Exception { Pipe continuityAggregate = controller.timestampContinuityAggregate( new Pipe("timestampContinuityAggregate") ); Flow flow = getPlatform().getFlowConnector().connect( source(new Fields("timestamp", "time_bucket"), "timeBucket"), sink(new Fields("time_bucket", "average_delta", "max_delta"), "timestampContinuityAggregate"), continuityAggregate ); flow.complete(); validateLength(flow, 2, null); List<Tuple> values = getSinkAsList(flow); for (Tuple value : values) { Assert.assertTrue( value.getString(0).equals("2007-02-23") || value.getString(0).equals("2007-02-24") ); Assert.assertTrue(value.getString(1).equals("2717.5925925925926")); Assert.assertTrue(value.getString(2).equals("19331")); } }
  • Simple unit tests @Test public void testContinuityAggregate() throws Exception { Pipe continuityAggregate = controller.timestampContinuityAggregate( new Pipe("timestampContinuityAggregate") ); Flow flow = getPlatform().getFlowConnector().connect( source(new Fields("timestamp", "time_bucket"), "timeBucket"), sink(new Fields("time_bucket", "average_delta", "max_delta"), "timestampContinuityAggregate"), continuityAggregate ); flow.complete(); validateLength(flow, 2, null); List<Tuple> values = getSinkAsList(flow); for (Tuple value : values) { Assert.assertTrue( value.getString(0).equals("2007-02-23") || value.getString(0).equals("2007-02-24") ); Assert.assertTrue(value.getString(1).equals("2717.5925925925926")); Assert.assertTrue(value.getString(2).equals("19331")); } } @Test public void testContinuityAggregate() throws Exception { Pipe continuityAggregate = controller.timestampContinuityAggregate( new Pipe("timestampContinuityAggregate") ); Flow flow = getPlatform().getFlowConnector().connect( source(new Fields("timestamp", "time_bucket"), "timeBucket"), sink(new Fields("time_bucket", "average_delta", "max_delta"), "timestampContinuityAggregate"), continuityAggregate );
  • Simple unit tests @Test public void testContinuityAggregate() throws Exception { Pipe continuityAggregate = controller.timestampContinuityAggregate( new Pipe("timestampContinuityAggregate") ); Flow flow = getPlatform().getFlowConnector().connect( source(new Fields("timestamp", "time_bucket"), "timeBucket"), sink(new Fields("time_bucket", "average_delta", "max_delta"), "timestampContinuityAggregate"), continuityAggregate ); flow.complete(); validateLength(flow, 2, null); List<Tuple> values = getSinkAsList(flow); for (Tuple value : values) { Assert.assertTrue( value.getString(0).equals("2007-02-23") || value.getString(0).equals("2007-02-24") ); Assert.assertTrue(value.getString(1).equals("2717.5925925925926")); Assert.assertTrue(value.getString(2).equals("19331")); } } flow.complete(); validateLength(flow, 2, null); List<Tuple> values = getSinkAsList(flow); for (Tuple value : values) { Assert.assertTrue( value.getString(0).equals("2007-02-23") || value.getString(0).equals("2007-02-24") ); Assert.assertTrue(value.getString(1).equals("2717.5925925925926")); Assert.assertTrue(value.getString(2).equals("19331")); } }
  • Integration testing package org.change.ml.extract.verification; import ... @PlatformRunner.Platform({LocalPlatform.class}) public class ControllerIntegrationTest extends VerificationTestCase { ... @Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...) Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink ); verifyContinuity.complete(); List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink); Assert.assertTrue(continuityTuples.get(0).toString("t").equals("2013-02-25t7.166666666666667t37")); Assert.assertTrue(continuityTuples.get(1).toString("t").equals("2013-02-26t720.6t3599")); Assert.assertTrue(verifiedTuples.get(0).toString("t").equals( "2013-02-25,2013-02-26t2013-02-26t2013-02-25,2013-02-26t2013-02-26") ); } ... }
  • Integration testing package org.change.ml.extract.verification; import ... @PlatformRunner.Platform({LocalPlatform.class}) public class ControllerIntegrationTest extends VerificationTestCase { ... @Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...) Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink ); verifyContinuity.complete(); List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink); Assert.assertTrue(continuityTuples.get(0).toString("t").equals("2013-02-25t7.166666666666667t37")); Assert.assertTrue(continuityTuples.get(1).toString("t").equals("2013-02-26t720.6t3599")); Assert.assertTrue(verifiedTuples.get(0).toString("t").equals( "2013-02-25,2013-02-26t2013-02-26t2013-02-25,2013-02-26t2013-02-26") ); } ... } package org.change.ml.extract.verification; import ... @PlatformRunner.Platform({LocalPlatform.class}) public class ControllerIntegrationTest extends VerificationTestCase { ... @Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...)
  • Integration testing Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink ); package org.change.ml.extract.verification; import ... @PlatformRunner.Platform({LocalPlatform.class}) public class ControllerIntegrationTest extends VerificationTestCase { ... @Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...) Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink ); verifyContinuity.complete(); List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink); Assert.assertTrue(continuityTuples.get(0).toString("t").equals("2013-02-25t7t37")); Assert.assertTrue(continuityTuples.get(1).toString("t").equals("2013-02-26t720.6t3599")); Assert.assertTrue(verifiedTuples.get(0).toString("t").equals( "2013-02-25,2013-02-26t2013-02-26t2013-02-25,2013-02-26t2013-02-26") ); } ... }
  • Integration testing package org.change.ml.extract.verification; import ... @PlatformRunner.Platform({LocalPlatform.class}) public class ControllerIntegrationTest extends VerificationTestCase { ... @Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...) Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink ); verifyContinuity.complete(); List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink); Assert.assertTrue(continuityTuples.get(0).toString("t").equals("2013-02-25t7t37")); Assert.assertTrue(continuityTuples.get(1).toString("t").equals("2013-02-26t720.6t3599")); Assert.assertTrue(verifiedTuples.get(0).toString("t").equals( "2013-02-25,2013-02-26t2013-02-26t2013-02-25,2013-02-26t2013-02-26") ); } ... } verifyContinuity.complete(); List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink); Assert.assertTrue(continuityTuples.get(0).toString("t").equals("2013-02-25t7t37")); Assert.assertTrue(continuityTuples.get(1).toString("t").equals("2013-02-26t720.6t3599")); Assert.assertTrue(verifiedTuples.get(0).toString("t").equals( "2013-02-25,2013-02-26t2013-02-26t2013-02-25,2013-02-26t2013-02-26") );
  • Running tests locally
  • is awesome! Check out https://github.com/Cascading/cascading.samples for examples using Gradle to drive Cascading
  • Now what? · Continuous Deployment · Gradle on Circle CI build to S3 · Elastic MapReduce scaleable · DataPipeline provisioned clusters · Production Monitoring · SNS to Pager Duty integration on failures
  • We’re hiring! Thanks to my colleagues at Change.org for all their help with this presentation. If you’re interested in building large-scale distributed systems for data processing and recommendations (along with lots of other cool stuff that helps empower people everywhere to create the change they want to see), we’re hiring! Drop me an email (vijay@change.org) or visit our website (http://www.change.org/hiring)