SlideShare a Scribd company logo

The Mechanics of Testing Large Data Pipelines

•
•
C4Media
C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1Vx8CwP. Mathieu Bastian explores the mechanics of testing large, complex data workflows and tries to identify the most common challenges developers face. He looks at good practices how to develop unit, integration, data and performance testing for data workflows. In terms of tools, he looks at what exists today for Hadoop, Pig and Spark with code examples. Filmed at qconlondon.com. Mathieu Bastian spent the last four years building data products at LinkedIn. He joined LinkedIn's Data Science team in late 2010 and first worked on InMaps, a system that served millions of complex graph visualizations to users. Previously, Bastian co-founded and led the development of Gephi, an award-winning open-source software to visualize and analyze large graphs.

Technology
1 of 78
THE MECHANICS OF TESTING LARGE DATA PIPELINES MATHIEU BASTIAN Head of Data Engineering, GetYourGuide @mathieubastian www.linkedin.com/in/mathieubastian QCon London 2015
InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /patterns-data-workflows
Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
Outline ▸ Motivating example ▸ Challenges ▸ Testing strategies ▸ Validation Strategies ▸ Tools Integration Tests ArchitectureUnit Test
Data Pipelines often start simple
Users E-commerce website Search App Views Offline Dashboard Search Metrics Views HDFS They have one use-case and one developer
But there are many other use- cases Recommender Systems Anomaly Detection Search Ranking A/B Testing Spam Detection Sentiment Analysis Topic Detection Trending TagsQuery Expansion Customer Churn Prediction Related searches Fraud Prediction Bidding Prediction Machine Translation Signal Processing Content Curation Sentiment Analysis Image recognition Optimal pricing Location normalization Standardization Funnel Analysis
additional events and logsDevelopers add Users E-commerce website Search App Clicks Views Offline Dashboard Search Metrics Clicks Views HDFS
third-party dataDevelopers add Users E-commerce website 3rd parties Search App Clicks Views A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views A/B Logs HDFS
search ranking predictionDevelopers add Users E-commerce website 3rd parties Search App Clicks Views A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views Training data Training & validation Model Clicks Views Features transformation A/B Logs HDFS
personalized user featuresDevelopers add Users E-commerce website 3rd parties Search App Clicks Views ProfilesUser Database A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views Training data Training & validation Model Clicks Views Profiles Features transformation A/B Logs HDFS
query extensionDevelopers add Users E-commerce website 3rd parties Search App Clicks Views ProfilesUser Database A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views Training data Training & validation Model Clicks Views Profiles Features transformation A/B Logs Filter queries Query extension RDBMS Views Training data HDFS
Developers add recommender system Users E-commerce website 3rd parties Search App Clicks Views ProfilesUser Database A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views Training data Training & validation Model Clicks Views Profiles Features Features transformation Features NoSQL Compute recommendations A/B Logs Filter queries Query extension RDBMS Views Training data HDFS
Data Pipelines can grow very large
That is a lot of code and data
Code contain bugs Industry Average: about 15 - 50 errors per 1000 lines of delivered code.
Data will change Industry Average: ?
Embrace automated testing of code validation of data
Because it delivers ▸ Testing ▸ Tested code has less bugs ▸ Gives the confidence to iterate quickly ▸ Scales well to multiple developers ▸ Validation ▸ Reduce manual testing ▸ Avoid catastrophic failures
But it’s challenging ▸ Testing ▸ Need data to test "realistically" ▸ Not running locally, can be expensive ▸ Tooling weaknesses ▸ Validation ▸ Data sources out of our control ▸ Difficult to test machine learning models
Reality check Source: @SteveGodwin, QCon London 2016
Manual testing Waiting Coding Looking at logs Code Upload Run workflow Look at logs ▸ Time Spent
Testing strategies
Prepare environment ▸ Care about tests from the start of your project ▸ All jobs should be functions (output only depends on input) ▸ Safe to re-run the job ▸ Does the input data still exists? ▸ Would it push partial results? ▸ Centralize configurations and no hard-coded paths ▸ Version code and timestamp data
Unit test locally ▸ Test locally each individual job ▸ Tests its good code ▸ Tests expected failures ▸ Need to overcome challenges with fake data creation ▸ Complex structures and numerous data sources ▸ Too small to be meaningful ▸ Need to specify a different configuration
Build from schemas Fake data creation based on schemas. Compare: Customer c = Customer.newBuilder().
 setId(42).
 setInterests(Arrays.asList(new Interest[]{
 Interest.newBuilder().setId(0).setName("Ping-Pong").build()
 Interest.newBuilder().setId(1).setName(“Pizza").build()})) .build(); vs Map<String, Object> c = new HashMap<>();
 c.put("id", 42);
 Map<String, Object> i1 = new HashMap<>();
 i1.put("id", 0);
 i1.put("name", "Ping-Pong");
 Map<String, Object> i2 = new HashMap<>();
 i2.put("id", 1);
 i2.put("name", "Pizza");
 c.put("interests", Arrays.asList(new Map[] {i1, i2}));
Build from schemas Avro Schema example { "type": "record", "name": "Customer", "fields": [{ "name": "id", "type": "int" }, { "name": "interests", "type": { "type": "array", "items": { "name": "Interest", "type": "record", "fields": [{ "name": "id", "type": "int" }, { "name": "name", "type": ["string", "null"] }] } } } ] } nullable field
Complex generators ▸ Developed in the field of property-based testing //Small Even Number Generator val smallEvenInteger = Gen.choose(0,200) suchThat (_ % 2 == 0) ▸ Goal is to simulate, not sample real data ▸ Define complex random generators that match properties (e.g. frequency) ▸ Can go beyond unit-testing and generate complex domain models ▸ https://www.scalacheck.org/ for Scala/Java is a good starting point for examples
Integration test on sample data ▸ Integration test the entire workflow ▸ File paths ▸ Configuration ▸ Evaluate performance ▸ Sample data ▸ Large enough to be meaningful ▸ Small enough to speed-up testing JOB A JOB B JOB C JOB D
Validation strategies
Where it fail Control Difficulty Model biases Bug Noisy data Schema changes Missing data
Input and output validation Make the pipeline robust by validating inputs and outputs Input Input Input Workflow Production ValidationValidation
Input Validation
Input data validation Input data validation is a key component of pipeline robustness. The goal is to test the entry points of our system for data quality. ETL RDBMS NOSQL EVENTS TWITTER DATA PIPELINE
Why it matters ▸ Bad input data will most likely degrade the output ▸ It likely will fail silently ▸ Because data will change ▸ Data migrations: maintenance, cluster update, new infrastructure ▸ Events change due to product evolution ▸ Data dependencies updated
Input data validation ▸ Validation code should ▸ Detect pathological data and fail early ▸ Deal with expected data variability ▸ Example issues: ▸ Missing values, encoding issues, etc. ▸ Schema changes ▸ Duplicates rows ▸ Data order changes
Pathological data ▸ Value ▸ Validity depends on a single, independent value. ▸ Easy to validate on streams of data ▸ Dataset ▸ Validity depends on the entire dataset ▸ More difficult to validate as it needs a window of data
Metadata validation Analyzing metadata is the quickest way to validate input data ▸ Number of records and file sizes ▸ Hadoop/Spark counters ▸ Number of map/reduce records, size ▸ Record-level custom counters ▸ Average text length ▸ Task-level custom counters ▸ Min/Max/Median values
Hadoop/Spark counters Results can be accessed programmatically and checked
Control inputs with Schemas ▸ CSVs aren’t robust to change, use Schemas ▸ Makes expected data explicit and easy to test against ▸ Gives basic validation for free with binary serialization (e.g. Avro, Thrift, Protocol Buffer) ▸ Typed (integer, boolean, lists etc.) ▸ Specify if value is optional ▸ Schemas can be evolved without breaking compatibility
Output Validation
Why it matters ▸ Humans makes mistake, we need a safeguard ▸ Rolling back data is often complex ▸ Bad output propagates to downstream systems Example with a recommender system // One recommendation set per user { "userId": 42, "recommendations": [{ "itemId": 1456, "score": 0.9 }, { "itemId": 4232, "score": 0.1 }], "model": "test01" }
Check for anomalies Simple strategies similar to input data validation ▸ Record level (e.g. values within bounds) ▸ Dataset level (e.g. counts, order) Challenges around relevance evaluation ▸ When supervised, use a validation dataset and threshold accuracy ▸ Introduce hypothetical examples
Incremental update as validation Join with the previous “best" output ▸ Allows fine comparisons ▸ Incremental framework can be extended to ▸ Only recompute recommendations that have changed ▸ Produce variations metric between different models Daily Recommendations Compute daily recommendationsHDFS Recommendations Yesterday Recommendations Join with previous result
External validation Even in automated environment it is possible to validate with humans ▸ Example: Search ranking evaluation ▸ Solution: Crowdsourcing ▸ Complex validation that requires training ▸ Can be automated through APIs
Mitigate risk with A/B testing Gradually rolling out data products improvements reduces the need for complex output validation ▸ Experiment can be controlled online or offline ▸ Online: Push multiple set of recommendations (1 per model) ▸ Offline: Split users and push unique set of recommendations userId -> [{
 "model": "test01",
 "recommendations": [{...}]
 }, {
 "model": "test02",
 "recommendations": [{...}]
 }] A B userId -> {
 "model": "test01",
 "recommendations": [{...}]
 }
Mitigate risk with A/B testing Important ▸ Log model variation downstream in logs ▸ Encapsulate model logic FEATURE 1-A MODEL A MODEL B FEATURE 1 FEATURE 2 MODEL A MODEL B A BA B FEATURE 1-B FEATURE 2-A FEATURE 2-B
Thank You! We are hiring! http://careers.getyourguide.com/
Appendix
Hadoop Testing
Two ways to test Hadoop jobs ▸ MRUnit ▸ Java library to test MapReduce jobs in a simulated environment ▸ Last release June 2014 ▸ MiniCluster ▸ Utility to locally run a fully-functional Hadoop cluster in a test environment ▸ Ships with Hadoop itself
MiniMRCluster ▸ Advantages ▸ Behaves like a real cluster, including setup and configuration ▸ Can be used to test multiple jobs (integration testing) ▸ Disadvantages ▸ Very slow compared to unit testing Java code
MRUnit ▸ Advantages ▸ Faster ▸ Less boilerplate code ▸ Disadvantages ▸ Need to replicate job configuration ▸ Only built to test map and reduce functions ▸ Difficult to make it work with custom input formats (e.g. Avro)
MiniMRCluster setup* Setup MR cluster and obtain FileSystem @BeforeClass
 public void setup() {
 Configuration dfsConf = new Configuration();
 dfsConf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, new File("./target/ hdfs/").getAbsolutePath());
 _dfsCluster = new MiniDFSCluster.Builder(dfsConf).numDataNodes(1).build();
 _dfsCluster.waitClusterUp();
 _fileSystem = _dfsCluster.getFileSystem();
 
 YarnConfiguration yarnConf = new YarnConfiguration();
 yarnConf.setFloat(YarnConfiguration.NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE, 99.0f);
 yarnConf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 64);
 yarnConf.setClass(YarnConfiguration.RM_SCHEDULER, FifoScheduler.class, ResourceScheduler.class);
 _mrCluster = new MiniMRYarnCluster(getClass().getName(), taskTrackers);
 yarnConf.set("fs.defaultFS", _fileSystem.getUri().toString());
 _mrCluster.init(yarnConf);
 _mrCluster.start();
 } * Hadoop version used 2.7.2
Keep the test file clean of boilerplate code Best is to wrap the start/stop code into a TestBase class /**
 * Default constructor with one task tracker and one node.
 */
 public TestBase() { ... }
 
 @BeforeClass
 public void startCluster() throws IOException { ... }
 
 @AfterClass
 public void stopCluster() throws IOException { ... }
 
 /**
 * Returns the Filesystem in use.
 *
 * @return the filesystem used by Hadoop.
 */
 protected FileSystem getFileSystem() {
 return _fileSystem;
 }
Initialize and clean HDFS before/after each test Clean up and initialize file system before each test private final Path _inputPath = new Path("/input");
 private final Path _cachePath = new Path("/cache");
 private final Path _outputPath = new Path("/output");
 
 @BeforeMethod
 public void beforeMethod(Method method) throws IOException {
 getFileSystem().delete(_inputPath, true);
 getFileSystem().mkdirs(_inputPath);
 getFileSystem().delete(_cachePath, true);
 }
 
 @AfterMethod
 public void afterMethod(Method method) throws IOException {
 getFileSystem().delete(_inputPath, true);
 getFileSystem().delete(_cachePath, true);
 getFileSystem().delete(_outputPath, true);
 }
Run MiniCluster Test Clean up and initialize file system before each test @Test
 public void testBasicWordCountJob() throws IOException, InterruptedException, ClassNotFoundException {
 writeWordCountInput();
 configureAndRunJob(new BasicWordCountJob(), "BasicWordCountJob", _inputPath, _outputPath);
 checkWordCountOutput();
 } private void configureAndRunJob(AbstractJob job, String name, Path inputPath, Path outputPath) throws IOException, ClassNotFoundException, InterruptedException {
 Properties _props = new Properties();
 _props.setProperty("input.path", inputPath.toString());
 _props.setProperty("output.path", outputPath.toString());
 job.setProperties(_props);
 job.setName(name);
 job.run();
 }
MRUnit setup Setup MapDriver and ReduceDriver BasicWordCountJob.Map mapper;
 BasicWordCountJob.Reduce reducer;
 MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
 ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
 
 @BeforeClass
 public void setup() {
 mapper = new BasicWordCountJob.Map();
 mapDriver = MapDriver.newMapDriver(mapper);
 reducer = new BasicWordCountJob.Reduce();
 reduceDriver = ReduceDriver.newReduceDriver(reducer);
 }
Run MRUnit test Set Input/Output and run Test @Test
 public void testMapper() throws IOException {
 mapDriver.withInput(new LongWritable(0), new Text("banana pear banana"));
 mapDriver.withOutput(new Text("banana"), new IntWritable(1));
 mapDriver.withOutput(new Text("pear"), new IntWritable(1));
 mapDriver.withOutput(new Text("banana"), new IntWritable(1));
 mapDriver.runTest();
 }
 
 @Test
 public void testReducer() throws IOException {
 reduceDriver.withInput(new Text("banana"), Arrays.asList(new IntWritable(1), new IntWritable(1)));
 reduceDriver.withInput(new Text("pear"), Arrays.asList(new IntWritable(1)));
 reduceDriver.withOutput(new Text("banana"), new IntWritable(2));
 reduceDriver.withOutput(new Text("pear"), new IntWritable(1));
 reduceDriver.runTest();
 }
Most common pitfall ▸ With both MiniMRCluster and MRUnit one spend most of the time ▸ Creating fake input data ▸ Verifying output data ▸ Solutions ▸ Use rich data structures format (e.g. Avro, Thrift) ▸ Use automated Java classes generation
Other common pitfalls ▸ MiniMRCluster ▸ Enable Hadoop INFO logging so you can see real job failure causes ▸ Beware of partitioning or sorting issues unrevealed when testing with too few rows and number of nodes ▸ The API has changed over the years, difficult to find examples ▸ MRUnit ▸ Custom serialization issues (e.g. Avro, Thrift)
Pig Testing
Introducing PigUnit ▸ PigUnit ▸ Official library to unit tests Pig script ▸ Ships with Pig (latest version 0.15.0) ▸ The principle is easy 1. Generate test data 2. Run script with PigUnit 3. Verify output ▸ Runs locally but can be run on a cluster too
Script example WordCount example ‣ Input and output are standard formats ‣ Uses variables $input and $output text = LOAD '$input' USING TextLoader();
 
 flattened = FOREACH text GENERATE flatten(TOKENIZE((chararray)$0)) as word;
 grouped = GROUP flattened by word;
 result = FOREACH grouped GENERATE group, (int)COUNT($1) AS cnt;
 sorted = ORDER result BY cnt DESC;
 
 STORE sorted INTO '$output' USING PigStorage('t');
PigTestBase Create PigTest object protected final FileSystem _fileSystem;
 
 protected PigTestBase() {
 System.setProperty("udf.import.list", StringUtils.join(Arrays.asList("oink.", "org.apache.pig.piggybank."), ":"));
 fileSystem = FileSystem.get(new Configuration());
 }
 
 /**
 * Creates a new <em>PigTest</em> instance ready to be used.
 *
 * @param scriptFile the path to the Pig script file
 * @param inputs the Pig arguments
 * @return new PigTest instance
 */
 protected PigTest newPigTest(String scriptFile, String[] inputs) {
 PigServer pigServer = new PigServer(ExecType.LOCAL);
 Cluster pigCluster = new Cluster(pigServer.getPigContext());
 return new PigTest(scriptFile, inputs, pigServer, pigCluster);
 }
Test using aliases getAlias() allows to obtain the data anywhere in the script @Test
 public void testWordCountAlias() throws IOException, ParseException {
 //Write input data
 BufferedWriter writer = new BufferedWriter(new FileWriter(new File("input.txt")));
 writer.write("banana pear banana");
 writer.close();
 
 PigTest t = newPigTest("pig/src/main/pig/wordcount_text.pig", new String[] {"input=input.txt", "output=result.csv"});
 
 Iterator<Tuple> tuples = t.getAlias("sorted");
 Assert.assertTrue(tuples.hasNext());
 Tuple tuple = tuples.next();
 Assert.assertEquals(tuple.get(0), "banana");
 Assert.assertEquals(tuple.get(1), 2);
 Assert.assertTrue(tuples.hasNext());
 tuple = tuples.next();
 Assert.assertEquals(tuple.get(0), "pear");
 Assert.assertEquals(tuple.get(1), 1);
 }
Test using mock and assert ▸ mockAlias allows to substitute input data ▸ assertOutput allows to compare String output data @Test
 public void testWordCountMock() throws IOException, ParseException {
 //Write input data
 BufferedWriter writer = new BufferedWriter(new FileWriter(new File("input.txt")));
 writer.write("banana pear banana");
 writer.close();
 
 PigTest t = newPigTest("pig/src/main/pig/wordcount_text.pig", new String[] {"input=input.txt", "output=null"});
 t.runScript();
 t.assertOutputAnyOrder("sorted", new String[]{"(banana,2)", "(pear,1)"});
 }
Both of these tools have limitations ▸ Built around standard input and output (Text, CSVs etc.) ▸ Realistically most of our data is in other formats (e.g. Avro, Thrift, JSON) ▸ Does not test the STORE function (e.g. schema errors) ▸ getAlias() is especially difficult to use ▸ Need to remember field position: tuple.get(0) ▸ assertOutput() only allows String comparison ▸ Cumbersome to write complex structures (e.g. bags of bags)
Example with Avro input/output ▸ Focus on testing script’s output ▸ Difficulty is to generate dummy Avro data and compare result text = LOAD '$input' USING AvroStorage();
 
 flattened = FOREACH text GENERATE flatten(TOKENIZE(body)) as word;
 grouped = GROUP flattened by word;
 result = FOREACH grouped GENERATE group AS word, (int)COUNT($1) AS cnt;
 sorted = ORDER result BY cnt DESC;
 
 STORE result INTO '$output' USING AvroStorage(); ▸ By default, PigUnit doesn’t execute the STORE, but it can be overridden pigTest.unoverride("STORE");
Simple utility classes for Avro ▸ BasicAvroWriter ▸ Writes Avro file on disk based on a schema ▸ Supports GenericRecord and SpecificRecord ▸ BasicAvroReader ▸ Reads Avro file, the schema heads the file ▸ Also supports GenericRecord and SpecificRecord
Test with Avro GenericRecord ▸ Create Schema with SchemaBuilder, write data, run script, read result and compare @Test
 public void testWordCountGenericRecord() throws IOException, ParseException {
 Schema schema = SchemaBuilder.builder().record("record").fields().
 name("text").type().stringType().noDefault().endRecord();
 GenericRecord genericRecord = new GenericData.Record(schema);
 genericRecord.put("text", "banana apple banana");
 
 BasicAvroWriter writer = new BasicAvroWriter(new Path(new File("input.avro").getAbsolutePath()), schema, getFileSystem());
 writer.append(genericRecord);
 
 PigTest t = newPigTest("pig/src/main/pig/wordcount_avro.pig", new String[] {"input=input.avro", "output=sorted.avro"});
 t.unoverride("STORE");
 t.runScript();
 
 //Check output
 BasicAvroReader reader = new BasicAvroReader(new Path(new File("sorted.avro").getAbsolutePath()), getFileSystem());
 Map<Utf8, GenericRecord> result = reader.readAndMapAll("word");
 Assert.assertEquals(result.size(), 2);
 Assert.assertEquals(result.get(new Utf8("banana")).get("cnt"), 2);
 Assert.assertEquals(result.get(new Utf8("apple")).get("cnt"), 1);
 }
Test with Avro SpecificRecord ▸ Use InputRecord and OutputRecord generated Java classes, write data, run script, read result and compare @Test
 public void testWordCountSpecificRecord() throws IOException, ParseException {
 InputRecord input = InputRecord.newBuilder().setText("banana apple banana").build();
 
 BasicAvroWriter<InputRecord> writer = new BasicAvroWriter<InputRecord>(new Path(new File("input.avro").getAbsolutePath()), input.getSchema(), getFileSystem());
 writer.writeAll(input);
 
 PigTest t = newPigTest("pig/src/main/pig/wordcount_avro.pig", new String[] {"input=input.avro", "output=sorted.avro"});
 t.unoverride("STORE");
 t.runScript();
 
 //Check output
 BasicAvroReader<OutputRecord> reader = new BasicAvroReader<OutputRecord>(new Path(new File("sorted.avro").getAbsolutePath()), getFileSystem());
 List<OutputRecord> result = reader.readAll();
 Assert.assertEquals(result.size(), 2);
 Assert.assertEquals(result.get(0), OutputRecord.newBuilder().setWord("banana").setCount(2).build());
 Assert.assertEquals(result.get(1), OutputRecord.newBuilder().setWord("apple").setCount(1).build());
 }
Common pitfalls ▸ PigUnit ▸ Mocking capabilities are very limited ▸ Overhead of 1-5 seconds per script ▸ Cryptic error messages sometimes (NullPointerException) ▸ Pig UDFs ▸ Can be tested independently
Spark Testing
Spark Testing Base Base classes to use when writing tests with Spark ▸ https://github.com/holdenk/spark-testing-base ▸ Functionalities ▸ Provides SparkContext ▸ Utilities to compare RDDs and DataFrames ▸ Simulate how Streaming works ▸ Includes cool RDD and DataFrames generator
Thank You! We are hiring! http://careers.getyourguide.com/
Extra Resources ▸ https://github.com/miguno/avro-hadoop-starter ▸ http://www.michael-noll.com/blog/2013/07/04/using-avro-in-mapreduce-jobs-with-hadoop-pig-hive/ ▸ http://blog.cloudera.com/blog/2015/09/making-apache-spark-testing-easy-with-spark-testing-base/ ▸ http://www.slideshare.net/hkarau/effective-testing-for-spark-programs-strata-ny-2015 ▸ http://avro.apache.org/docs/current/ ▸ http://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you-really-need-one ▸ http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/ ▸ https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about- real-time-datas-unifying
Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/patterns- data-workflows

Recommended

Automated Testing For Protecting Data Pipelines from Undocumented Assumptions
Automated Testing For Protecting Data Pipelines from Undocumented AssumptionsAutomated Testing For Protecting Data Pipelines from Undocumented Assumptions
Automated Testing For Protecting Data Pipelines from Undocumented Assumptions
Databricks
 
Test strategies for data processing pipelines
Test strategies for data processing pipelinesTest strategies for data processing pipelines
Test strategies for data processing pipelines
Lars Albertsson
 
KSQL - Stream Processing simplified!
KSQL - Stream Processing simplified!KSQL - Stream Processing simplified!
KSQL - Stream Processing simplified!
Guido Schmutz
 
Migrating from RDBMS to MongoDB
Migrating from RDBMS to MongoDBMigrating from RDBMS to MongoDB
Migrating from RDBMS to MongoDB
MongoDB
 
Autorização de transações no Nubank
Autorização de transações no NubankAutorização de transações no Nubank
Autorização de transações no Nubank
Luiz Alberto Hespanha
 
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GCHadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Erik Krogen
 
Unit testing best practices
Unit testing best practicesUnit testing best practices
Unit testing best practices
nickokiss
 
bigquery.pptx
bigquery.pptxbigquery.pptx
bigquery.pptx
Harissh16
 

Recommended

Automated Testing For Protecting Data Pipelines from Undocumented Assumptions
Automated Testing For Protecting Data Pipelines from Undocumented AssumptionsAutomated Testing For Protecting Data Pipelines from Undocumented Assumptions
Automated Testing For Protecting Data Pipelines from Undocumented Assumptions
Databricks
 
Test strategies for data processing pipelines
Test strategies for data processing pipelinesTest strategies for data processing pipelines
Test strategies for data processing pipelines
Lars Albertsson
 
KSQL - Stream Processing simplified!
KSQL - Stream Processing simplified!KSQL - Stream Processing simplified!
KSQL - Stream Processing simplified!
Guido Schmutz
 
Migrating from RDBMS to MongoDB
Migrating from RDBMS to MongoDBMigrating from RDBMS to MongoDB
Migrating from RDBMS to MongoDB
MongoDB
 
Autorização de transações no Nubank
Autorização de transações no NubankAutorização de transações no Nubank
Autorização de transações no Nubank
Luiz Alberto Hespanha
 
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GCHadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Erik Krogen
 
Unit testing best practices
Unit testing best practicesUnit testing best practices
Unit testing best practices
nickokiss
 
bigquery.pptx
bigquery.pptxbigquery.pptx
bigquery.pptx
Harissh16
 
ATDD - Desarrollo Dirigido por Test de Aceptación
ATDD - Desarrollo Dirigido por Test de AceptaciónATDD - Desarrollo Dirigido por Test de Aceptación
ATDD - Desarrollo Dirigido por Test de Aceptación
Paulo Clavijo
 
Grafana introduction
Grafana introductionGrafana introduction
Grafana introduction
Rico Chen
 
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDBHow to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
InfluxData
 
Boost Performance With My S Q L 51 Partitions
Boost Performance With My S Q L 51 PartitionsBoost Performance With My S Q L 51 Partitions
Boost Performance With My S Q L 51 Partitions
PerconaPerformance
 
MySQL for Large Scale Social Games
MySQL for Large Scale Social GamesMySQL for Large Scale Social Games
MySQL for Large Scale Social Games
Yoshinori Matsunobu
 
Migration from Oracle to PostgreSQL: NEED vs REALITY
Migration from Oracle to PostgreSQL: NEED vs REALITYMigration from Oracle to PostgreSQL: NEED vs REALITY
Migration from Oracle to PostgreSQL: NEED vs REALITY
Ashnikbiz
 
Scaling Redis To 1M Ops/Sec: Jane Paek
Scaling Redis To 1M Ops/Sec: Jane PaekScaling Redis To 1M Ops/Sec: Jane Paek
Scaling Redis To 1M Ops/Sec: Jane Paek
Redis Labs
 
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
ScyllaDB
 
Distributed Load Testing with k6 - DevOps Barcelona
Distributed Load Testing with k6 - DevOps BarcelonaDistributed Load Testing with k6 - DevOps Barcelona
Distributed Load Testing with k6 - DevOps Barcelona
Thijs Feryn
 
Logical Replication in PostgreSQL
Logical Replication in PostgreSQLLogical Replication in PostgreSQL
Logical Replication in PostgreSQL
EDB
 
Introduction to Structured Streaming
Introduction to Structured StreamingIntroduction to Structured Streaming
Introduction to Structured Streaming
Knoldus Inc.
 
Bucket your partitions wisely - Cassandra summit 2016
Bucket your partitions wisely - Cassandra summit 2016Bucket your partitions wisely - Cassandra summit 2016
Bucket your partitions wisely - Cassandra summit 2016
Markus Höfer
 
QuestDB: ingesting a million time series per second on a single instance. Big...
QuestDB: ingesting a million time series per second on a single instance. Big...QuestDB: ingesting a million time series per second on a single instance. Big...
QuestDB: ingesting a million time series per second on a single instance. Big...
javier ramirez
 
Taking Testing to the Cloud
Taking Testing to the CloudTaking Testing to the Cloud
Taking Testing to the Cloud
Cognizant
 
Aerospike: Key Value Data Access
Aerospike: Key Value Data AccessAerospike: Key Value Data Access
Aerospike: Key Value Data Access
Aerospike, Inc.
 
SQL Server Alta disponibilidad en ambientes empresariales
SQL Server Alta disponibilidad en ambientes empresarialesSQL Server Alta disponibilidad en ambientes empresariales
SQL Server Alta disponibilidad en ambientes empresariales
Eduardo Castro
 
Exploratory testing workshop
Exploratory testing workshopExploratory testing workshop
Exploratory testing workshop
Anne-Marie Charrett
 
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdfMySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
Alkin Tezuysal
 
Engineering data quality
Engineering data qualityEngineering data quality
Engineering data quality
Lars Albertsson
 
Fluentd loves MongoDB, at MongoDB SV User Group, July 17, 2012
Fluentd loves MongoDB, at MongoDB SV User Group, July 17, 2012Fluentd loves MongoDB, at MongoDB SV User Group, July 17, 2012
Fluentd loves MongoDB, at MongoDB SV User Group, July 17, 2012
Treasure Data, Inc.
 
The Mechanics of Testing Large Data Pipelines (QCon London 2016)
The Mechanics of Testing Large Data Pipelines (QCon London 2016)The Mechanics of Testing Large Data Pipelines (QCon London 2016)
The Mechanics of Testing Large Data Pipelines (QCon London 2016)
Mathieu Bastian
 
StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?
mabl
 

More Related Content

What's hot

How to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDBHow to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
InfluxData
 
Boost Performance With My S Q L 51 Partitions
Boost Performance With My S Q L 51 PartitionsBoost Performance With My S Q L 51 Partitions
Boost Performance With My S Q L 51 Partitions
PerconaPerformance
 
Scaling Redis To 1M Ops/Sec: Jane Paek
Scaling Redis To 1M Ops/Sec: Jane PaekScaling Redis To 1M Ops/Sec: Jane Paek
Scaling Redis To 1M Ops/Sec: Jane Paek
Redis Labs
 
Engineering data quality
Engineering data qualityEngineering data quality
Engineering data quality
Lars Albertsson
 

What's hot (20)

ATDD - Desarrollo Dirigido por Test de Aceptación
ATDD - Desarrollo Dirigido por Test de AceptaciónATDD - Desarrollo Dirigido por Test de Aceptación
ATDD - Desarrollo Dirigido por Test de Aceptación
 
Grafana introduction
Grafana introductionGrafana introduction
Grafana introduction
 
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDBHow to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
 
Boost Performance With My S Q L 51 Partitions
Boost Performance With My S Q L 51 PartitionsBoost Performance With My S Q L 51 Partitions
Boost Performance With My S Q L 51 Partitions
 
MySQL for Large Scale Social Games
MySQL for Large Scale Social GamesMySQL for Large Scale Social Games
MySQL for Large Scale Social Games
 
Migration from Oracle to PostgreSQL: NEED vs REALITY
Migration from Oracle to PostgreSQL: NEED vs REALITYMigration from Oracle to PostgreSQL: NEED vs REALITY
Migration from Oracle to PostgreSQL: NEED vs REALITY
 
Scaling Redis To 1M Ops/Sec: Jane Paek
Scaling Redis To 1M Ops/Sec: Jane PaekScaling Redis To 1M Ops/Sec: Jane Paek
Scaling Redis To 1M Ops/Sec: Jane Paek
 
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
 
Distributed Load Testing with k6 - DevOps Barcelona
Distributed Load Testing with k6 - DevOps BarcelonaDistributed Load Testing with k6 - DevOps Barcelona
Distributed Load Testing with k6 - DevOps Barcelona
 
Logical Replication in PostgreSQL
Logical Replication in PostgreSQLLogical Replication in PostgreSQL
Logical Replication in PostgreSQL
 
Introduction to Structured Streaming
Introduction to Structured StreamingIntroduction to Structured Streaming
Introduction to Structured Streaming
 
Bucket your partitions wisely - Cassandra summit 2016
Bucket your partitions wisely - Cassandra summit 2016Bucket your partitions wisely - Cassandra summit 2016
Bucket your partitions wisely - Cassandra summit 2016
 
QuestDB: ingesting a million time series per second on a single instance. Big...
QuestDB: ingesting a million time series per second on a single instance. Big...QuestDB: ingesting a million time series per second on a single instance. Big...
QuestDB: ingesting a million time series per second on a single instance. Big...
 
Taking Testing to the Cloud
Taking Testing to the CloudTaking Testing to the Cloud
Taking Testing to the Cloud
 
Aerospike: Key Value Data Access
Aerospike: Key Value Data AccessAerospike: Key Value Data Access
Aerospike: Key Value Data Access
 
SQL Server Alta disponibilidad en ambientes empresariales
SQL Server Alta disponibilidad en ambientes empresarialesSQL Server Alta disponibilidad en ambientes empresariales
SQL Server Alta disponibilidad en ambientes empresariales
 
Exploratory testing workshop
Exploratory testing workshopExploratory testing workshop
Exploratory testing workshop
 
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdfMySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
 
Engineering data quality
Engineering data qualityEngineering data quality
Engineering data quality
 
Fluentd loves MongoDB, at MongoDB SV User Group, July 17, 2012
Fluentd loves MongoDB, at MongoDB SV User Group, July 17, 2012Fluentd loves MongoDB, at MongoDB SV User Group, July 17, 2012
Fluentd loves MongoDB, at MongoDB SV User Group, July 17, 2012
 

Similar to The Mechanics of Testing Large Data Pipelines

The Mechanics of Testing Large Data Pipelines (QCon London 2016)
The Mechanics of Testing Large Data Pipelines (QCon London 2016)The Mechanics of Testing Large Data Pipelines (QCon London 2016)
The Mechanics of Testing Large Data Pipelines (QCon London 2016)
Mathieu Bastian
 
StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?
mabl
 
What are Software Testing Methodologies | Software Testing Techniques | Edureka
What are Software Testing Methodologies | Software Testing Techniques | EdurekaWhat are Software Testing Methodologies | Software Testing Techniques | Edureka
What are Software Testing Methodologies | Software Testing Techniques | Edureka
Edureka!
 
End to-end test automation at scale
End to-end test automation at scaleEnd to-end test automation at scale
End to-end test automation at scale
mabl
 
Testing Big Data solutions fast and furiously
Testing Big Data solutions fast and furiouslyTesting Big Data solutions fast and furiously
Testing Big Data solutions fast and furiously
Katherine Golovinova
 
How to Automate your Enterprise Application / ERP Testing
How to Automate your Enterprise Application / ERP TestingHow to Automate your Enterprise Application / ERP Testing
How to Automate your Enterprise Application / ERP Testing
RTTS
 
QA Fest 2019. Дмитрий Собко. Testing Big Data solutions fast and furiously
QA Fest 2019. Дмитрий Собко. Testing Big Data solutions fast and furiouslyQA Fest 2019. Дмитрий Собко. Testing Big Data solutions fast and furiously
QA Fest 2019. Дмитрий Собко. Testing Big Data solutions fast and furiously
QAFest
 
Building functional Quality Gates with ReportPortal
Building functional Quality Gates with ReportPortalBuilding functional Quality Gates with ReportPortal
Building functional Quality Gates with ReportPortal
Dmitriy Gumeniuk
 
How to build an automated customer data onboarding pipeline
How to build an automated customer data onboarding pipelineHow to build an automated customer data onboarding pipeline
How to build an automated customer data onboarding pipeline
CloverDX
 
Test Requirements
Test RequirementsTest Requirements
Test Requirements
telab
 

Similar to The Mechanics of Testing Large Data Pipelines (20)

The Mechanics of Testing Large Data Pipelines (QCon London 2016)
The Mechanics of Testing Large Data Pipelines (QCon London 2016)The Mechanics of Testing Large Data Pipelines (QCon London 2016)
The Mechanics of Testing Large Data Pipelines (QCon London 2016)
 
StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?
 
Software Quality and Test Strategies for Ruby and Rails Applications
Software Quality and Test Strategies for Ruby and Rails ApplicationsSoftware Quality and Test Strategies for Ruby and Rails Applications
Software Quality and Test Strategies for Ruby and Rails Applications
 
What are Software Testing Methodologies | Software Testing Techniques | Edureka
What are Software Testing Methodologies | Software Testing Techniques | EdurekaWhat are Software Testing Methodologies | Software Testing Techniques | Edureka
What are Software Testing Methodologies | Software Testing Techniques | Edureka
 
End to-end test automation at scale
End to-end test automation at scaleEnd to-end test automation at scale
End to-end test automation at scale
 
Testing Big Data solutions fast and furiously
Testing Big Data solutions fast and furiouslyTesting Big Data solutions fast and furiously
Testing Big Data solutions fast and furiously
 
How to Automate your Enterprise Application / ERP Testing
How to Automate your Enterprise Application / ERP TestingHow to Automate your Enterprise Application / ERP Testing
How to Automate your Enterprise Application / ERP Testing
 
QA Fest 2019. Дмитрий Собко. Testing Big Data solutions fast and furiously
QA Fest 2019. Дмитрий Собко. Testing Big Data solutions fast and furiouslyQA Fest 2019. Дмитрий Собко. Testing Big Data solutions fast and furiously
QA Fest 2019. Дмитрий Собко. Testing Big Data solutions fast and furiously
 
Lightning Talks by Globant - Automation (This app runs by itself )
Lightning Talks by Globant - Automation (This app runs by itself ) Lightning Talks by Globant - Automation (This app runs by itself )
Lightning Talks by Globant - Automation (This app runs by itself )
 
Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)
Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)
Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)
 
#DOAW16 - DevOps@work Roma 2016 - Testing your databases
#DOAW16 - DevOps@work Roma 2016 - Testing your databases#DOAW16 - DevOps@work Roma 2016 - Testing your databases
#DOAW16 - DevOps@work Roma 2016 - Testing your databases
 
Shuvam dutta | Performance testing & engineering
Shuvam dutta | Performance testing & engineeringShuvam dutta | Performance testing & engineering
Shuvam dutta | Performance testing & engineering
 
Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)
 
Resume sailaja
Resume sailajaResume sailaja
Resume sailaja
 
Shuvam dutta | Performance testing and engineering
Shuvam dutta | Performance testing and engineeringShuvam dutta | Performance testing and engineering
Shuvam dutta | Performance testing and engineering
 
Building functional Quality Gates with ReportPortal
Building functional Quality Gates with ReportPortalBuilding functional Quality Gates with ReportPortal
Building functional Quality Gates with ReportPortal
 
Neotys PAC 2018 - Tingting Zong
Neotys PAC 2018 - Tingting ZongNeotys PAC 2018 - Tingting Zong
Neotys PAC 2018 - Tingting Zong
 
How to build an automated customer data onboarding pipeline
How to build an automated customer data onboarding pipelineHow to build an automated customer data onboarding pipeline
How to build an automated customer data onboarding pipeline
 
Devops (start walking in the same direction) by ops
Devops (start walking in the same direction) by opsDevops (start walking in the same direction) by ops
Devops (start walking in the same direction) by ops
 
Test Requirements
Test RequirementsTest Requirements
Test Requirements
 

More from C4Media

Streaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live VideoStreaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
C4Media
 
Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020
C4Media
 
Understand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java ApplicationsUnderstand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java Applications
C4Media
 
High Performing Teams Act Like Owners
High Performing Teams Act Like OwnersHigh Performing Teams Act Like Owners
High Performing Teams Act Like Owners
C4Media
 
Shifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDShifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CD
C4Media
 
Architectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsArchitectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep Systems
C4Media
 
ML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.js
C4Media
 
Scaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeScaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's Edge
C4Media
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
C4Media
 

More from C4Media (20)

Streaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live VideoStreaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
 
Next Generation Client APIs in Envoy Mobile
Next Generation Client APIs in Envoy MobileNext Generation Client APIs in Envoy Mobile
Next Generation Client APIs in Envoy Mobile
 
Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020
 
Understand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java ApplicationsUnderstand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java Applications
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No Keeper
 
High Performing Teams Act Like Owners
High Performing Teams Act Like OwnersHigh Performing Teams Act Like Owners
High Performing Teams Act Like Owners
 
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to JavaDoes Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
 
Service Meshes- The Ultimate Guide
Service Meshes- The Ultimate GuideService Meshes- The Ultimate Guide
Service Meshes- The Ultimate Guide
 
Shifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDShifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CD
 
CI/CD for Machine Learning
CI/CD for Machine LearningCI/CD for Machine Learning
CI/CD for Machine Learning
 
Fault Tolerance at Speed
Fault Tolerance at SpeedFault Tolerance at Speed
Fault Tolerance at Speed
 
Architectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsArchitectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep Systems
 
ML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.js
 
Build Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerBuild Your Own WebAssembly Compiler
Build Your Own WebAssembly Compiler
 
User & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleUser & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix Scale
 
Scaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeScaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's Edge
 
Make Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereMake Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home Everywhere
 
The Talk You've Been Await-ing For
The Talk You've Been Await-ing ForThe Talk You've Been Await-ing For
The Talk You've Been Await-ing For
 
Future of Data Engineering
Future of Data EngineeringFuture of Data Engineering
Future of Data Engineering
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
 

Recently uploaded

In-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT ProfessionalsIn-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT Professionals
Expeed Software
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
Ralf Eggert
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
David Michel
 

Recently uploaded (20)

Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
In-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT ProfessionalsIn-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT Professionals
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 

The Mechanics of Testing Large Data Pipelines

  • 1. THE MECHANICS OF TESTING LARGE DATA PIPELINES MATHIEU BASTIAN Head of Data Engineering, GetYourGuide @mathieubastian www.linkedin.com/in/mathieubastian QCon London 2015
  • 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /patterns-data-workflows
  • 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
  • 4. Outline ▸ Motivating example ▸ Challenges ▸ Testing strategies ▸ Validation Strategies ▸ Tools Integration Tests ArchitectureUnit Test
  • 5. Data Pipelines often start simple
  • 6. Users E-commerce website Search App Views Offline Dashboard Search Metrics Views HDFS They have one use-case and one developer
  • 7. But there are many other use- cases Recommender Systems Anomaly Detection Search Ranking A/B Testing Spam Detection Sentiment Analysis Topic Detection Trending TagsQuery Expansion Customer Churn Prediction Related searches Fraud Prediction Bidding Prediction Machine Translation Signal Processing Content Curation Sentiment Analysis Image recognition Optimal pricing Location normalization Standardization Funnel Analysis
  • 8. additional events and logsDevelopers add Users E-commerce website Search App Clicks Views Offline Dashboard Search Metrics Clicks Views HDFS
  • 9. third-party dataDevelopers add Users E-commerce website 3rd parties Search App Clicks Views A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views A/B Logs HDFS
  • 10. search ranking predictionDevelopers add Users E-commerce website 3rd parties Search App Clicks Views A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views Training data Training & validation Model Clicks Views Features transformation A/B Logs HDFS
  • 11. personalized user featuresDevelopers add Users E-commerce website 3rd parties Search App Clicks Views ProfilesUser Database A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views Training data Training & validation Model Clicks Views Profiles Features transformation A/B Logs HDFS
  • 12. query extensionDevelopers add Users E-commerce website 3rd parties Search App Clicks Views ProfilesUser Database A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views Training data Training & validation Model Clicks Views Profiles Features transformation A/B Logs Filter queries Query extension RDBMS Views Training data HDFS
  • 13. Developers add recommender system Users E-commerce website 3rd parties Search App Clicks Views ProfilesUser Database A/B LogsMobile Analytics Offline Dashboard Search Metrics Clicks Views Training data Training & validation Model Clicks Views Profiles Features Features transformation Features NoSQL Compute recommendations A/B Logs Filter queries Query extension RDBMS Views Training data HDFS
  • 14. Data Pipelines can grow very large
  • 15. That is a lot of code and data
  • 16. Code contain bugs Industry Average: about 15 - 50 errors per 1000 lines of delivered code.
  • 18. Embrace automated testing of code validation of data
  • 19. Because it delivers ▸ Testing ▸ Tested code has less bugs ▸ Gives the confidence to iterate quickly ▸ Scales well to multiple developers ▸ Validation ▸ Reduce manual testing ▸ Avoid catastrophic failures
  • 20. But it’s challenging ▸ Testing ▸ Need data to test "realistically" ▸ Not running locally, can be expensive ▸ Tooling weaknesses ▸ Validation ▸ Data sources out of our control ▸ Difficult to test machine learning models
  • 22. Manual testing Waiting Coding Looking at logs Code Upload Run workflow Look at logs ▸ Time Spent
  • 24. Prepare environment ▸ Care about tests from the start of your project ▸ All jobs should be functions (output only depends on input) ▸ Safe to re-run the job ▸ Does the input data still exists? ▸ Would it push partial results? ▸ Centralize configurations and no hard-coded paths ▸ Version code and timestamp data
  • 25. Unit test locally ▸ Test locally each individual job ▸ Tests its good code ▸ Tests expected failures ▸ Need to overcome challenges with fake data creation ▸ Complex structures and numerous data sources ▸ Too small to be meaningful ▸ Need to specify a different configuration
  • 26. Build from schemas Fake data creation based on schemas. Compare: Customer c = Customer.newBuilder().
 setId(42).
 setInterests(Arrays.asList(new Interest[]{
 Interest.newBuilder().setId(0).setName("Ping-Pong").build()
 Interest.newBuilder().setId(1).setName(“Pizza").build()})) .build(); vs Map<String, Object> c = new HashMap<>();
 c.put("id", 42);
 Map<String, Object> i1 = new HashMap<>();
 i1.put("id", 0);
 i1.put("name", "Ping-Pong");
 Map<String, Object> i2 = new HashMap<>();
 i2.put("id", 1);
 i2.put("name", "Pizza");
 c.put("interests", Arrays.asList(new Map[] {i1, i2}));
  • 27. Build from schemas Avro Schema example { "type": "record", "name": "Customer", "fields": [{ "name": "id", "type": "int" }, { "name": "interests", "type": { "type": "array", "items": { "name": "Interest", "type": "record", "fields": [{ "name": "id", "type": "int" }, { "name": "name", "type": ["string", "null"] }] } } } ] } nullable field
  • 28. Complex generators ▸ Developed in the field of property-based testing //Small Even Number Generator val smallEvenInteger = Gen.choose(0,200) suchThat (_ % 2 == 0) ▸ Goal is to simulate, not sample real data ▸ Define complex random generators that match properties (e.g. frequency) ▸ Can go beyond unit-testing and generate complex domain models ▸ https://www.scalacheck.org/ for Scala/Java is a good starting point for examples
  • 29. Integration test on sample data ▸ Integration test the entire workflow ▸ File paths ▸ Configuration ▸ Evaluate performance ▸ Sample data ▸ Large enough to be meaningful ▸ Small enough to speed-up testing JOB A JOB B JOB C JOB D
  • 31. Where it fail Control Difficulty Model biases Bug Noisy data Schema changes Missing data
  • 32. Input and output validation Make the pipeline robust by validating inputs and outputs Input Input Input Workflow Production ValidationValidation
  • 34. Input data validation Input data validation is a key component of pipeline robustness. The goal is to test the entry points of our system for data quality. ETL RDBMS NOSQL EVENTS TWITTER DATA PIPELINE
  • 35. Why it matters ▸ Bad input data will most likely degrade the output ▸ It likely will fail silently ▸ Because data will change ▸ Data migrations: maintenance, cluster update, new infrastructure ▸ Events change due to product evolution ▸ Data dependencies updated
  • 36. Input data validation ▸ Validation code should ▸ Detect pathological data and fail early ▸ Deal with expected data variability ▸ Example issues: ▸ Missing values, encoding issues, etc. ▸ Schema changes ▸ Duplicates rows ▸ Data order changes
  • 37. Pathological data ▸ Value ▸ Validity depends on a single, independent value. ▸ Easy to validate on streams of data ▸ Dataset ▸ Validity depends on the entire dataset ▸ More difficult to validate as it needs a window of data
  • 38. Metadata validation Analyzing metadata is the quickest way to validate input data ▸ Number of records and file sizes ▸ Hadoop/Spark counters ▸ Number of map/reduce records, size ▸ Record-level custom counters ▸ Average text length ▸ Task-level custom counters ▸ Min/Max/Median values
  • 39. Hadoop/Spark counters Results can be accessed programmatically and checked
  • 40. Control inputs with Schemas ▸ CSVs aren’t robust to change, use Schemas ▸ Makes expected data explicit and easy to test against ▸ Gives basic validation for free with binary serialization (e.g. Avro, Thrift, Protocol Buffer) ▸ Typed (integer, boolean, lists etc.) ▸ Specify if value is optional ▸ Schemas can be evolved without breaking compatibility
  • 42. Why it matters ▸ Humans makes mistake, we need a safeguard ▸ Rolling back data is often complex ▸ Bad output propagates to downstream systems Example with a recommender system // One recommendation set per user { "userId": 42, "recommendations": [{ "itemId": 1456, "score": 0.9 }, { "itemId": 4232, "score": 0.1 }], "model": "test01" }
  • 43. Check for anomalies Simple strategies similar to input data validation ▸ Record level (e.g. values within bounds) ▸ Dataset level (e.g. counts, order) Challenges around relevance evaluation ▸ When supervised, use a validation dataset and threshold accuracy ▸ Introduce hypothetical examples
  • 44. Incremental update as validation Join with the previous “best" output ▸ Allows fine comparisons ▸ Incremental framework can be extended to ▸ Only recompute recommendations that have changed ▸ Produce variations metric between different models Daily Recommendations Compute daily recommendationsHDFS Recommendations Yesterday Recommendations Join with previous result
  • 45. External validation Even in automated environment it is possible to validate with humans ▸ Example: Search ranking evaluation ▸ Solution: Crowdsourcing ▸ Complex validation that requires training ▸ Can be automated through APIs
  • 46. Mitigate risk with A/B testing Gradually rolling out data products improvements reduces the need for complex output validation ▸ Experiment can be controlled online or offline ▸ Online: Push multiple set of recommendations (1 per model) ▸ Offline: Split users and push unique set of recommendations userId -> [{
 "model": "test01",
 "recommendations": [{...}]
 }, {
 "model": "test02",
 "recommendations": [{...}]
 }] A B userId -> {
 "model": "test01",
 "recommendations": [{...}]
 }
  • 47. Mitigate risk with A/B testing Important ▸ Log model variation downstream in logs ▸ Encapsulate model logic FEATURE 1-A MODEL A MODEL B FEATURE 1 FEATURE 2 MODEL A MODEL B A BA B FEATURE 1-B FEATURE 2-A FEATURE 2-B
  • 48. Thank You! We are hiring! http://careers.getyourguide.com/
  • 51. Two ways to test Hadoop jobs ▸ MRUnit ▸ Java library to test MapReduce jobs in a simulated environment ▸ Last release June 2014 ▸ MiniCluster ▸ Utility to locally run a fully-functional Hadoop cluster in a test environment ▸ Ships with Hadoop itself
  • 52. MiniMRCluster ▸ Advantages ▸ Behaves like a real cluster, including setup and configuration ▸ Can be used to test multiple jobs (integration testing) ▸ Disadvantages ▸ Very slow compared to unit testing Java code
  • 53. MRUnit ▸ Advantages ▸ Faster ▸ Less boilerplate code ▸ Disadvantages ▸ Need to replicate job configuration ▸ Only built to test map and reduce functions ▸ Difficult to make it work with custom input formats (e.g. Avro)
  • 54. MiniMRCluster setup* Setup MR cluster and obtain FileSystem @BeforeClass
 public void setup() {
 Configuration dfsConf = new Configuration();
 dfsConf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, new File("./target/ hdfs/").getAbsolutePath());
 _dfsCluster = new MiniDFSCluster.Builder(dfsConf).numDataNodes(1).build();
 _dfsCluster.waitClusterUp();
 _fileSystem = _dfsCluster.getFileSystem();
 
 YarnConfiguration yarnConf = new YarnConfiguration();
 yarnConf.setFloat(YarnConfiguration.NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE, 99.0f);
 yarnConf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 64);
 yarnConf.setClass(YarnConfiguration.RM_SCHEDULER, FifoScheduler.class, ResourceScheduler.class);
 _mrCluster = new MiniMRYarnCluster(getClass().getName(), taskTrackers);
 yarnConf.set("fs.defaultFS", _fileSystem.getUri().toString());
 _mrCluster.init(yarnConf);
 _mrCluster.start();
 } * Hadoop version used 2.7.2
  • 55. Keep the test file clean of boilerplate code Best is to wrap the start/stop code into a TestBase class /**
 * Default constructor with one task tracker and one node.
 */
 public TestBase() { ... }
 
 @BeforeClass
 public void startCluster() throws IOException { ... }
 
 @AfterClass
 public void stopCluster() throws IOException { ... }
 
 /**
 * Returns the Filesystem in use.
 *
 * @return the filesystem used by Hadoop.
 */
 protected FileSystem getFileSystem() {
 return _fileSystem;
 }
  • 56. Initialize and clean HDFS before/after each test Clean up and initialize file system before each test private final Path _inputPath = new Path("/input");
 private final Path _cachePath = new Path("/cache");
 private final Path _outputPath = new Path("/output");
 
 @BeforeMethod
 public void beforeMethod(Method method) throws IOException {
 getFileSystem().delete(_inputPath, true);
 getFileSystem().mkdirs(_inputPath);
 getFileSystem().delete(_cachePath, true);
 }
 
 @AfterMethod
 public void afterMethod(Method method) throws IOException {
 getFileSystem().delete(_inputPath, true);
 getFileSystem().delete(_cachePath, true);
 getFileSystem().delete(_outputPath, true);
 }
  • 57. Run MiniCluster Test Clean up and initialize file system before each test @Test
 public void testBasicWordCountJob() throws IOException, InterruptedException, ClassNotFoundException {
 writeWordCountInput();
 configureAndRunJob(new BasicWordCountJob(), "BasicWordCountJob", _inputPath, _outputPath);
 checkWordCountOutput();
 } private void configureAndRunJob(AbstractJob job, String name, Path inputPath, Path outputPath) throws IOException, ClassNotFoundException, InterruptedException {
 Properties _props = new Properties();
 _props.setProperty("input.path", inputPath.toString());
 _props.setProperty("output.path", outputPath.toString());
 job.setProperties(_props);
 job.setName(name);
 job.run();
 }
  • 58. MRUnit setup Setup MapDriver and ReduceDriver BasicWordCountJob.Map mapper;
 BasicWordCountJob.Reduce reducer;
 MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
 ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
 
 @BeforeClass
 public void setup() {
 mapper = new BasicWordCountJob.Map();
 mapDriver = MapDriver.newMapDriver(mapper);
 reducer = new BasicWordCountJob.Reduce();
 reduceDriver = ReduceDriver.newReduceDriver(reducer);
 }
  • 59. Run MRUnit test Set Input/Output and run Test @Test
 public void testMapper() throws IOException {
 mapDriver.withInput(new LongWritable(0), new Text("banana pear banana"));
 mapDriver.withOutput(new Text("banana"), new IntWritable(1));
 mapDriver.withOutput(new Text("pear"), new IntWritable(1));
 mapDriver.withOutput(new Text("banana"), new IntWritable(1));
 mapDriver.runTest();
 }
 
 @Test
 public void testReducer() throws IOException {
 reduceDriver.withInput(new Text("banana"), Arrays.asList(new IntWritable(1), new IntWritable(1)));
 reduceDriver.withInput(new Text("pear"), Arrays.asList(new IntWritable(1)));
 reduceDriver.withOutput(new Text("banana"), new IntWritable(2));
 reduceDriver.withOutput(new Text("pear"), new IntWritable(1));
 reduceDriver.runTest();
 }
  • 60. Most common pitfall ▸ With both MiniMRCluster and MRUnit one spend most of the time ▸ Creating fake input data ▸ Verifying output data ▸ Solutions ▸ Use rich data structures format (e.g. Avro, Thrift) ▸ Use automated Java classes generation
  • 61. Other common pitfalls ▸ MiniMRCluster ▸ Enable Hadoop INFO logging so you can see real job failure causes ▸ Beware of partitioning or sorting issues unrevealed when testing with too few rows and number of nodes ▸ The API has changed over the years, difficult to find examples ▸ MRUnit ▸ Custom serialization issues (e.g. Avro, Thrift)
  • 63. Introducing PigUnit ▸ PigUnit ▸ Official library to unit tests Pig script ▸ Ships with Pig (latest version 0.15.0) ▸ The principle is easy 1. Generate test data 2. Run script with PigUnit 3. Verify output ▸ Runs locally but can be run on a cluster too
  • 64. Script example WordCount example ‣ Input and output are standard formats ‣ Uses variables $input and $output text = LOAD '$input' USING TextLoader();
 
 flattened = FOREACH text GENERATE flatten(TOKENIZE((chararray)$0)) as word;
 grouped = GROUP flattened by word;
 result = FOREACH grouped GENERATE group, (int)COUNT($1) AS cnt;
 sorted = ORDER result BY cnt DESC;
 
 STORE sorted INTO '$output' USING PigStorage('t');
  • 65. PigTestBase Create PigTest object protected final FileSystem _fileSystem;
 
 protected PigTestBase() {
 System.setProperty("udf.import.list", StringUtils.join(Arrays.asList("oink.", "org.apache.pig.piggybank."), ":"));
 fileSystem = FileSystem.get(new Configuration());
 }
 
 /**
 * Creates a new <em>PigTest</em> instance ready to be used.
 *
 * @param scriptFile the path to the Pig script file
 * @param inputs the Pig arguments
 * @return new PigTest instance
 */
 protected PigTest newPigTest(String scriptFile, String[] inputs) {
 PigServer pigServer = new PigServer(ExecType.LOCAL);
 Cluster pigCluster = new Cluster(pigServer.getPigContext());
 return new PigTest(scriptFile, inputs, pigServer, pigCluster);
 }
  • 66. Test using aliases getAlias() allows to obtain the data anywhere in the script @Test
 public void testWordCountAlias() throws IOException, ParseException {
 //Write input data
 BufferedWriter writer = new BufferedWriter(new FileWriter(new File("input.txt")));
 writer.write("banana pear banana");
 writer.close();
 
 PigTest t = newPigTest("pig/src/main/pig/wordcount_text.pig", new String[] {"input=input.txt", "output=result.csv"});
 
 Iterator<Tuple> tuples = t.getAlias("sorted");
 Assert.assertTrue(tuples.hasNext());
 Tuple tuple = tuples.next();
 Assert.assertEquals(tuple.get(0), "banana");
 Assert.assertEquals(tuple.get(1), 2);
 Assert.assertTrue(tuples.hasNext());
 tuple = tuples.next();
 Assert.assertEquals(tuple.get(0), "pear");
 Assert.assertEquals(tuple.get(1), 1);
 }
  • 67. Test using mock and assert ▸ mockAlias allows to substitute input data ▸ assertOutput allows to compare String output data @Test
 public void testWordCountMock() throws IOException, ParseException {
 //Write input data
 BufferedWriter writer = new BufferedWriter(new FileWriter(new File("input.txt")));
 writer.write("banana pear banana");
 writer.close();
 
 PigTest t = newPigTest("pig/src/main/pig/wordcount_text.pig", new String[] {"input=input.txt", "output=null"});
 t.runScript();
 t.assertOutputAnyOrder("sorted", new String[]{"(banana,2)", "(pear,1)"});
 }
  • 68. Both of these tools have limitations ▸ Built around standard input and output (Text, CSVs etc.) ▸ Realistically most of our data is in other formats (e.g. Avro, Thrift, JSON) ▸ Does not test the STORE function (e.g. schema errors) ▸ getAlias() is especially difficult to use ▸ Need to remember field position: tuple.get(0) ▸ assertOutput() only allows String comparison ▸ Cumbersome to write complex structures (e.g. bags of bags)
  • 69. Example with Avro input/output ▸ Focus on testing script’s output ▸ Difficulty is to generate dummy Avro data and compare result text = LOAD '$input' USING AvroStorage();
 
 flattened = FOREACH text GENERATE flatten(TOKENIZE(body)) as word;
 grouped = GROUP flattened by word;
 result = FOREACH grouped GENERATE group AS word, (int)COUNT($1) AS cnt;
 sorted = ORDER result BY cnt DESC;
 
 STORE result INTO '$output' USING AvroStorage(); ▸ By default, PigUnit doesn’t execute the STORE, but it can be overridden pigTest.unoverride("STORE");
  • 70. Simple utility classes for Avro ▸ BasicAvroWriter ▸ Writes Avro file on disk based on a schema ▸ Supports GenericRecord and SpecificRecord ▸ BasicAvroReader ▸ Reads Avro file, the schema heads the file ▸ Also supports GenericRecord and SpecificRecord
  • 71. Test with Avro GenericRecord ▸ Create Schema with SchemaBuilder, write data, run script, read result and compare @Test
 public void testWordCountGenericRecord() throws IOException, ParseException {
 Schema schema = SchemaBuilder.builder().record("record").fields().
 name("text").type().stringType().noDefault().endRecord();
 GenericRecord genericRecord = new GenericData.Record(schema);
 genericRecord.put("text", "banana apple banana");
 
 BasicAvroWriter writer = new BasicAvroWriter(new Path(new File("input.avro").getAbsolutePath()), schema, getFileSystem());
 writer.append(genericRecord);
 
 PigTest t = newPigTest("pig/src/main/pig/wordcount_avro.pig", new String[] {"input=input.avro", "output=sorted.avro"});
 t.unoverride("STORE");
 t.runScript();
 
 //Check output
 BasicAvroReader reader = new BasicAvroReader(new Path(new File("sorted.avro").getAbsolutePath()), getFileSystem());
 Map<Utf8, GenericRecord> result = reader.readAndMapAll("word");
 Assert.assertEquals(result.size(), 2);
 Assert.assertEquals(result.get(new Utf8("banana")).get("cnt"), 2);
 Assert.assertEquals(result.get(new Utf8("apple")).get("cnt"), 1);
 }
  • 72. Test with Avro SpecificRecord ▸ Use InputRecord and OutputRecord generated Java classes, write data, run script, read result and compare @Test
 public void testWordCountSpecificRecord() throws IOException, ParseException {
 InputRecord input = InputRecord.newBuilder().setText("banana apple banana").build();
 
 BasicAvroWriter<InputRecord> writer = new BasicAvroWriter<InputRecord>(new Path(new File("input.avro").getAbsolutePath()), input.getSchema(), getFileSystem());
 writer.writeAll(input);
 
 PigTest t = newPigTest("pig/src/main/pig/wordcount_avro.pig", new String[] {"input=input.avro", "output=sorted.avro"});
 t.unoverride("STORE");
 t.runScript();
 
 //Check output
 BasicAvroReader<OutputRecord> reader = new BasicAvroReader<OutputRecord>(new Path(new File("sorted.avro").getAbsolutePath()), getFileSystem());
 List<OutputRecord> result = reader.readAll();
 Assert.assertEquals(result.size(), 2);
 Assert.assertEquals(result.get(0), OutputRecord.newBuilder().setWord("banana").setCount(2).build());
 Assert.assertEquals(result.get(1), OutputRecord.newBuilder().setWord("apple").setCount(1).build());
 }
  • 73. Common pitfalls ▸ PigUnit ▸ Mocking capabilities are very limited ▸ Overhead of 1-5 seconds per script ▸ Cryptic error messages sometimes (NullPointerException) ▸ Pig UDFs ▸ Can be tested independently
  • 75. Spark Testing Base Base classes to use when writing tests with Spark ▸ https://github.com/holdenk/spark-testing-base ▸ Functionalities ▸ Provides SparkContext ▸ Utilities to compare RDDs and DataFrames ▸ Simulate how Streaming works ▸ Includes cool RDD and DataFrames generator
  • 76. Thank You! We are hiring! http://careers.getyourguide.com/
  • 77. Extra Resources ▸ https://github.com/miguno/avro-hadoop-starter ▸ http://www.michael-noll.com/blog/2013/07/04/using-avro-in-mapreduce-jobs-with-hadoop-pig-hive/ ▸ http://blog.cloudera.com/blog/2015/09/making-apache-spark-testing-easy-with-spark-testing-base/ ▸ http://www.slideshare.net/hkarau/effective-testing-for-spark-programs-strata-ny-2015 ▸ http://avro.apache.org/docs/current/ ▸ http://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you-really-need-one ▸ http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/ ▸ https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about- real-time-datas-unifying
  • 78. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/patterns- data-workflows