Alexander Kolb - Flinkspector – Taming the squirrel

1. Flinkspector Taming the Squirrel

2. Alexander Kolb otto group BI @lofifnc Alexander Kolb  Otto Group BI @loﬁfnc

3. Alexander Kolb - 2016 - otto group Agenda • Introduction • Testing Apache Flink Applications • Flinkspector • Concept • Input • Expectations • Execution • Conclusion • Outlook

4. Alexander Kolb - 2016 - otto group Introduction

5. Alexander Kolb - 2016 - otto group • Multichannel Retail • Financial Services • Services • 30 countries • 123 companies

6. Alexander Kolb - 2016 - otto group Problem • Transparency • Traceability • Reproducibility

7. Alexander Kolb - 2016 - otto group Solutions • Visualisation • Datalineage graph / Message tracing • Unit Testing

8. Alexander Kolb - 2016 - otto group Testing Apache Flink Applications

9. Alexander Kolb - 2016 - otto group Apache Flink Application final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();    DataStream<String> text = env.socketTextStream("localhost", 9999);      DataStream<Tuple2<String, Integer>> words = text.flatMap(new WordCount.LineSplitter());      DataStream<Tuple2<String, Integer>> aWords = words.filter(new StartsWithAFilter());      DataStream<Tuple2<String, Integer>> counts = aWords.keyBy(0).timeWindow(Time.of(2, TimeUnit.MINUTES)).sum(1);    counts.print();    env.execute("Wordcount Example");

10. Alexander Kolb - 2016 - otto group User Defined Functions public final class StartsWithAFilter  implements FilterFunction<Tuple2<String,Integer>> {   @Override  public boolean filter(Tuple2<String, Integer> t) throws Exception {  return t.f0.startsWith("a");  }  }

11. Alexander Kolb - 2016 - otto group User Defined Functions   @Test  public void testAFilter() throws Exception {  StartsWithAFilter filter = new StartsWithAFilter();    Assert.assertEquals(true, filter.filter(Tuple2.of("aTest",1)));  Assert.assertEquals(false, filter.filter(Tuple2.of("bTest",1)));  }

12. Alexander Kolb - 2016 - otto group Stream Transformations public static DataStream<Tuple2<String,Integer>>  countAWords(DataStream<Tuple2<String,Integer>> aWords) {  return aWords  .keyBy(0)  .timeWindow(Time.of(2, TimeUnit.MINUTES))  .sum(1);  }

13. Alexander Kolb - 2016 - otto group Stream Transformations @Test  public void aWordCountTest() {  StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();  env.setParallelism(1);    DataStream<Tuple2<String, Integer>> words = env.fromCollection(Arrays.asList(  Tuple2.of("a1", 1),  Tuple2.of("a2", 1),  Tuple2.of("a1", 1)  ));    DataStream<Tuple2<String, Integer>> results = SocketTextStreamWordCount.countAWords(words);    Iterator<Tuple2<String, Integer>> output = DataStreamUtils.collect(results);      Assert.assertEquals(output.next(), Tuple2.of("a1", 1));  Assert.assertEquals(output.next(), Tuple2.of("a2", 2));  Assert.assertEquals(output.hasNext(), false);  }

14. Alexander Kolb - 2016 - otto group Flinkspector

15. Alexander Kolb - 2016 - otto group Concept 1. Specify input. 2. Aquire data stream from the input speciﬁcation. 3. Deﬁne expectations for the resulting data stream. 4. Apply the expectations to the produced data stream.

16. Alexander Kolb - 2016 - otto group TestStreamEnvironment Runtime Subscriber Test Source Input Filter KeyedWindowReducer Concept Veriﬁer Test Sink Publisher Trigger (default) paired

17. Alexander Kolb - 2016 - otto group Expectations

18. Alexander Kolb - 2016 - otto group Input EventTimeInput<Tuple2<String,Integer>> input =  EventTimeInputBuilder.startWith(Tuple2.of("a1", 1))  .emit(Tuple2.of("a2", 1), after(1, minutes))  .emit(Tuple2.of("a3", 1), before(1, seconds), times(2)) .emit(Tuple3.of(“a4”, 1), intoWindow(4, seconds))  .repeatAll(after(1, minutes), times(3));

19. Alexander Kolb - 2016 - otto group List Based ExpectedRecords.create(asList(1,2,3,4));     ExpectedRecords.create(asList(1,2,3,4))  .refine().only();     ExpectedRecords.create(asList(1,2,3,4))  .refine().sameFrequency();     (4,1,2,3,3,5) (4,1,2,3,3,5) (4,1,2,3,3,5)

20. Alexander Kolb - 2016 - otto group List Based (4,1,2,3,3,5) (4,1,2,3,3,5) (4,1,2,3,3,5) ExpectedRecords.create(asList(1,2,3,4));     ExpectedRecords.create(asList(1,2,3,4))  .refine().only();     ExpectedRecords.create(asList(1,2,3,4))  .refine().sameFrequency();    

21. Alexander Kolb - 2016 - otto group List Based (4,1,2,3,3,5) (4,1,2,3,3,5) (4,1,2,3,3,5) ExpectedRecords.create(asList(1,2,3,4));     ExpectedRecords.create(asList(1,2,3,4))  .refine().only();     ExpectedRecords.create(asList(1,2,3,4))  .refine().sameFrequency();    

22. Alexander Kolb - 2016 - otto group List Based (4,1,2,3,3,5) (4,1,2,3,3) (4,1,2,3,3,5) ExpectedRecords.create(asList(1,2,3,4));     ExpectedRecords.create(asList(1,2,3,4))  .refine().only();     ExpectedRecords.create(asList(1,2,3,4))  .refine().sameFrequency();    

23. Alexander Kolb - 2016 - otto group List Based (4,1,2,3,3,5) (4,1,2,3,3) (4,1,2,3,3,5) ExpectedRecords.create(asList(1,2,3,4));     ExpectedRecords.create(asList(1,2,3,4))  .refine().only();     ExpectedRecords.create(asList(1,2,3,4))  .refine().sameFrequency();    

24. Alexander Kolb - 2016 - otto group List Based (4,1,2,3,3,5) (4,1,2,3,3) (4,1,2,3,5) ExpectedRecords.create(asList(1,2,3,4));     ExpectedRecords.create(asList(1,2,3,4))  .refine().only();     ExpectedRecords.create(asList(1,2,3,4))  .refine().sameFrequency();    

25. Alexander Kolb - 2016 - otto group List Based ExpectedRecords.create(asList(1,2,3,4))  .refine().inOrder(notStrict).all();     ExpectedRecords.create(asList(1,2,3,4))  .refine().inOrder(strict).from(1); List(4,1,2,3,3,5) List(5,2,3,4,1)

26. Alexander Kolb - 2016 - otto group List Based List(4,1,2,3,3,5) List(5,2,3,4,1) ExpectedRecords.create(asList(1,2,3,4))  .refine().inOrder(notStrict).all();     ExpectedRecords.create(asList(1,2,3,4))  .refine().inOrder(strict).from(1);

29. Alexander Kolb - 2016 - otto group List Based List(1,2,3,4,1) MatcherBuilder only sameFrequency order     ExpectedRecords.create(asList(1,2,3,4))  .refine() .only() .sameFrequency() .inOrder(strict).from(1);

30. Alexander Kolb - 2016 - otto group List Based     ExpectedRecords.create(asList(1,2,3,4))  .refine() .only() .sameFrequency() .inOrder(strict).from(1); List(1,2,3,4,1) MatcherBuilder only sameFrequency order

33. Alexander Kolb - 2016 - otto group List Based List(2,3,4,1) MatcherBuilder only sameFrequency order     ExpectedRecords.create(asList(1,2,3,4))  .refine() .only() .sameFrequency() .inOrder(strict).from(1);

34. Alexander Kolb - 2016 - otto group List Based List(2,3,4,1) MatcherBuilder only sameFrequency order     ExpectedRecords.create(asList(1,2,3,4))  .refine() .only() .sameFrequency() .inOrder(strict).from(1);

35. Alexander Kolb - 2016 - otto group Assertion Based new MatchTuples<Tuple2<String,Integer>>("word","count")  .assertThat("word", startsWith("a"))  .assertThat("count", greaterThan(3))  .onEachRecord();

36. Alexander Kolb - 2016 - otto group Assertion Based new MatchTuples<Tuple2<String,Integer>>("word","count")  .assertThat("word", startsWith("a"))  .assertThat("count", greaterThan(3))  .oneOfThem()  .onEachRecord();

37. Alexander Kolb - 2016 - otto group Assertion Based .assertThat(A,…) .assertThat(B,…) .assertThat(C,…) .eachOfThem() .onAnyRecord() A B C

38. Alexander Kolb - 2016 - otto group Assertion Based A B C .assertThat(A,…) .assertThat(B,…) .assertThat(C,…) .eachOfThem() .onAnyRecord()

39. Alexander Kolb - 2016 - otto group Assertion Based .assertThat(A,…) .assertThat(B,…) .eachOfThem() .onAnyRecord() A B C

40. Alexander Kolb - 2016 - otto group Assertion Based .assertThat(A,…) .assertThat(B,…) .assertThat(C,…) .eachOfThem() .onAnyRecord() A B C

41. Alexander Kolb - 2016 - otto group Assertion Based .assertThat(A,…) .assertThat(A,…) .assertThat(A,…) .atLeastNOfThem(2) .onEach() A B C

42. Alexander Kolb - 2016 - otto group Assertion Based .assertThat(A,…) .assertThat(B,…) .assertThat(C,…) .atExactlyNOfThem(2) .onEach() A B C

43. Alexander Kolb - 2016 - otto group Assertion Based public class Each<T> extends UntilCombineMatcher<T> {    public Each(Iterable<Matcher<? super T>> matchers) {  super(matchers);  }    @Override  public String prefix() {  return "each of";  }    @Override  public boolean validWhen(int matches, int possibleMatches) {  return matches == possibleMatches;  }    @Factory  public static <T> Each<T> each(Iterable<Matcher<? super T>> matchers) {  return new Each<T>(matchers);  }  }

44. Alexander Kolb - 2016 - otto group TestStreamEnvironment Runtime Subscriber Test Source Input Filter KeyedWindowReducer Execution Veriﬁer Test Sink Publisher Trigger (default) paired

45. Alexander Kolb - 2016 - otto group Execution Test SourceInput Filter KeyedWindowReducer Test Sink Publisher Test Source Input Filter KeyedWindowReducer Test Sink Publisher 1 1 1 1 2 2 2 2 LocalCluster TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger (default)

46. Alexander Kolb - 2016 - otto group Execution LocalCluster TS I1 F KWR TSi P1 1 1 TS I2 2 F 2 KWR 1 1 2 TSi P2 2 TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger (default) Timeout

47. Alexander Kolb - 2016 - otto group Execution TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger (default) LocalCluster TS I1 F KWR TSi P1 1 1 TS I2 2 F 2 KWR 1 1 2 TSi P2 2 Timeout

48. Alexander Kolb - 2016 - otto group Execution TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger (default) LocalCluster TS I1 F KWR TSi P1 1 1 TS I2 2 F 2 KWR 1 1 2 TSi P2 2 Timeout

49. Alexander Kolb - 2016 - otto group LocalCluster TSI1 F KWR TSiP1 1 1 TSI2 2 F 2 KWR 1 1 2 TSiP2 2 Execution TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger (default) Timeout CLOSE 1 CLOSE 2

50. Alexander Kolb - 2016 - otto group TSI1 F KWR TSiP1 1 1 TSI2 2 F 2 KWR 1 1 2 TSiP2 2 Execution TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger (default) LocalCluster

51. Alexander Kolb - 2016 - otto group Execution TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger LocalCluster TS I1 F KWR TSi P1 1 1 TS I2 2 F 2 KWR 1 1 2 TSi P2 2 TimeoutTimeout

52. Alexander Kolb - 2016 - otto group Execution TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger LocalCluster TS I1 F KWR TSi P1 1 1 TS I2 2 F 2 KWR 1 1 2 TSi P2 2

53. Alexander Kolb - 2016 - otto group Execution TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger (default) LocalCluster TS I1 F KWR TSi P1 1 1 TS I2 2 F 2 KWR 1 1 2 TSi P2 2 TimeoutTimeout

54. Alexander Kolb - 2016 - otto group Execution TestStreamEnvironment Runtime Subscriber Veriﬁer Trigger (default) LocalCluster TS I1 F KWR TSi P1 1 1 TS I2 2 F 2 KWR 1 1 2 TSi P2 2 TimeoutTimeout

55. Alexander Kolb - 2016 - otto group Conclusion

56. Alexander Kolb - 2016 - otto group Conclusion • Small and concise test cases. • Extensible dedicated runtime. • Integration into existing test platforms.  • ProcessingTimeWindows. • Input speciﬁcation too expansive. 

57. Alexander Kolb - 2016 - otto group Outlook

58. Alexander Kolb - 2016 - otto group Outlook • Scala (Test) support. • Symbolic testing integration. • System tests on the actual cluster.

59. Alexander Kolb - 2016 - otto group https://github.com/ottogroup/flink-spector

60. Alexander Kolb - 2016 - otto group ottogroup.com WE ARE HIRING!

Alexander Kolb - Flinkspector – Taming the squirrel

Flink Forward

Alexander Kolb - Flinkspector – Taming the squirrel