W świecie mikrousługowym architektura Lambda zadomowiła się już na dobre. Tak przetwarzania streamingowe, jak i batchowe buduje wiele firm. Na rynku (o ile o rynku można mówić w kontekście open source) istnieje wiele frameworków, każdy jednak ma pewne cechy, które — zwłaszcza przy dużych projektach — utrudniają pracę. Jedne służą do przetwarzania real-time, drugie lepiej spisują się w workloadach batchowych. Niektóre z nich zaś można uznać za „rock-solid” tylko jeśli uruchamiamy je na Hadoopie. Nie brak tych problemów jest jednak główną zaletą Beama. A co nią jest? Dowiecie się na prezentacji! Poruszymy takie kwestie jak model przetwarzania, use-case’y, w których Beam się sprawdza, a także środowiska uruchomieniowe. Zobaczycie też, jak uruchamiać joby Apache Beam na Google Cloud Platform.

1. APACHE BEAM –
THE DATA
ENGINEER’S
HOPE
Robert Mroczkowski,
Piotr Wikieł

2. ABOUT US
▸ Data Platform Engineers at Allegro
▸ Maintaining probably one of the
largest Hadoop cluster in Poland
▸ We use public clouds for data
processing on a daily basis
▸ Both interested in ML
▸ Roots:
▸ Robert — sysop
▸ Piotr — dev
2
VegeTables

3. AGENDA
▸ ETL and Lambda Architecture
▸ Apache Beam framework foundations
▸ Transformations, windows, tags, etc.
▸ Batch and streaming
▸ Examples, use cases
3

4. LAMBDA ARCHITECTURE
BATCH SERVING
SPEED
DATA
QUERY
QUERY

5. LAMBDA ARCHITECTURE
BATCH SERVING
SPEED
QUERY
QUERY
DATA
Spark Druid
Spark/Flink
Kafka
Analyst
Microservice

6. LAMBDA ARCHITECTURE
▸ Complicated, huh?
▸ We have to build separate software for real-time and batch
computations
▸ … which have to be maintained, probably by different
teams
▸ Why not use one tool to rule them all?
6

7. APACHE
BEAM

8. APACHE
BEAM
UNIFIED MODEL FOR EXECUTING BOTH BATCH
AND STREAM DATA PROCESSING PIPELINES

9. APACHE BEAM
▸ Born in Google, and then open-sourced
▸ Designed especially for ETL pipelines
▸ Use for both streaming and batch processing
▸ Heavily parallel processing
▸ Exactly once semantics
9

10. IN CODE
▸ Backends (Spark, Flink, Apex, Dataflow, Gearpump, Direct)
▸ Java (rich), Python (pretty) SDK and recently added GO
SDK
▸ Experimental SQL on PCollections
▸ Open-source Scala API (github –> spotify/scio)
10APACHE BEAM

11. APACHE BEAM FRAMEWORK
▸ Pipeline
▸ Input/Output
▸ PCollection — distributed data representation (Spark RDD-
like)
▸ Transformation — operation applied on PCollection
11

12. PCOLLECTION
▸ Any type but all one type - serializable
▸ Immutable
▸ Any size - bounded, unbounded
▸ Timestamps
12APACHE BEAM FRAMEWORK FOUNDATIONS

13. TRANSFORMATIONS
▸ ParDo — like map in MapReduce
▸ Filter elements of PCollection
▸ Format values in PCollection
▸ Cast types
▸ Computations on each single element
▸ collection.apply(ParDo.of(SomeDoFn()))
13APACHE BEAM FRAMEWORK FOUNDATIONS

14. TRANSFORMATIONS
▸ GroupByKey
▸ group values of k/v pairs for the same key
▸ like Shuffle phase in Map Reduce
▸ For streaming - windowing or triggers are necessary
14APACHE BEAM FRAMEWORK FOUNDATIONS

15. TRANSFORMATIONS
▸ CoGroupByKey
▸ join values of k/v pairs for the same key for separate
PCollection
▸ .apply(CoGroupByKey.create())
▸ For streaming — windowing or triggers are necessary
15APACHE BEAM FRAMEWORK FOUNDATIONS

16. TRANSFORMATIONS
▸ Combine
▸ Reduce from Map Reduce paradigm
▸ Combines all elements in PCollection
▸ Combines elements for specific key in k/v pairs or entire
PCollection
▸ Comutative & Associative Function
▸ For streaming accumulates elements per window
16APACHE BEAM FRAMEWORK FOUNDATIONS

17. TRANSFORMATIONS
▸ Flatten
▸ Merge several PCollections
▸ Partition
▸ Split PCollection
▸ Partitioning function(element, numberOfPartitions)
17APACHE BEAM FRAMEWORK FOUNDATIONS

18. TRANSFORMATIONS
▸ Several already defined transformations:
▸ Filter.By
▸ Count
▸ Custom Transformations
▸ Serializable
▸ Thread-compatible
▸ Idempotent
18APACHE BEAM FRAMEWORK FOUNDATIONS

19. TRANSFORMATIONS – SPLITTABLE DOFN
▸ Split processing one element to many workers
▸ Possibly unbounded result of ParDo’ing one element
▸ Examples:
▸ tail -f logs-directory
▸ running jobs outside of Beam and process result within
it
▸ Currently supported in Dataflow and Flink runners
19APACHE BEAM FRAMEWORK FOUNDATIONS

20. TAGGED OUTPUT
20APACHE BEAM FRAMEWORK FOUNDATIONS

21. SIDE INPUT – ENRICHMENT
▸ Additional data in ParDo
▸ Computed at runtime
▸ words.apply(ParDo.of(...).withSideInputs(dataView);
21APACHE BEAM FRAMEWORK FOUNDATIONS

22. IO
22APACHE BEAM FRAMEWORK FOUNDATIONS
FILE MESSAGING DATABASE
HDFS Kinesis Cassandra
GCS Kafka Hbase
S3 PubSub Hive
Local JMS BigQuery
Avro MQTT BigTable
Text DataStore
TFRecord Spanner
XML Mongo
Tika Redis
ParquetIO Solr

23. TEKST
SEEN SO FAR
RUNNER
SDK
TRANSFORMS, IO
USER CODE
MODEL

24. APACHE BEAM
TWEETS HASHTAGS AUTOCOMPLETE
Predictions
Tweets
READS
WRITES
#juzwiosna; #jug w Warszawie; #java 10 GA released
24

25. APACHE BEAM
TWEETS HASHTAGS AUTOCOMPLETE
Predictions
Tweets
READS
WRITES
#juzwiosna; #jug w Warszawie; #java 10 GA released
EXTRACT jug, java, juzwiosna
25

26. APACHE BEAM
TWEETS HASHTAGS AUTOCOMPLETE
Predictions
Tweets
READS
WRITES
#juzwiosna; #jug w Warszawie; #java 10 GA released
EXTRACT jug, java, juzwiosna
COUNT jug -> 10k, java -> 4M, juzwiosna -> 100
26

27. APACHE BEAM
TWEETS HASHTAGS AUTOCOMPLETE
Predictions
Tweets
READS
WRITES
#juzwiosna; #jug w Warszawie; #java 10 GA released
EXTRACT jug, java, juzwiosna
COUNT
EXPAND
jug -> 10k, java -> 4M, juzwiosna -> 100
{j -> [jug -> 10k, java -> 4M, juzwiosna -> 100],
ju-> [jug -> 10k, juzwiosna -> 100]}
27

28. APACHE BEAM
TWEETS HASHTAGS AUTOCOMPLETE
Predictions
Tweets
READS
WRITES
#juzwiosna; #jug w Warszawie; #java 10 GA released
{j->[java, jug, juzwiosna], ju->[jug, juzwiosna]}
EXTRACT jug, java, juzwiosna
COUNT
EXPAND
TOP(3)
jug -> 10k, java -> 4M, juzwiosna -> 100
{j -> [jug -> 10k, java -> 4M, juzwiosna -> 100],
ju-> [jug -> 10k, juzwiosna -> 100]}
28

29. APACHE BEAM
TWEETS — BATCH
READS
WRITES
EXTRACT
COUNT
EXPAND
TOP(3)
Pipeline p = Pipeline.create(new PipelineOptions());
p.begin()
.apply(TextIO.Read.from("..."))
.apply(ParDo.of(new ExtractTags()))
.apply(Count.perElement())
.apply(ParDo.of(new ExpandPrefixes()))
.apply(Top.largestPerKey(3))
.apply(TextIO.Write.to("...");
p.run();
29

30. APACHE BEAM FRAMEWORK – STREAMING
▸ Windows
▸ One global window by default
▸ Applied for group, combine or output transformations
▸ GroupByKey — data is grouped by both key and window
30

31. WINDOWS
FIXED TIME WINDOWS
31

32. WINDOWS
SLIDING WINDOWS
32

33. WINDOWS
SESSION WINDOWS
33

34. APACHE BEAM
TWEETS – BATCH
READS
WRITES
EXTRACT
COUNT
EXPAND
TOP(3)
Pipeline p = Pipeline.create(new PipelineOptions());
p.begin()
.apply(PubsubIO.Read.topic("..."))
.apply(ParDo.of(new ExtractTags()))
.apply(Count.perElement())
.apply(ParDo.of(new ExpandPrefixes()))
.apply(Top.largestPerKey(3))
.apply(PubsubIO.Write.topic(„...");
p.run();
34

35. APACHE BEAM
TWEETS – STREAMING
READS
WRITES
EXTRACT
COUNT
EXPAND
TOP(3)
Pipeline p = Pipeline.create(new PipelineOptions());
p.begin()
.apply(PubsubIO.Read.topic("..."))
.apply(Window.into(SlidingWindows.of(
Duration.standardMinutes(60))))
.apply(ParDo.of(new ExtractTags()))
.apply(Count.perElement())
.apply(ParDo.of(new ExpandPrefixes()))
.apply(Top.largestPerKey(3))
.apply(CassandraIO.<Hashtag>write());
p.run();
35

36. APACHE BEAM FRAMEWORK – STREAMING
▸ Watermark is approximate lag between event timestamp
and processing time
▸ Beam keeps track of watermark and use it to fire aggregates
▸ when window passes watermark data is considered late and
is discarded
▸ but... you can allow for lateness
▸ FixedWindows.of(..)
.withAllowedLateness(Duration.standardDays(2))
36

37. APACHE BEAM FRAMEWORK – STREAMING
▸ Triggers
▸ Change default windowing behaviour
▸ Completness / Latency / Cost
▸ Event Time / Processing Time / Data
37

38. APACHE BEAM FRAMEWORK – STATEFUL PROCESSING
38
(k1,w1) (k2,w2) (k3,w3)
"s1" 12 33 -5
"s2" "kot" "pies" "okoń"
"s3" 0,03 0,12 0,33
"s4" "ala" "ma" "kota"

39. CAPABILITY MATRIX
39APACHE BEAM — BACKENDS

40. CAPABILITY MATRIX
40APACHE BEAM — BACKENDS

41. APACHE BEAM – RUN
mvn compile exec:java —Dexec.mainClass=org.apache.beam.examples.WordCount
-Dexec.args="--inputFile=pom.xml --output=counts" -Pdirect-runner
41

42. APACHE BEAM – RUN
mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount
-Dexec.args="--runner=SparkRunner --inputFile=pom.xml —output=counts"
-Pspark-runner
42

43. APACHE BEAM – RUN
mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount
-Dexec.args="--runner=DataflowRunner --gcpTempLocation=gs://bb/tmp
--inputFile=gs://apache-beam-samples/shakespeare/*
-—output=gs://bb/counts”
-Pdataflow-runner
43

44. APACHE BEAM – USE CASE’Y
▸ ETL
▸ Fraud detection
▸ Ads pricing (similar: Uber pricing)
▸ Sentiment analysis
44

45. LINKS
▸ Google Dataflow paper: https://research.google.com/pubs/
pub43864.html
▸ Apache Beam: https://beam.apache.org/
▸ Design documents: https://wtanaka.com/beam/design-doc
45

46. Q&A
THANK YOU

47. Q&A
THANK YOU
We are hiring ;-)
https://goo.gl/zzqXLS