Scylla Summit 2018: Building Recoverable (and optionally Async) Spa...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Building Recoverable Pipelines
With Apache Spark
Holden Karau
Open Source Developer Advocate @ Google

Some links (slides & recordings
will be at):
http://bit.ly/2QMUaRc
^ Slides & Code
(only after the talk because early is h...

Holden:
 Prefered pronouns are she/her
 Developer Advocate at Google
 Apache Spark PMC/Committer, contribute to many ot...

Who y’all are?
 Nice folk
 Like databases of a certain kind
 Occasionally have big data jobs on your big data fail
mxms...

What are we going to explore?
 Brief: what is Spark and why it’s related to this conference
 Also brief: Some of the way...

What is Spark?
• General purpose distributed system
• With a really nice API including Python :)
• Apache project (one of ...

The different pieces of Spark
Apache Spark
SQL, DataFrames & Datasets
Structured
Streaming
Scala,
Java,
Python, &
R
Spark ...

Why people come to Spark:
Well this MapReduce job
is going to take 16 hours -
how long could it take to
learn Spark?
dougw...

Why people come to Spark:
My DataFrame won’t fit in
memory on my cluster
anymore, let alone my
MacBook Pro :( Maybe
this S...

Big Data == Wordcount
lines = sc.textFile(src)
words = lines.flatMap(lambda x: x.split(" "))
word_count =
(words.map(lambd...

Big Data != Wordcount
▪ ETL (keeping your databases in sync)
▪ SQL on top of non-SQL (hey what about if we added a SQL
eng...

f ford Pinto by Morven

Why Spark fails & fails late
▪ Lazy evaluation can make predicting behaviour difficulty
▪ Out of memory errors (from JVM h...

f ford Pinto by Morven
ayphen

Why isn’t it recoverable?
▪ Seperate jobs - no files, no VMs, only sadness
▪ If same job (e.g. notebook failure and retry)...

“Recoverable” Wordcount: Take 1
lines = sc.textFile(src)
words_raw = lines.flatMap(lambda x: x.split(" "))
words_path = "w...

So what can we do better?
▪ Well if the pipeline fails in certain ways this will fail
▪ We don’t have any clean up on succ...

“Recoverable” Wordcount: Take 2
lines = sc.textFile(src)
words_raw = lines.flatMap(lambda x: x.split(" "))
words_path = "w...

So what can we do better?
▪ Well if the pipeline fails in certain ways this will fail
• Fixed
▪ We don’t have any clean up...

$“Recoverable” [X]: Take 3 def non_blocking_df_save_or_load(df, target): success_files = ["{0}/SUCCESS.txt", "{0}/_SUCCESS"...$

So what can we do better?
▪ Try and not slow down our code on the happy path
• async?
▪ Cleanup on success (damn meant to ...

Adding async?
def non_blocking_df_save(df, target):
import threading
def save_panda():
df.write.mode("overwrite").save(tar...

What could go wrong?
▪ Turns out… a lot
▪ Multiple executions on the DAG are not super great
(getting better but)
▪ How do...

Spark’s (core) magic: the DAG
▪ In Spark most of our work is done by transformations
• Things like map
▪ Transformations r...

The DAG The query
plan Susanne Nilsson

$cache + sync count + async save def non_blocking_df_save_or_load(df, target): s = "{0}/SUCCESS.txt" if fs.exists(hadoop_fs...$

Well that was “fun”?
▪ Replace wordcount with your back-fill operation and it
becomes less fun
▪ You also need to clean up...

Spark Videos
▪ Apache Spark Youtube Channel
▪ My Spark videos on YouTube -
• http://bit.ly/holdenSparkVideos
▪ Spark Summi...

Learning Spark
Fast Data Processing
with Spark
(Out of Date)
Fast Data Processing with
Spark (2nd edition)
Advanced
Analyt...

I also have a book...
High Performance Spark, it’s available today & the gift of
the season.
Unrelated to this talk, but i...

Cat wave photo by Quinn Dombrowski
k thnx bye!
If you <3 Spark testing & want to fill out
survey: http://bit.ly/holdenTest...