Using Apache Spark to Tune Spark with Shivnath Babu and Adrian Popescu
Spark’s Catalyst Optimizer uses cost-based optimization (CBO) to pick the best execution plan for a SparkSQL query. The CBO can choose which join strategy to use (e.g., a broadcast join versus repartitioned join), which table to use as the build side for the hash-join, which join order to use in a multi-way join query, which filter to push down, and others. To get its decisions right, the CBO makes a number of assumptions including availability of up-to-date statistics about the data, accurate estimation of result sizes, and availability of accurate models to estimate query costs. These assumptions may not hold in real-life settings such as multi-tenant clusters and agile cloud environments; unfortunately, causing the CBO to pick suboptimal execution plans. Dissatisfied users then have to step in and tune the queries manually. In this talk, we will describe how we built Elfino, a Self-Driving Query Optimizer. Elfino tracks each and every query over time—before, during, and after execution—and uses machine learning algorithms to learn from mistakes made by the CBO in estimating properties of the input datasets, intermediate query result sizes, speed of the underlying hardware, and query costs. Elfino can further use an AI algorithm (modeled on multi-armed bandits with expert advice) to “explore and experiment” in a safe and nonintrusive manner using otherwise idle cluster resources. This algorithm enables Elfino to learn about execution plans that the CBO will not consider otherwise. Our talk will cover how these algorithms help guide the CBO towards better plans in real-life settings while reducing its reliance on assumptions and manual steps like query tuning, setting configuration parameters by trial-and-error, and detecting when statistics are stale. We will also share our experiences with evaluating Elfino in multiple environments and highlight interesting avenues for future work.
Recommended
More Related Content
Similar to Using Apache Spark to Tune Spark with Shivnath Babu and Adrian Popescu
Similar to Using Apache Spark to Tune Spark with Shivnath Babu and Adrian Popescu (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Using Apache Spark to Tune Spark with Shivnath Babu and Adrian Popescu
- 2. Meet$the$speakers • Data%engineer%at%Unravel • PhD%from%EPFL,%Switzerland • 8+%years%of%experience%in%performance% monitoring%&%modeling%of%data%management% systems • Focusing%on%tuning%and%optimization%of%Big% Data%apps 2#Exp8SAIS Adrian$Popescu • Cofounder%and%CTO%at%Unravel,%Adjunct% Professor%at%Duke%University • Focusing%on%easeKofKuse%and%manageability% of%dataKintensive%systems • Recipient%of%US%National%Science% Foundation%CAREER%Award,%three%IBM% Faculty%Awards,%HP%Labs%Innovation% Research%Award Shivnath Babu
- 8. This%rogue%app%is%wasting%resources% and%reducing%cluster%throughput 8 OPERATIONS%TEAMS #Exp8SAIS
- 10. To#add#to#Spark’s#complexity • Many#types#of#Spark#apps########################### • SQL • Streaming • AI/ML • Graph • Scala/Python/R • Many#app#submission#methods#in#Spark#### • CLI • Thrift=Server • Notebooks=like=Zeppelin,=Jupyter,=Hue • ETL=tools=like=Informatica,=Pentaho,=and=Talend • Schedulers=like=Airflow,=Autosys,=Control=M,=Oozie,=Tidal,=TWS • Many#infrastructure#choices#for#Spark######### • OnNpremises=multiNtenant=clusters • Transient=cloud=clusters • AutoNscaling=clusters • Containerized=deployments= Simple#SQL#and# Programming#Interface# 10#Exp8SAIS
- 14. 14#Exp8SAIS
- 16. Let$us$take$three$(hard)$tasks 16#Exp8SAIS • Failures$in$Spark • SLA%management%for%real0time%data%pipelines • Application%autotuning
- 22. Unravel’s Large,scale.Lab.Framework.for. Automatic.Root.Cause.Analysis Spark.and.multi,tenant.Workloads: , Variety(of(workloads:(Batch,(ML,(SQL,(Streaming,(etc. Failures: = Large(set(of(root(causes(learned(from(customers(&( partners.(Constantly(updated = Continuously(inject(these(root(causes(to(train(&(test( models(for(root=cause(prediction( Environment: = Lab(created(on(demand(on(cloud(or(on=premises = Workloads(are(run(and(failures(are(injected 22#Exp8SAIS
- 23. Injecting)Failures Application Execution Application Monitor FAILED Injected6 Failure Label Labeled6 Failures • Invalid6input • Invalid6memory6 configuration • OOME:6Java6heap6space • OOME:6GC6overhead6limit6 • Container6killed6by6YARN • Runtime6incompatibility Injected)failure)examples: • No6space6left6on6device • Transformations6inside6 other6transformations • Runtime6error • Arithmetic6error • Invalid6configuration6 settings Input6Feature6 Extraction 23#Exp8SAIS
- 24. Extracting*Input*Features*from*Logs java.lang.OutOfMemoryError: Java heap space at scala.reflect.ManifestFactory$$anon$9.newArray(Manifest.scala:114) at scala.reflect.ManifestFactory$$anon$9.newArray(Manifest.scala:112) at … • Extracting+stack+traces+and+error+messages • Tokenize+by+class+names+and+words Tokens*example: java.lang.OutOfmemoryError Java heap space at scala.reflect.ManifestFactory$$anon$9.newArray(Manife st.scala:114) • Create+a+vocabulary+of+words+from+all+words+collected 24#Exp8SAIS
- 25. Input&Feature&extraction • Bag&of&Words&with&TF8IDF – Computes+a+vocabulary+of+words – Uses+TF9IDF+to+reflect+importance+of+words+in+a+document • Doc2Vec – Maps+words,+paragraphs,+or+documents+to+multi9dimensional+vectors – Evaluates+the+placement+of+words+wrt neighboring+words – Uses+a+39layer+neural+network 25#Exp8SAIS
- 27. Predictive)Models • Shallow)Learning – Logistic*Regression – Random*forests • Deep)Learning – Neural*networks Very)easy)to) implement)these) in)Spark 27#Exp8SAIS
- 28. Predicting*the*Root*Cause*of*Failures • Training and%testing*with%injected%failures • Test%to%train%data%set%ratio%75%*to*25% • Models:%logistic%regression,%random%forests% 80 85 90 95 100 TF>IDF Doc2Vec Accuracy*Score* [%] Logistic%Regression Random%Forests Work%with% deep%learning% is%in%progress 28#Exp8SAIS See*our*talk*at* Strata,*NY*2017* for*more*details
- 29. Let$us$take$three$(hard)$tasks 29#Exp8SAIS • Failures*in*Spark • SLA$management$for$real>time$data$pipelines • Application*autotuning
- 30. Many%problems%can%cause%unreliable% performance%in%streaming%pipelines 0 STREAM%STORE Kafka HBase Spark REAL?TIME%PROCESSORIoT Sensors Database Other%Data Dashboard Result%Store Other%Output Untimely% results Poor% partitioning Inefficient% Configuration Resource% contention + + = 30#Exp8SAIS
- 33. 3 Detecting(anomalies(is(tricky Is(this(an(unexpected(lag( worth(alerting/acting(on? 33#Exp8SAIS
- 34. What%is%an%anomaly? 1. An%unexpected%change%that%needs%your%attention 2. Smart%alerts/actions:% 1. False%positives%should%be%minimal 2. False%negatives%should%be%minimal 4 34#Exp8SAIS
- 37. Let$us$take$three$(hard)$tasks 37#Exp8SAIS • Failures*in*Spark • SLA*management*for*real6time*data*pipelines • Application$autotuning
- 38. spark.driver.cores 2 spark.executor.cores … 10 spark.sql.shuffle.partitions 300 spark.sql.autoBroadcastJoinThres hold 20MB … SKEW('orders',@'o_custId') true spark.catalog.cacheTable(“orders") true … PERFORMANCE #AI7SAIS 38 Today,@tuning@is@often@by@trialNandNerror