In this demo based talk with live coding, we’ll present a functional typeful framework for developing Apache Spark applications. We’ll walk through the following key topics: – turning unmanageable Spark scripts into typeful Spark Functions – serverless deployment of Spark functions into the cloud – unit testing Spark functions to save cluster resources and developers time – seamless Spark session management between concurrent Spark jobs in exclusive or share modes

1. MISTMIST
Serverless proxy to Apache Spark

2. VADIM CHELYSHOVVADIM CHELYSHOV
github.com/dos65
hydrosphere.io
Scalalaz podcast (RUS)
Scala-digest (RUS)

3. HYDROSPHERE MISTHYDROSPHERE MIST
1.0.0-RC16
https://github.com/Hydrospheredata/mist

4. WHAT PROBLEM DOES IT SOLVE?WHAT PROBLEM DOES IT SOLVE?

5. Getting Big Data Projects to Production Is aGetting Big Data Projects to Production Is a
ChallengeChallenge
Only 15 percent of businesses reported deploying their
big data project to production, effectively unchanged
from last year (14 percent).
https://www.gartner.com/newsroom/id/3466117

6. Data engineers vs. data scientistsData engineers vs. data scientists
A common starting point is 2-3 data engineers for every
data scientist. For some organizations with more
complex data engineering requirements, this can be 4-5
data engineers per data scientist
https://www.oreilly.com/ideas/data-engineers-vs-
data-scientists

7. The lack of tooling?The lack of tooling?

9. The lack of right tooling?The lack of right tooling?

10. Getting Spark Projects to ProductionGetting Spark Projects to Production
How to
provide a way to run our job?
return a result?
report an error?
run multiply jobs in parallel?

11. LET'S COMPARE WITH USUALLET'S COMPARE WITH USUAL
THINGSTHINGS

12. DatabasesDatabases
Postgresql - 1995
CREATE OR REPLACE FUNCTION foo(param VARCHAR)
RETURNS TABLE
...

13. Web applicationsWeb applications
Java servlet - 1997
public class NewServlet extends HttpServlet {
@Override
protected void doGet(
HttpServletRequest request,
HttpServletResponse response) throws ServletException, IOException {
...
}
}

14. Big data processingBig data processing
Hadoop - 2005
Apache Spark - 2011
object MyApp {
def main(arg: String[]): Unit = {
val conf = new SparkConf().setAppName(...)
val sc = new SparkContext(conf)
...
}
}
./bin/spark-submit ...

15. RunRun
Databases SELECT * FROM foo(param)
Web applications curl
Spark Shell + Trigger

16. Return resultReturn result
Databases built-in protocol
Web applications built-in protocol
Spark Read from fs/storages

17. Result errorResult error
Databases explicit error information
Web
applications
HTTP CODE + explicit error
infomation
Spark exit code

18. ParallelismParallelism
Databases by default
Web applications by default
Spark Good luck!

19. What a hell is going on!
These thing aren't
about big data!

20. It isn't normal development experinceIt isn't normal development experince
A lot of actions over ssh-session to test, deploy and
run your code
Improssible to run job without direct shell
command
Process more that one job in parallel is difficult
Poor interfaces

21. Just a special service!Just a special service!
Artifacts/settings CRUD
API for launching jobs and receiving their status
Spark driver launcher
A library do describe operations using Spark

22. MIST BASICSMIST BASICS

23. What is Mist?What is Mist?
Serverless proxy to Apache Spark

24. What is Mist?What is Mist?
Serverless proxy ?
Apache Spark ✔

25. What is Mist?What is Mist?
It's a combinations of
Programming framework
HTTP/Async interface to run deploy and run spark
programms
Spark contexts / driver manager

26. ConceptsConcepts
Function
user code with Spark program to be deployed on Mist
Artifact
file (.jar or .py) that contains a Function
Context
settings for Mist Worker where Function is being
executed
Job
a result of the Function execution

27. Mist instancesMist instances
Mist Worker
Spark driver application which invokes functions
Mist Master
exposes http/async api
stores functions/artifacts/contexts
run/manage workers
run functions on workers

28. Mist Master - Http ApiMist Master - Http Api
/v2/api/functions
/v2/api/artifacts
/v2/api/contexts
/v2/api/jobs

29. WORD COUNTWORD COUNT

30. Vanilla SparkVanilla Spark
object WordCount {
def main(arg: Array[String]): Unit = {
val conf = new SparkConf().setAppName(...)
val sc = new SparkContext(conf)
val textFile = sc.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")
}
}

31. MistFnMistFn
object WordCount extends MistFn {
override def handle: Handle = {
arg[String]("path").onSparkContext((path: String, sc: SparkContext) => {
val counts = sc.textFile(path)
.flatMap(_.split(" "))
.map(w => w -> 1)
.reduceByKey(_ + _ )
// counts.saveAsTextFile("hdfs://...")
counts.collect().toMap
}).asHandle
}
}

32. Vanilla SparkVanilla Spark
./bin/spark-submit
--class <main-class>
--master <master-url>
--deploy-mode <deploy-mode>
--conf <key>=<value>
... # other options
<application-jar>
[application-arguments]

33. Mist-cli configMist-cli config
model = Context
name = mycontext
data {
spark-conf {
spark.master = <master url>
}
}
model = Artifact
name = wordcount
data.file-path = "./target/scala-2.11/<application-jar>"
model = Function
name = word-count
data {
path = <artifact_jar>
class-name = "HelloMist$"
context = mycontext
}

34. Mist-cli apply & runMist-cli apply & run
$ mist-cli -f apply conf
$ curl -X POST -d <json-input>
'http://localhost:2004/v2/api/functions/word-count-example/jobs'
{"id":"00e5294d-77f4-463b-85b6-4b73185eab1a"}

35. A lot of actions over ssh-session to test, deploy and
run your code
Improssible to run job without direct shell
command
Process more that one job in parallel is difficult
Poor interfaces

36. SERVERLESS?SERVERLESS?

37. Parallelism/MultitenancyParallelism/Multitenancy
Usual 1 req ~= thread
Spark 1 req ~= process

38. ServerlessServerless

39. ContextContext
Mist Context describes parameters of Mist worker and
Spark context
model = Context
name = cluster_ctx
data {
precreated = false
spark-conf {
spark.master = yarn
spark.submit.deployMode = cluster
spark.executor.instances = 2
spark.executor.cores = 2
spark.executor.memory = 1G
}
}

40. Mist instancesMist instances
Mist Worker
receives req -> invokes functions -> returns result
back to master
Mist Master
queue requests
sends them on workers

41. ApplyApply

42. RunRun
$ curl -X POST -d <json-input>
'http://localhost:2004/v2/api/functions/word-count-example/jobs'
{"id":"00e5294d-77f4-463b-85b6-4b73185eab1a"}

43. RunRun

44. RunRun

45. ScalingScaling
model = Context
name = cluster_ctx
data {
precreated = false
max-parallel-jobs = 2
...
}

47. Worker modesWorker modes
Exclusive
starts new worker for every request
Shared
reuses worker instance
model = Context
name = cluster_ctx
data {
precreated = false
max-parallel-jobs = 2
worker-mode = "shared"
// or
worker-mode = "exlsusive"
...
}

48. Multi clusterMulti cluster
Contexts may be configured to work on different
clusters
model = Context
name = cluster_ctx
data {
precreated = false
spark-conf {
spark.master = yarn
}
}
model = Context
name = cluster_ctx
data {
precreated = false
spark-conf {
spark.master = spark://

49. Multi clusterMulti cluster

50. A lot of actions over ssh-session to test, deploy and
run your code
Improssible to run job without direct shell
command
Process more that one job in parallel is difficult
Poor interfaces

51. SPARK FUNCTIONSPARK FUNCTION

52. Pi EstimationPi Estimation
def main(arg: Array[String]): Unit = {
val count = sc.parallelize(1 to samples).filter { _ =>
val x = math.random
val y = math.random
x*x + y*y < 1
}.count()
println(s"Pi is roughly ${4.0 * count / samples}")
}

53. Pi EstimationPi Estimation
What signature is better?
def estimatePi1(samples: Int): Unit
// vs
def estimatePi2(samples: Int): Double

54. Word count againWord count again
def main(arg: Array[String]): Unit = {
val textFile = sc.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")
}

55. Word count againWord count again
What signature is better?
def wordCount(from: String, to: String): Unit
// vs
def wordCount(from: String, to: String): Map[String, Int]
// or
sealed trait OutputFile
case class HdfsFile(path: String) extends OutputFile
case class S3File(path: String) extends OutputFile
def wordCount(from: String, to: String): OutputFile

56. HOW TO REPRESENT A SPARKHOW TO REPRESENT A SPARK
JOB?JOB?

57. Spark ApplicationSpark Application
def main(args: Array[String]): Unit
Array[String] => Unit

58. MistFnMistFn
Context is already described
Json input instead of arguments
Json output instead of Unit
Array[String] => Unit
// vs
(Json, SparkContext) => Json

59. SparkContext typesSparkContext types
SparkContext
SparkSession
StreamingContext
SQLContext
Array[String] => Unit
// vs
(Json, ?) => Json

60. GeneralizeGeneralize
// developer
// pi estimation: samples => pi
(Int, SparkContext) => Double
// word count - input path => output path
(String, SparkSession) => String
...
(A, ?) => B
(A, B, ?) => C

61. SPARK FUNCTIONSPARK FUNCTION

62. A lot of actions over ssh-session to test, deploy and
run your code
Improssible to run job without direct shell
command
Process more that one job in parallel is difficult
Poor interfaces

63. Future plansFuture plans
Release 1.0.0
AWS EMR integration - on-demand EMR clusters
...