Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Quark: A Purely-Functional
Scala DSL for Data
Processing & Analytics
John A. De Goes
@jdegoes - http://degoes.net
Apache Spark
Apache Spark is a fast and general engine for big data
processing, with built-in modules for streaming, SQL,
...
Spark Sucks
— Functional-ish
— Exceptions, typecasts
— SparkContext
— Serializable
— Unsafe type-safe programs
— Second-cl...
Why Does Spark Have to Suck?
Computation
val textFile = sc.textFile("hdfs://...")
val counts =
textFile.flatMap(line => li...
WWFPD?
— Purely functional
— No exceptions, no casts, no nulls
— No global variables
— No serialization
— Safe type-safe p...
Rule #1 in Functional
Programming
Don't solve the problem, describe the solution.
AKA the "Do Nothing" rule
=> Don't compu...
Quark
Compilation
Quark is a Scala DSL built on Quasar Analytics, a general-
purpose compiler for translating data process...
More Quark
Compilation
val dataset = Dataset.load("/prod/profiles")
val averageAge = dataset.groupBy(_.country[Str]).map(_...
Quark Targets
One DSL to Rule Them All
— MongoDB
— Couchbase
— MarkLogic
— Hadoop / HDFS
— Add your connector here!
Both Quark and Quasar Analytics are purely-functional,
open source projects written in 100% Scala.
https://github.com/quas...
How To DSL
Adding Integers
sealed trait Expr
final case class Integer(v: Int) extends Expr
final case class Addition(v: Ex...
How To DSL
Adding Strings
sealed trait Expr
final case class Integer(v: Int) extends Expr
final case class Addition(l: Exp...
How To DSL
Phantom Type
sealed trait Expr[A]
final case class Integer(v: Int) extends Expr[Int]
final case class Addition(...
How To DSL
GADTs in Scala still have bugs
SI-8563, SI-9345, SI-6680
FRIENDS DON'T LET FRIENDS USE GADTS IN SCALA.
How To DSL
Finally Tagless
trait Expr[F[_]] {
def int(v: Int): F[Int]
def str(v: String): F[String]
def add(l: F[Int], r: ...
How To DSL
Finally Tagless
type Id[A] = A
def interpret: Expr[Id] = new Expr[Id] {
def int(v: Int): Id[Int] = v
def str(v:...
Quark 101
The Building Blocks
— Type. Represents a reified type of an element in a dataset.
— **Dataset[A]**. Represents a...
Let's Build Us a Mini-Quark!
Mini-Quark
Type System
sealed trait Type
object Type {
final case class Unknown() extends Type
final case class Timestamp(...
Mini-Quark
Set-Level Operations
sealed trait SetOps[F[_]] {
def read(path: String): F[Unknown]
}
Mini-Quark
Dataset
sealed trait Dataset[A] {
def apply[F[_]](implicit F: SetOps[F]): F[A]
}
object Dataset {
def read(path...
Mini-Quark
Mapping
sealed trait SetOps[F[_]] {
def read(path: String): F[Unknown]
def map[A, B](v: F[A], f: ???) // What g...
Mini-Quark
Mapping: Attempt #1
sealed trait SetOps[F[_]] {
def read(path: String): F[Unknown]
def map[A, B](v: F[A], f: F[...
Mini-Quark
Mapping: Attempt #2
sealed trait MappingFunc[A, B] {
def apply[F[_]](v: F[A])(implicit F: MappingOps[F]): F[B]
...
Mini-Quark
Mapping: Attempt #2
trait SetOps[F[_]] {
def read(path: String): F[Unknown]
def map[A, B](v: F[A], f: MappingFu...
Mini-Quark
Dataset: Mapping
sealed trait Dataset[A] {
def apply[F[_]](implicit F: SetOps[F]): F[A]
def map[B](f: ???): Dat...
Mini-Quark
Dataset: Mapping Attempt #1
sealed trait Dataset[A] { self =>
def apply[F[_]](implicit F: SetOps[F]): F[A]
def ...
Mini-Quark
Dataset: Mapping Attempt #2
sealed trait Dataset[A] {
def apply[F[_]](implicit F: SetOps[F]): F[A]
def map[B](f...
Mini-Quark
Dataset: Mapping Binary Operators
val netProfit = dataset.map(v => v.netRevenue[Dec] - v.netCosts[Dec])
Mini-Quark
MappingFuncs Are Arrows!
trait MappingFunc[A <: Type, B <: Type] extends Dynamic { self =>
import MappingFunc.C...
Mini-Quark
Applicative Composition
MappingFunc[A, B]
A -----------------------------B
 /
 /
 /
 / MappingFunc[A, B ⊕ C]
 /...
Learn More
— Finally Tagless: http://okmij.org/ftp/tagless-final/
— Quark: https://github.com/quasar-analytics/quark
— Qua...
Upcoming SlideShare
Loading in …5
×

Quark: A Purely-Functional Scala DSL for Data Processing & Analytics

5,035 views

Published on

Quark is a new Scala DSL for data processing and analytics that runs on top of the Quasar Analytics compiler. Quark is adept at processing semi-structured data and compiles query plans to operations that run entirely inside a target data source. In this presentation, John A. De Goes provides an overview of the open source library, showing several use cases in data processing and analytics. John also demonstrates a powerful technique that every developer can use to create their own purely-functional, type-safe DSLs in the Scala programming language.

Published in: Technology

Quark: A Purely-Functional Scala DSL for Data Processing & Analytics

  1. 1. Quark: A Purely-Functional Scala DSL for Data Processing & Analytics John A. De Goes @jdegoes - http://degoes.net
  2. 2. Apache Spark Apache Spark is a fast and general engine for big data processing, with built-in modules for streaming, SQL, machine learning and graph processing. val textFile = sc.textFile("hdfs://...") val counts = textFile.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _)
  3. 3. Spark Sucks — Functional-ish — Exceptions, typecasts — SparkContext — Serializable — Unsafe type-safe programs — Second-class support for databases — Dependency hell (>100) — Painful debugging — Implementation-dependent performance
  4. 4. Why Does Spark Have to Suck? Computation val textFile = sc.textFile("hdfs://...") val counts = textFile.flatMap(line => line.split(" ")) <---- Where Spark goes wrong .map(word => (word, 1)) <---- Where Spark goes wrong .reduceByKey(_ + _) <---- Where Spark goes wrong
  5. 5. WWFPD? — Purely functional — No exceptions, no casts, no nulls — No global variables — No serialization — Safe type-safe programs — First-class support for databases — Few dependencies — Better debugging — Implementation-independent performance
  6. 6. Rule #1 in Functional Programming Don't solve the problem, describe the solution. AKA the "Do Nothing" rule => Don't compute, embed a compiled language into Scala
  7. 7. Quark Compilation Quark is a Scala DSL built on Quasar Analytics, a general- purpose compiler for translating data processing over semi-structured data into efficient plans that execute 100% inside the target infrastructure. val textFile = Dataset.load("...") val counts = textFile.flatMap(line => line.typed[Str].split(" ")) .map(word => (word, 1)) .reduceByKey(_.sum)
  8. 8. More Quark Compilation val dataset = Dataset.load("/prod/profiles") val averageAge = dataset.groupBy(_.country[Str]).map(_.age[Int]).reduceBy(_.average)
  9. 9. Quark Targets One DSL to Rule Them All — MongoDB — Couchbase — MarkLogic — Hadoop / HDFS — Add your connector here!
  10. 10. Both Quark and Quasar Analytics are purely-functional, open source projects written in 100% Scala. https://github.com/quasar-analytics/
  11. 11. How To DSL Adding Integers sealed trait Expr final case class Integer(v: Int) extends Expr final case class Addition(v: Expr, v: Expr) extends Expr def int(v: Int): Expr = Integer(v) def add(l: Expr, r: Expr): Expr = Addition(l, r) add(add(int(1), int(2)), int(3)) : Expr def interpret(e: Expr): Int = e match { case Integer(v) => v case Addition(l, r) => interpret(l) + interpret(r) } def serialize(v: Expr): Json = ??? def deserialize(v: Json): Expr = ???
  12. 12. How To DSL Adding Strings sealed trait Expr final case class Integer(v: Int) extends Expr final case class Addition(l: Expr, r: Expr) extends Expr // Uh, oh! final case class Str(v: String) extends Expr final case class StringConcat(l: Expr, r: Expr) extends Expr // Uh, oh!
  13. 13. How To DSL Phantom Type sealed trait Expr[A] final case class Integer(v: Int) extends Expr[Int] final case class Addition(l: Expr[Int], r: Expr[Int]) extends Expr[Int] final case class Str(v: String) extends Expr[String] final case class StringConcat(l: Expr[String], r: Expr[String]) extends Expr[String] def interpret[A](e: Expr[A]): A = e match { case Integer(v) => v case Addition(l, r) => interpret(l) + interpret(r) case Str(v) => v case StringConcat(l, r) => interpret(l) ++ interpret(r) } def serialize[A](v: Expr[A]): Json = ??? def deserialize[Z](v: Json): Expr[A] forSome { type A } = ???
  14. 14. How To DSL GADTs in Scala still have bugs SI-8563, SI-9345, SI-6680 FRIENDS DON'T LET FRIENDS USE GADTS IN SCALA.
  15. 15. How To DSL Finally Tagless trait Expr[F[_]] { def int(v: Int): F[Int] def str(v: String): F[String] def add(l: F[Int], r: F[Int]): F[Int] def concat(l: F[String], r: F[String]): F[String] } trait Dsl[A] { def apply[F[_]](implicit F: Expr[F]): F[A] } def int(v: Int): Dsl[Int] = new Dsl[Int] { def apply[F[_]](implicit F: Expr[F]): F[Int] = F.int(v) } def add(l: Dsl[Int], r: Dsl[Int]): Dsl[Int] = new Dsl[Int] { def apply[F[_]](implicit F: Expr[F]): F[Int] = F.add(l.apply[F], r.apply[F]) } // ...
  16. 16. How To DSL Finally Tagless type Id[A] = A def interpret: Expr[Id] = new Expr[Id] { def int(v: Int): Id[Int] = v def str(v: String): Id[String] = v def add(l: Id[Int], r: Id[Int]): Id[Int] = l + r def concat(l: Id[String], r: Id[String]): Id[String] = l + r } add(int(1), int(2)).apply(interpret) // Id(3) final case class Const[A, B](a: A) def serialize: Expr[Const[Json, ?]] = ??? def deserialize[F[_]: Expr](json: Json): F[A] forSome { type A } = ???
  17. 17. Quark 101 The Building Blocks — Type. Represents a reified type of an element in a dataset. — **Dataset[A]**. Represents a dataset, produced by successive application of set-level operations (SetOps). Describes a directed- acyclic graph. — **MappingFunc[A, B]**. Represents a function from A to B that is produced by successive application of mapping-level operations (MapOps) to the input. — **ReduceFunc[A, B]**. Represents a reduction from A to B, produced by application of reduction-level operations (ReduceOps) to the input.
  18. 18. Let's Build Us a Mini-Quark!
  19. 19. Mini-Quark Type System sealed trait Type object Type { final case class Unknown() extends Type final case class Timestamp() extends Type final case class Date() extends Type final case class Time() extends Type final case class Interval() extends Type final case class Int() extends Type final case class Dec() extends Type final case class Str() extends Type final case class Map[A <: Type, B <: Type](key: A, value: B) extends Type final case class Arr[A <: Type](element: A) extends Type final case class Tuple2[A <: Type, B <: Type](_1: A, _2: B) extends Type final case class Bool() extends Type final case class Null() extends Type type UnknownMap = Map[Unknown, Unknown] val UnknownMap : UnknownMap = Map(Unknown(), Unknown()) type UnknownArr = Arr[Unknown] val UnknownArr : UnknownArr = Arr(Unknown()) type Record[A <: Type] = Map[Str, A] type UnknownRecord = Record[Unknown] }
  20. 20. Mini-Quark Set-Level Operations sealed trait SetOps[F[_]] { def read(path: String): F[Unknown] }
  21. 21. Mini-Quark Dataset sealed trait Dataset[A] { def apply[F[_]](implicit F: SetOps[F]): F[A] } object Dataset { def read(path: String): Dataset[Unknown] = new Dataset[Unknown] { def apply[F[_]](implicit F: SetOps[F]): F[Unknown] = F.read(path) } }
  22. 22. Mini-Quark Mapping sealed trait SetOps[F[_]] { def read(path: String): F[Unknown] def map[A, B](v: F[A], f: ???) // What goes here? }
  23. 23. Mini-Quark Mapping: Attempt #1 sealed trait SetOps[F[_]] { def read(path: String): F[Unknown] def map[A, B](v: F[A], f: F[A] => F[B]) // Doesn't really work... }
  24. 24. Mini-Quark Mapping: Attempt #2 sealed trait MappingFunc[A, B] { def apply[F[_]](v: F[A])(implicit F: MappingOps[F]): F[B] } trait MappingOps[F[_]] { def str(v: String): F[Type.Str] def project[K <: Type, V <: Type](v: F[Type.Map[K, V]], k: F[K]): F[V] def add(l: F[Type.Int], r: F[Type.Int]): F[Type.Int] def length[A <: Type](v: F[Type.Arr[A]]): F[Type.Int] ... } object MappingOps { def id[A]: MappingFunc[A, B] = new MappingFunc[A, A] { def apply[F[_]](v: F[A])(implicit F: MappingOps[F]): F[A] = v } }
  25. 25. Mini-Quark Mapping: Attempt #2 trait SetOps[F[_]] { def read(path: String): F[Unknown] def map[A, B](v: F[A], f: MappingFunc[A, B]): F[B] // Yay!!! }
  26. 26. Mini-Quark Dataset: Mapping sealed trait Dataset[A] { def apply[F[_]](implicit F: SetOps[F]): F[A] def map[B](f: ???): Dataset[B] = ??? // What goes here??? } object Dataset { def read(path: String): Dataset[Unknown] = new Dataset[Unknown] { def apply[F[_]](implicit F: SetOps[F]): F[Unknown] = F.read(path) } }
  27. 27. Mini-Quark Dataset: Mapping Attempt #1 sealed trait Dataset[A] { self => def apply[F[_]](implicit F: SetOps[F]): F[A] def map[B](f: MappingFunc[A, B]): Dataset[B] = new Dataset[B] { def apply[F[_]](implicit F: SetOps[F]): F[B] = F.map(self.apply, f) } } object Dataset { def read(path: String): Dataset[Unknown] = new Dataset[Unknown] { def apply[F[_]](implicit F: SetOps[F]): F[Unknown] = F.read(path) } } // dataset.map(_.length) // Cannot ever work! // dataset.map(v => v.profits[Dec] - v.losses[Dec]) // Cannot ever work!
  28. 28. Mini-Quark Dataset: Mapping Attempt #2 sealed trait Dataset[A] { def apply[F[_]](implicit F: SetOps[F]): F[A] def map[B](f: MappingFunc[A, A] => MappingFunc[A, B]): Dataset[B] = new Dataset[B] { def apply[F[_]](implicit F: SetOps[F]): F[B] = F.map(self.apply, f(MappingFunc.id[A])) } } object Dataset { def read(path: String): Dataset[Unknown] = new Dataset[Unknown] { def apply[F[_]](implicit F: SetOps[F]): F[Unknown] = F.read(path) } } // dataset.map(_.length) // Works with right methods on MappingFunc! // dataset.map(v => v.profits[Dec] - v.losses[Dec]) // Works with right methods on MappingFunc!
  29. 29. Mini-Quark Dataset: Mapping Binary Operators val netProfit = dataset.map(v => v.netRevenue[Dec] - v.netCosts[Dec])
  30. 30. Mini-Quark MappingFuncs Are Arrows! trait MappingFunc[A <: Type, B <: Type] extends Dynamic { self => import MappingFunc.Case def apply[F[_]: MappingOps](v: F[A]): F[B] def >>> [C <: Type](that: MappingFunc[B, C]): MappingFunc[A, C] = new MappingFunc[A, C] { def apply[F[_]: MappingOps](v: F[A]): F[C] = that.apply[F](self.apply[F](v)) } def + (that: MappingFunc[A, B])(implicit W: NumberLike[B]): MappingFunc[A, B] = new MappingFunc[A, B] { def apply[F[_]: MappingOps](v: F[A]): F[B] = MappingOps[F].add(self(v), that(v)) } def - (that: MappingFunc[A, B])(implicit W: NumberLike[B]): MappingFunc[A, B] = new MappingFunc[A, B] { def apply[F[_]: MappingOps](v: F[A]): F[B] = MappingOps[F].subtract(self(v), that(v)) } ... }
  31. 31. Mini-Quark Applicative Composition MappingFunc[A, B] A -----------------------------B / / / / MappingFunc[A, B ⊕ C] / MappingFunc[A, C] / / C
  32. 32. Learn More — Finally Tagless: http://okmij.org/ftp/tagless-final/ — Quark: https://github.com/quasar-analytics/quark — Quasar: https://github.com/quasar-analytics/quasar THANK YOU @jdegoes - http://degoes.net

×