Shapeless is used to create a generic and type-level API for Spark jobs. It allows defining job arguments and context in a typesafe way using HList and type classes. The boilerplate of defining job arguments and context is eliminated by using Shapeless combinators and type level programming.

1. Пользовательское API на базе
Shapeless

2. Челышов Вадим - qtankle@gmail.com
hydrosphere.io
Scalalaz Podcast

3. INTRO

5. Scala styles
better java
std scala fp
pure functional
generic + type level

6. Generic + type level

7. Shapeless

8. Generic + type level
akka-http
slick
finch
SBT

9. Spark app

10. Spark app
spark/bin/spark-submit
--class "SimpleApp"
--master local[4]
target/scala-2.11/simple-project_2.11-1.0.jar

11. Frontend

12. Frontend

13. Frontend

14. Mist
https://github.com/Hydrospheredata/mist

15. Library Api
object UserJob extends Job {
def execute(a: String, b: Int): Map[String, Any] = {
val wordsToCount = sc.textFile("hdfs://...").flatMap(line
wordsToCount
}
}

16. Минусы
serialization/deserialization
нельзя override def execute
inject spark context

17. Прототип
object MyJob extends Job[R] {
override def defineJob: JobDef[R] = ???
}

18. Прототип
override def define = {
args(
arg[Int]("x"),
arg[String]("str")
)
}

19. Прототип
object MyJob extends Job[Int] {
override def define = {
args(arg[String]("path")).withSparkContext(
(p: String, sc: SparkContext) => {
sc.textFile("hdfs://" + p)
})
}
}

20. Akka-http
val route =
parameters('foo, 'bar) { (foo, bar) =>
complete(s"Hello $foo $bar")
}

21. Finch
val route = get("foo" :: path[String]) { s: String =>
Ok(s"Hello $foo")
}

22. Sbt
packageBin in Compile <<= (name, organization, version) map {
(n, o, v) => file(o + "-" + n + "-" + v + ".jar")
}

23. Fake it till you make it

24. Arg
trait Arg[A]{
def extract(ctx: Ctx): Option[A]
}

25. Arg
def arg[A](name: String): Arg[A]
def arg[A](name: String, default: A): Arg[A]

26. spread your boilerplate
def args[A](a1: Arg[A]): Args1[A]
def args[A](a1: Arg[A], a2: Arg[B]): Args2[A, B]

27. и еще
class Args1[A](a1: Arg[A]) {
def withContext[Res](
f: (A, SparkContext) => Res): JobDef[Res]
}
class Args2[A, B](a1: Arg[A], a2: Arg[B]) {
def withContext[Res](
f: (A, B, SparkContext) => Res): JobDef[Res]
}

28. Все дороги ведут в shapeless
shapeless/project/Boilerplate.scala

29. boilerplate
block"""
|trait Methods {
- def args[${`A..N`}](${`a1..n`}): Args${arity}[${`A..N`}]
- new Args${arity}(${`a1..aN`})
|}
|
|object Methods extends Methods
"""

30. Что возвращать
trait Job[R] {
def define: JobDef[R]
}

31. No way
trait Job[A] {
def define: JobDef1[A, R]
def define: JobDef2[A, B, R]
}

32. Type member
trait Generic[A] {
def foo(a: A)
}
trait Member {
type A
def foo(a: A)
}

33. Сохранить типы
trait JobDef[R] {
type Args
def args: Args
def run(ctx: Ctx): R
}

34. HList
class Job1[A, R](
a: Arg[A]
f: (A, SparkContext) => R)
extends JobDef[R] {
type Args = A :: HNil
def args = a :: HNil
}

35. Последний рывок бойлерплейта
def run(ctx: Context): R = {
val options = args.map(a => a.extract(ctx.params)
if (options.exists(_.isEmpty))
//error
else {
val asTuple = ..
f.tupled(asTuple)
}
}

36. Это так не работает
import poly._
// а как сюда передать Ctx с параметрами??
object mapExtract extends (ArgDef ~> Option) {
def apply[T](a : ArgDef[T]) = a.extract(?)
}
https://milessabin.com/blog/2012/04/27/shapeless-
polymorphic-function-values-1/

37. Что получилось

38. Что получилось
21 - withArgs, ArgN, JobDefN
вот он тот boilrepalte который скрапит shapeless
придется читать сорцы

39. Понять type-level
type parameters
implicit paramters
type-params + implicit params = type level lang

40. Могучие типы
trait Foo[A]{
def map(f: A => B): Foo[B]
}

41. Могучие типы
val a: Int :: HNil = 1 :: HNil
val ab: Int :: String :: HNil = 1 :: "str" :: HNil

42. implicit ~= match
trait Show[A] {
def show(a: A): String
}
object Show {
implicit val forString: Show[String] = ...
implicit val forInt: Show[Int] = ...
}

43. implicit ~= match
trait DepFn[A] {
type Out
def apply(a: A): Out
}
object DepFn {
implicit val forString: DepFn[String]
implicit val forInt: DepFn[Int]
}

44. DepFn
def size(a: Any): Int = a match {
case i: Int = i
case s: String = s
}
trait Size[A] {
type Out
dep apply(a: A): Out
}
object Size[A] {
implicit val forStr: Size[String] = ...
implicit val forInt: Size[Int] = ...
}
def size[A](a: A)(implicit sz: Size[A]): sz.Out =
size(a)

45. DepFn
object ToHList {
implicit def tuple1[A] = new ToHList[Tuple1[A]] {
type Out = A :: HNil
def apply(a: Tuple1[A]): A :: HNil = a._1 :: HNil
}
}
def foo[A](a: A)(implicit thl: ToHList[A]): thl.Out =
thl(a)

46. Композиция depfn
def foo[A](a: A)(implicit
thl: ToHList[A],
rev: Reverse[?]
): rev.Out =
rev(thl(a))

47. Aux
object ToHList {
type Aux[A, Out0] = ToHList[A] {type Out = Out0 }
implicit def tuple1[A]: Aux[Tuple1[A], A :: HNil] =
new ToHList[Tuple1[A]] {
type Out = A :: HNil
def apply(a: Tuple1[A]): A :: HNil = a._1 :: HNil
}
}

48. Aux
def reverse[A, H <: HList](a: A)(
implicit
thl: ToHList.Aux[A, H],
r: Reverse[H]): r.Out = r(thl(a))

49. И это все?
The Type Astronaut's Guide to Shapeless
Type Level Programming in Scala step by step

50. Время поправить
object MyJob extends Job[Int] {
override def define = {
args(arg[String]("path"), arg[Int]("n")).withSparkContext(
(p: String, sc: SparkContext) => {
})
}
}

51. Scrap your boilerplate
class ArgsN[A ... N ](a1 ... aN)
class JobDefN[A .. N, R](
a1: Arg[A],
...
aN: Arg[N],
f: (A.. N, SparkContext) => R)
extends JobDef[R] {
type RunArgs = A :: N
def args = a :: n
}

52. Комбинаторы
trait Arg[A] { self =>
def &(b: Arg[B]): Arg[A :: B :: HNil] =
}

53. Комбинаторы
val a: ArgDef[A] = arg[A]("a")
val b: ArgDef[B] = arg[B]("b")
val c: ArgDef[C] = arg[C]("c")
val ab: ArgDef[A :: B :: HNil] = a & b
val abc: ArgDef[A :: B :: C :: HNil] = a & b & c

54. Extract
def &(b: Arg[B]): Arg[A :: B :: HNil] = {
new Arg[A :: B :: HNil] {
def extract(ctx: Ctx): Option[A :: B :: HNil] = {
val opt1 = self.extract(p)
val opt2 = b.extract(p)
(opt1, opt2) match {
case (Some(a), Some(b)) => Some(a :: b :: HNil)
}
}
}
}

55. Больше кейсов
val ab = arg[Int]("a") & arg[Int]("b")
val dc = arg[String]("c") & arg[Int]("d")
val abdc = ab & dc

56. Adjoin
def &(b: Arg[B])(
implicit
adj: Adjoin[A :: B :: HNil]
): Arg[adj.Out] = {
(opt1, opt2) match {
case (Some(a), Some(b)) => Some(adj(a :: b :: HNil))
}

57. Контекст это тоже аргумент
val sparkContext = new Arg[SparkContext] {
def extract(ctx: Ctx): = Some(ctx.sparkContext)
}
val args = arg[Int]("n") & sparkContext

58. FnToProduct
trait Arg[A] { self =>
def apply[F, Int <: HList, Res](f: F)(
implicit
fntp: FnToProduct.Aux[F, In => Res]): JobDef[Res] =
}
val args = arg[Int]("n") & sparkContex
val jobdef = args {(n, sparkContext) => ...}

59. adapted-args
def foo(ab: (Int, String, Double))
foo((1, "foo", 1.0))
foo(1, "foo", 1.0)
def args[A, H <: HList, ROut, Z](a : A)(
implicit
gen: Generic.Aux[A, H],
r: LeftReducer.Aux[H, reducer.type, ROut],
ev: ROut <:< ArgDef[Z]
)

60. Репортинг ошибок

61. implicitNotFound
@implicitNoFound("Not found Foo {A}")
trait Foo[A]
def foo[A](implicit foo: Foo[A])
foo(1) // Not found Foo Int

62. implicitNotFound
def bar[X, R])(x: X)(implicit x: Aux[X, R], foo: Foo[R])
bar(1) //Not found Foo R

63. Итог
generic + type level - необходиомость
с этим можно жить