In this introduction to type-level programming in Scala, we are going to discuss how we can leverage the full power of the type system to verify domain properties of an application at compile-time, instead of doing runtime verifications at the value-level

1. Exploring
Type-Level Programming
in Scala

2. JORGE VÁSQUEZ
SCALA DEVELOPER

4. Agenda
● Motivations around Type-Level Programming in Scala
● Background
○ Dependent types
○ Path-dependent types
○ Abstract type members
● Examples of libraries that use Type-Level
Programming

5. Motivations
around
Type-Level
Programming

6. Why Type-Level Programming?
● We turn to Scala for type-safety
● But… Sometimes it’s not enough!

7. Concrete example: Spark DataFrame API
We have lots of potential for runtime bugs!

8. Concrete example: Spark DataFrame API
df.select(
$"username",
$"tweet"
)

9. Concrete example: Spark DataFrame API
Error: org.apache.spark.sql.
AnalysisException: cannot
resolve '`username`'

10. Concrete example: Spark DataFrame API
df.select(
$"timestamp" * 10
)

11. Concrete example: Spark DataFrame API
Error:
org.apache.spark.sql.
AnalysisException: cannot
resolve '(`timestamp` *
10)' due to data type
mismatch

12. Concrete example: Spark DataFrame API
df.filter(
$"text" === true
)
// Does not fail

13. Concrete example: Spark DataFrame API
df.select(
$"text" / 1000
)
// Does not fail

15. Can we do better?
Scala has a VERY powerful type system,
why not use it?

16. Concrete example: Spark with Frameless
Frameless helps us to eliminate a lot of bugs…
At compile time!

17. Concrete example: Spark with Frameless
tds.select(
tds('username),
tds('tweet)
)

18. Concrete example: Spark with Frameless
// Error:
No column Symbol with
shapeless.tag.Tagged[Str
ing("username")]

19. Concrete example: Spark with Frameless
tds.select(
tds('timestamp) * 10
)

20. Concrete example: Spark with Frameless
// Error:
overloaded method value
* with alternatives
[...] cannot be applied
to (Int)

21. Concrete example: Spark with Frameless
tds.filter(
tds('text) === true
)

22. Concrete example: Spark with Frameless
// Error:
overloaded method value
=== with alternatives
[...] cannot be applied
to (Boolean)

23. Concrete example: Spark with Frameless
tds.select(
tds('text) / 1000
)

24. Concrete example: Spark with Frameless
// Error:
overloaded method value /
with alternatives [...]
cannot be applied to
(Int)

25. In conclusion...
Type-level Programming lets you eliminate
bugs at compile-time

26. In conclusion...

27. Our focus today: Dependent types
Dependent types are the heart of
Type-Level Programming in Scala

28. What are
Dependent Types?

29. What are Dependent Types?
● Dependent Types are types that depend on
values.
● With this, we remove the usual separation
between the type and value worlds.

30. What are Dependent Types?
● Scala is not a fully dependently typed
language.
● However, it supports some form of
Dependent Types, which is called Path
Dependent Types.

31. How we define Path Dependent Types?
● In Scala, we can define nested components
● For example, a class inside a trait, a
trait inside a class, etc.

32. How we define Path Dependent Types?
sealed trait Foo {
sealed trait Bar
}
val foo1 = new Foo {}
val foo2 = new Foo {}
val a: Foo#Bar = new foo1.Bar {} // OK
val b: Foo#Bar = new foo2.Bar {} // OK
val c: foo1.Bar = new foo1.Bar {} // OK
val d: foo2.Bar = new foo1.Bar {}
// Required: foo2.Bar, Found: foo1.Bar

33. How we define Path Dependent Types?
● Another useful tool is Abstract Type
Members, which are types we don’t know
yet and that we can define later
trait Bar {
type T
}

34. Example 1: Merging Files
Define a merge function, which should take:
● A list of files
● A merge strategy: Single/Multiple/None
● A callback function: Which should expect:
○ A single file if merge strategy is Single
○ A list of files if merge strategy is Multiple
○ A unit value if merge strategy is None

35. Example 1: Merging Files
import java.io.File
sealed trait MergeStrategy {
type Output
}
object MergeStrategy {
case object Single extends MergeStrategy { type Output = File }
case object Multiple extends MergeStrategy { type Output = List[File] }
case object None extends MergeStrategy { type Output = Unit }
}
def mergeFiles(files: List[File]): File = ???

36. Example 1: Merging Files
def merge[T](files: List[File], mergeStrategy: MergeStrategy)
(f: mergeStrategy.Output => T): T =
mergeStrategy match {
case MergeStrategy.Single => f(mergeFiles(files))
case MergeStrategy.Multiple => f(files)
case MergeStrategy.None => f(())
}

37. Example 1: Merging Files

38. Example 1: Merging Files
def merge[O, T](
files: List[File],
mergeStrategy: MergeStrategy { type Output = O }
)(f: O => T): T =
mergeStrategy match {
case MergeStrategy.Single => f(mergeFiles(files))
case MergeStrategy.Multiple => f(files)
case MergeStrategy.None => f(())
}

39. Example 1: Merging Files
val files: List[File] = ???
merge(files, MergeStrategy.Single) { file: File =>
// Do some processing
}
merge(files, MergeStrategy.Multiple) { files: List[File] =>
// Do some processing
}
merge(files, MergeStrategy.None) { _: Unit =>
// Do some processing
}

40. Example 2: Merging Elements
Define a merge function, which should take:
● A list of elements of any type
● A merge strategy: Single/Multiple/None
● A callback function: Which should expect:
○ A single element if merge strategy is Single
○ A list of elements if merge strategy is Multiple
○ A unit value if merge strategy is None

41. Example 2: Merging Elements
sealed trait MergeStrategy {
type Output[_]
}
object MergeStrategy {
case object Single extends MergeStrategy { type Output[A] = A }
case object Multiple extends MergeStrategy { type Output[A] = List[A] }
case object None extends MergeStrategy { type Output[_] = Unit }
}
def mergeElements[E](elements: List[E]): E = ???

42. Example 2: Merging Elements
def merge[E, O[_], T](
elements: List[E],
mergeStrategy: MergeStrategy { type Output[A] = O[A] }
)(f: O[E] => T): T =
mergeStrategy match {
case MergeStrategy.Single => f(mergeElements(elements))
case MergeStrategy.Multiple => f(elements)
case MergeStrategy.None => f(())
}

43. Example 2: Merging Elements
val messages: List[String] = ???
merge(messages, MergeStrategy.Single) { message: String =>
// Do some processing
}
merge(messages, MergeStrategy.Multiple) { messages: List[String] =>
// Do some processing
}
merge(messages, MergeStrategy.None) { _: Unit =>
// Do some processing
}

44. In conclusion...
● Path-dependent types are the heart and
soul of Scala's type system
● They help you to improve compile-time type
safety

45. In conclusion...
DOTTY = DOT Calculus = Path Dependent Types

46. Some example
libraries

47. Examples of libraries that use
Type Level Programming
● Shapeless: Generic programming
○ Generic Product Type: HList
○ Generic Sum Type: Coproduct

48. Examples of libraries that use
Type Level Programming
● Frameless: Expressive types for Spark

49. Examples of libraries that use
Type Level Programming
● Refined: Refinement types

50. Examples of libraries that use
Type Level Programming
● ZIO SQL: Type-safe SQL queries

51. References
● Dependent types in Scala, blog post by Yao Li
● Type Level Programming in Scala step by
step, blog series by Luigi Antonini
● The Type Astronaut’s Guide to Shapeless
Book, by Dave Gurnell
● Introduction to Apache Spark with Frameless,
by Brian Clapper

52. Special thanks
● To micro sphere.it organizers for hosting this
presentation
● To John De Goes for guidance and support

53. Thank You!

54. @jorvasquez2301
jorge.vasquez@scalac.io
jorge-vasquez-2301
Contact me