This document discusses writing a domain specific language (DSL) for data transformations using applicative functors in Scala. It introduces the concepts of Picker, Reader, and Result to parse heterogeneous data formats into a common format. Reader is defined as an applicative functor to allow combining multiple readers. Later, Reader is enhanced to take type parameters for both input and output to avoid reparsing data and support XML parsing. Type lambdas are used to make Reader work as an applicative functor.

1. Writing DSL with
Applicative Functors
David Galichet
Freelance functional programmer
!
twitter: @dgalichet

2. Content normalization
• We want to parse heterogeneous data formats
(CSV, XML …) and transform them to a pivot format
(Scala object in our case)
• The transformation should be described as a DSL
• This DSL must be simple to use, and enable any
kinds of data transformations or verifications

3. Expected DSL format
val reader = ( !
Pick(0).as[String].map(_.capitalize) and !
Pick(1).as[Date].check(_.after(now())) !
).reduce(FutureEvent)!
!
reader("PSUG; 21/08/2014") // returns Success(FutureEvent("PSUG",
date))
Inspired by Play2 Json API

4. tag: step1
Conventions & material
• Code is available on Github : https://github.com/
dgalichet/PsugDSLWritingWithApplicative
• Code revision is define on the top of any slides
including source code (just checkout the specified
tag)

5. tag: step1
Reading single entry
• We have a CSV line and we want to read one column
• We will introduce several abstractions:
• Picker: fetch a data from CSV or XML
• Result: either a Success or a Failure
• Converter: convert value from Picker to Reader
• Reader: container with methods to process its
content

6. tag: step1
Introducing Picker
case class Picker(p: String => Result[String]) {!
def as[T](implicit c: Converter[T]): Reader[T] =
c.convert(p)!
}!
!
object CsvPicker {!
def apply[T](i: Int)(implicit separator: Char): Picker =
Picker { s: String =>!
val elems = s.trim.split(separator)!
if (i > 0 && elems.size > i) Success(elems(i).trim)!
else Failure(s"No column ${i} for ${s}")!
}!
}

7. tag: step1
Introducing Picker
case class Picker(p: String => Result[String]) {!
def as[T](implicit c: Converter[T]): Reader[T] =
c.convert(p)!
}!
!
object CsvPicker {!
def apply[T](i: Int)(implicit separator: Char): Picker =
Picker { s: String =>!
val elems = s.trim.split(separator)!
if (i > 0 && elems.size > i) Success(elems(i).trim)!
else Failure(s"No column ${i} for ${s}")!
}!
}
Picker wraps a function from String to Result

8. The Result
tag: step1
sealed trait Result[+T]
case class Success[T](t: T) extends Result[T]
case class Failure(error: String) extends Result[Nothing]!

9. The Converter
tag: step1
trait Converter[T] {
def convert(p: String => Result[String]): Reader[T]
}!
!
object Converter {!
implicit val string2StringConverter = new Converter[String] {!
override def convert(p: String => Result[String]) = Reader[String]
(p)!
// See code on Github for more converters!
}

10. The Converter
tag: step1
trait Converter[T] {
def convert(p: String => Result[String]): Reader[T]
}!
!
Convert the content of the Picker to a Reader
!
object Converter {!
implicit val string2StringConverter = new Converter[String] {!
override def convert(p: String => Result[String]) = Reader[String]
(p)!
// See code on Github for more converters!
}

11. The Reader
tag: step1
case class Reader[O](p: String => Result[O]) {
def apply(s: String): Result[O] = p(s)
}
A Reader doesn’t contain a value but a process to
transform original data (CSV line or XML) to a
Result

12. Usage sample
tag: step1
import Converter._ // import implicit converters!
implicit val separator = ‘;’!
!
CsvPicker(1).as[String].apply("foo;bar") === "bar"

13. tag: step2
Enhancing the Reader
• The first defines a very simple Reader. We must
add a method to combine two instances of Reader
• We will also enhance Failure to store multiple
error messages

14. tag: step2
Enhancing the Reader
case class Reader[O](p: String => Result[O]) {!
def apply(s: String): Result[O] = p(s)!
!
def and[O2](r2: Reader[O2]): Reader[(O, O2)] = Reader { s: String =>!
(p(s), r2.p(s)) match {!
case (Success(s1), Success(s2)) => Success((s1, s2))!
case (Success(_), Failure(f)) => Failure(f)!
case (Failure(f), Success(_)) => Failure(f)!
case (Failure(f1), Failure(f2)) => Failure(f1 ++ f2)!
}!
}!
def map[T](f: O => T): Reader[T] = Reader { s: String =>!
p(s) match {!
case Success(o) => Success(f(o))!
case f: Failure => f!
}!
}!
def reduce[T] = map[T] _ // alias for map!
}

15. tag: step2
Enhancing Result type
sealed trait Result[+T]!
case class Success[T](t: T) extends Result[T]!
case class Failure(error: NonEmptyList[String]) extends
Result[Nothing]!
!
object Failure {!
def apply(s: String): Failure = Failure(NEL(s))!
}!
!
case class NonEmptyList[T](head: T, tail: List[T]) {
def toList = head::tail
def ++(l2: NonEmptyList[T]): NonEmptyList[T] = NonEmptyList(head,
tail ++ l2.toList)
}
object NEL {
def apply[T](h: T, t: T*) = NonEmptyList(h, t.toList)
}

16. Usage sample
tag: step2
implicit val separator = ';'
implicit val dtFormatter = new SimpleDateFormat("dd/MM/yyyy")
import Converter.string2StringConverter
import Converter.string2DateConverter!
!
val reader = (
CsvPicker(1).as[String] and
CsvPicker(2).as[Date]
).reduce { case (n, d) => FutureEvent(n, d) }!
reader("foo;bar;12/10/2014") === Success(FutureEvent("bar",
dtFormatter.parse("12/10/2014")))!
!
case class FutureEvent(name: String, dt: Date)

17. tag: step2
Usability problem
• The use of reduce (or map) method to transform a
Reader[(0, 02)] into an instance of
Reader[FutureEvent] for example is quite
verbose
• This will be even more verbose for instances of
Reader[(0, (02, 03))]
• We want the API to automatically bind tuple
elements to a constructor as we can encounter in
Play2 Json API

18. tag: step3
Applicative functors
• To tackle our problem, we will use Applicative
Functors and play2 functional library (and
especially FunctionalBuilder)
• This approach is inspired by @sadache (Sadek
Drobi) article https://gist.github.com/sadache/
3646092
• An Applicative Functor is a Type Class relying on
ad-hoc polymorphism to extends a Class with some
properties
• Play2 functional library (or Scalaz) provides
mechanism to compose Applicatives in a smart way

19. tag: step3
Applicative functors
M is an Applicative Functor if there exists the following methods :
def pure[A](a: A): M[A]
def map[A, B](m: M[A], f: A => B): M[B]
def apply[A, B](mf: M[A => B], ma: M[A]): M[B]!
with the following Laws :
• Identity: apply(pure(identity), ma) === ma where ma is an Applicative M[A]
• Homomorphism: apply(pure(f), pure(a)) === pure(f(a)) where f: A =>
B and a an instance of A
• Interchange: mf if an instance of M[A => B]
apply(mf, pure(a)) === apply(pure {(g: A => B) => g(a)}, mf)!
• Composition: map(ma, f) === apply(pure(f), ma)

20. tag: step3
Applicative functors
trait Applicative[M[_]] {
def pure[A](a: A): M[A]
def map[A, B](m: M[A], f: A => B): M[B]
def apply[A, B](mf: M[A => B], ma: M[A]): M[B]
} Applicative is an Higher Kinded type
(parameterized with M that take a single type parameter)

21. tag: step3
Reader is an Applicative
case class Reader[O](p: String => Result[O]) {
def apply(s: String): Result[O] = p(s)!
def map[T](f: O => T): Reader[T] = Reader { s: String =>
p(s) match {
case Success(o) => Success(f(o))
case f: Failure => f
}
}!
}!
object Reader {
def map2[O, O1, O2](r1: Reader[O1], r2: Reader[O2])(f: (O1, O2) =>
O): Reader[O] = Reader { s: String =>
(r1.p(s), r2.p(s)) match {
case (Success(s1), Success(s2)) => Success(f(s1, s2))
case (Success(_), Failure(e)) => Failure(e)
case (Failure(e), Success(_)) => Failure(e)
case (Failure(e1), Failure(e2)) => Failure(e1 ++ e2)
}
} …!
}

22. tag: step3
Reader is an Applicative
object Reader {
… // map2
implicit val readerIsAFunctor: Functor[Reader] = new
Functor[Reader] {
override def fmap[A, B](m: Reader[A], f: (A) => B) = m.map(f)
}
implicit val readerIsAnApplicative: Applicative[Reader] = new
Applicative[Reader] {
override def pure[A](a: A) = Reader { _ => Success(a) }
override def apply[A, B](mf: Reader[A => B], ma: Reader[A]) =
map2(mf, ma)((f, a) => f(a))
override def map[A, B](m: Reader[A], f: A => B) = m.map(f)
}
}

23. Usage sample
tag: step3
import Converter.string2StringConverter
import Converter.string2DateConverter!
import play.api.libs.functional.syntax._
import Reader.readerIsAnApplicative!
!
implicit val separator = ';'
implicit val dtFormatter = new SimpleDateFormat("dd/MM/yyyy")!
!
val reader = (
CsvPicker(1).as[String] and
CsvPicker(2).as[Date]
)(FutureEvent) // here we use CanBuild2.apply
reader("foo;bar;12/10/2014") === Success(FutureEvent("bar",
dtFormatter.parse("12/10/2014")))!

24. Usage sample
tag: step3
(errors accumulation)
import Converter.string2StringConverter
import Converter.string2DateConverter!
import play.api.libs.functional.syntax._
import Reader.readerIsAnApplicative!
!
implicit val separator = ';'
implicit val dtFormatter = new SimpleDateFormat("dd/MM/yyyy")!
!
val reader = (
CsvPicker(1).as[Int] and
CsvPicker(2).as[Date]
)((_, _))
reader(List("foo", "not a number", "not a date")) === Failure(NEL(!
"Unable to format 'not a number' as Int", !
"Unable to format 'not a date' as Date"))!

25. Benefits
tag: step3
• Making Reader an Applicative Functor give ability
to combine efficiently instances of Reader
• Due to Applicative properties, we still accumulate
errors
• Play2 functional builder give us a clean syntax to
define our DSL

26. tag: step4
Introducing XML Picker
case class Picker(p: String => Result[String]) {
def as[T](implicit c: Converter[T]): Reader[T] = c.convert(p)
}!
!
object XmlPicker {
def apply[T](query: Elem => NodeSeq): Picker = Picker { s: String =>
try {
val xml = XML.loadString(s)
Success(query(xml).text)
} catch {
case e: Exception => Failure(e.getMessage)
}
}
}!

27. Usage sample
tag: step4
import play.api.libs.functional.syntax._
import Reader.readerIsAnApplicative!
import Converter._
implicit val dF = new SimpleDateFormat("dd/MM/yyyy")!
val xml = """
<company name="Dupont and Co">
<owner>
<person firstname="jean" lastname="dupont" birthdate="11/03/1987"/>
</owner>
</company>"""
val r = (
XmlPicker(_ "person" "@firstname").as[String] and
XmlPicker(_ "person" "@lastname").as[String] and
XmlPicker(_ "person" "@birthdate").as[Date]
)(Person)!
r(xml) === Success(Person("jean","dupont",dF.parse("11/03/1987")))
case class Person(firstname: String, lastname: String, birthDt: Date)

28. tag: step4
Implementation problem
• The Reader[O] takes a type argument for the
output. The input is always a String
• With this implementation, an XML content will be
parsed (with XML.load) as many times as we use
XmlPicker. This will cause unnecessary overhead
• We will have the same issue (with lower overhead)
with our CsvPicker

29. tag: step5
Introducing Reader[I, 0]
To resolve this problem, we will modify Reader to
take a type parameter for the input

30. tag: step5
Introducing Reader[I, 0]
case class Reader[I, O](p: I => Result[O]) {
def apply(s: I): Result[O] = p(s)
def map[T](f: O => T): Reader[I, T] = Reader { s: I =>
p(s) match {
case Success(o) => Success(f(o))
case f: Failure => f
}
}!
}!
object Reader {
def map2[I, O, O1, O2](r1: Reader[I, O1], r2: Reader[I, O2])(f: (O1,
O2) => O): Reader[I, O] = Reader { s: I =>
(r1.p(s), r2.p(s)) match {
case (Success(s1), Success(s2)) => Success(f(s1, s2))
case (Success(_), Failure(e)) => Failure(e)
case (Failure(e), Success(_)) => Failure(e)
case (Failure(e1), Failure(e2)) => Failure(e1 ++ e2)
}
}

31. tag: step5
Introducing Reader[I, 0]
object Reader {!
implicit def readerIsAFunctor[I] = new Functor[({type λ[A] =
Reader[I, A]})#λ] {
override def fmap[A, B](m: Reader[I, A], f: (A) => B) = m.map(f)
}
implicit def readerIsAnApplicative[I] = new Applicative[({type λ[A] =
Reader[I, A]})#λ] {
override def pure[A](a: A) = Reader { _ => Success(a) }
override def apply[A, B](mf: Reader[I, A => B], ma: Reader[I, A]) =
map2(mf, ma)((f, a) => f(a))
override def map[A, B](m: Reader[I, A], f: (A) => B) = m.map(f)
}

32. tag: step5
What are Type lambdas ?
If we go back to Applicative definition, we can see that it’s
an Higher Kinded type (same with Functor) :
!
trait Applicative[M[_]] { … } // Applicative accept
parameter M that take itself any type as parameter!
!
Our problem is that Reader[I, 0] takes two parameters
but Applicative[M[_]] accept types M with only one
parameter. We use Type Lambdas to resolve this issue:
!
new Applicative[({type λ[A] = Reader[I, A]})#λ]!

33. tag: step5
Go back to Reader[I, 0]
object Reader {!
implicit def readerIsAFunctor[I] = new Functor[({type λ[A] = Reader[I,
A]})#λ] {
override def fmap[A, B](m: Reader[I, A], f: A => B) = m.map(f)
}
implicit def readerIsAnApplicative[I] = new Applicative[({type λ[A] =
Reader[I, A]})#λ] {
override def pure[A](a: A) = Reader { _ => Success(a) }
override def apply[A, B](mf: Reader[I, A => B], ma: Reader[I, A]) =
map2(mf, ma)((f, a) => f(a))
override def map[A, B](m: Reader[I, A], f: A => B) = m.map(f)
}

34. tag: step5
Go back to Reader[I, 0]
object Reader {!
import scala.language.implicitConversions
// Here we help the compiler a bit. Thanks @skaalf (Julien Tournay) !
// and https://github.com/jto/validation
implicit def fcbReads[I] = functionalCanBuildApplicative[({type λ[A] =
Reader[I, A]})#λ]
implicit def fboReads[I, A](a: Reader[I, A])(implicit fcb:
FunctionalCanBuild[({type λ[x] = Reader[I, x]})#λ]) = new
FunctionalBuilderOps[({type λ[x] = Reader[I, x]})#λ, A](a)(fcb)

35. Converter[I, T]
tag: step5
trait Converter[I, T] {
def convert(p: I => Result[String]): Reader[I, T]
}!
object Converter {
implicit def stringConverter[I] = new Converter[I, String] {
override def convert(p: I => Result[String]) = Reader[I, String](p)
}!
!
implicit def dateConverter[I](implicit dtFormat: DateFormat) = new
Converter[I, Date] {
override def convert(p: I => Result[String]) = Reader[I, Date] { s:
I =>
p(s) match {
case Success(dt) => try { !
Success(dtFormat.parse(dt))
} catch { case e: ParseException => Failure(s"...") }
case f: Failure => f
}}}

36. Picker[I]
tag: step5
case class Picker[I](p: I => Result[String]) {
def as[T](implicit c: Converter[I, T]): Reader[I, T] = c.convert(p)
}
object CsvPicker {
def apply[T](i: Int): Picker[List[String]] = Picker { elems:
List[String] =>
if (i > 0 && elems.size > i) Success(elems(i).trim)
else Failure(s"No column ${i} found in ${elems.mkString(";")}")
}}
object XmlPicker {
def apply[T](query: Elem => NodeSeq): Picker[Elem] = Picker { elem:
Elem =>
try {
Success(query(elem).text)
} catch {
case e: Exception => Failure(e.getMessage)
}}}!

37. Usage sample
tag: step5
import play.api.libs.functional.syntax._
import Reader._!
import Converter._!
implicit val dF = new SimpleDateFormat("dd/MM/yyyy")!
val xml = XML.loadString("""
<company name="Dupont and Co">
<owner>
<person firstname="jean" lastname="dupont" birthdate="11/03/1987"/>
</owner>
</company>""")!
!
val r = (
XmlPicker(_ "person" "@firstname").as[String] and
XmlPicker(_ "person" "@lastname").as[String] and
XmlPicker(_ "person" "@birthdate").as[Date]
)(Person)!
r(xml) === Success(Person("jean", "dupont", dF.parse("11/03/1987")))

38. tag: step6
Adding combinators
• We now add new abilities to Reader
• We especially want a method to validate content

39. tag: step6
Adding combinators
case class Reader[I, O](p: I => Result[O]) {!
!
def flatMap[T](f: O => Reader[I, T]): Reader[I, T] = Reader { s: I =>
p(s) match {
case Success(o) => f(o)(s)
case f: Failure => f
}
}
def verify(f: O => Result[O]): Reader[I, O] = flatMap { o: O =>
Reader( _ => f(o)) }!
}

40. Usage sample
tag: step6
val r: Reader[String, String] = Reader { Success(_) }
r.verify { x => if (x == "OK") Success(x) else Failure("KO") }("OK")
=== Success("OK")

41. Conclusion
• We have created a simple and powerful DSL for
processing CSV and XML content
• This DSL give us ability to Pick data, transform and
verify it and also accumulate encountered errors
• We have seen that making Reader an instance of the
Applicative Functor Type Class add it new capabilities
• Using ad-hoc polymorphism using Type Classes gives
us ability to extends Reader without altering it

42. Follow-up
• Type Classes are defined by functions that must be
implemented with regards to their laws (left/right
identity …)
• Proving the correctness of Type Class Laws can
be a bit tricky we usual approach
• I will introduce the framework ScalaCheck at
scala.io 2014, and show how to test them

43. Follow-up
• In the roadmap that has been announced by
@typesafe (http://scala-lang.org/news/roadmap-next),
it seems that Scala 2.14 (aka « Don
Giovanni ») will clean up lambda types syntax