Upcoming SlideShare
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Standard text messaging rates apply

# Introduction to Iteratees (Scala)

3,360

Published on

Slides originating from a talk I gave at ScalaMUC on 2013-12-17. The corresponding code can be found at https://github.com/afwlehmann/iteratee-tutorial .

Slides originating from a talk I gave at ScalaMUC on 2013-12-17. The corresponding code can be found at https://github.com/afwlehmann/iteratee-tutorial .

14 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total Views
3,360
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
50
0
Likes
14
Embeds 0
No embeds

No notes for slide

### Transcript

• 1. History Iterator pattern Iteratees Discussion Introduction to Iteratees Motivation and implementation basics Alexander Lehmann <afwlehmann@googlemail.com> Technische Universit¨t M¨nchen a u December 17th, 2013 1/29
• 2. History Iterator pattern Iteratees Discussion Brief history 2008 Oleg Kiselyov, “Incremental multi-level input processing with left-fold enumerator”, DEFUN 2008 2010-05-12 John W. Lato, “Teaching an Old Fool New Tricks’, Monad Reader #16 ´ 2010-05-26 Available in scalaz 5.0 R´nar Oli u 2012-03-13 Initial release of the play framework 2.0 2/29
• 3. History Iterator pattern Iteratees Discussion We’ve often seen something along the lines of import io.Source.fromFile val it: Iterator[String] = fromFile("foo.txt").getLines var result = 0 while (it.hasNext) { val line = it.next result += line.toInt } // Do something with the result 3/29
• 4. History Iterator pattern Iteratees Discussion Issues Repetitive pattern (DRY principle) Manual pulling Mutability, imperative style No error handling (we sometimes just forget, right?) No (one|two)-way communication (Exceptions don’t count) Resource handling (how long do we need a resource and who is responsible for opening/closing/recovering it?) Missing or rather diﬃcult composability What if the input stream were inﬁnite? 4/29
• 5. History Iterator pattern Iteratees Discussion Try to be more “functional” and invert control... val it: Iterator[String] = fromFile(foo.txt).getLines var result = 0 it foreach { line = result += line.toInt } 5/29
• 6. History Iterator pattern Iteratees Discussion deﬁne ourselves a re-usable utility function... def enum(it: Iterator[String]): Int = { var result = 0 it foreach { line = result += line.toInt } result } val foo = enum(fromFile(foo.txt).getLines) 6/29
• 7. History Iterator pattern Iteratees Discussion being more versatile for the greater good... def enum(it: Iterator[String])(init: Int) (f: (Int, String) = Int): Int = { var result = init it foreach { line = result = f(result, line) } result } val it: Iterator[String] = fromFile(foo.txt).getLines val foo: Int = enum(it)(0)(_ + _.toInt) 7/29
• 8. History Iterator pattern Iteratees Discussion possibly even generic... def enum[In,Out](it: Iterator[In])(init: Out) (f: (Out, In) = Out): Out = { var result = init it foreach { x = result = f(result, x) } result } val it: Iterator[String] = fromFile(foo.txt).getLines val foo: Int = enum(it)(0)(_ + _.toInt) 8/29
• 9. History Iterator pattern Iteratees Discussion provide sophisticated error handling... def enum[In,Out](it: Iterator[In])(init: Out) (f: (Out, In) = Out): Out = { var result = init try { it foreach { x = result = f(result, x) } } catch { case t: Throwable = /* TODO (fingers crossed) */ } result } val it: Iterator[String] = fromFile(foo.txt).getLines val foo: Int = enum(it)(0)(_ + _.toInt) 9/29
• 10. History Iterator pattern Iteratees Discussion or rather use {your-favorite-“monad”-here} for error handling, asynchronism and composability... def enum[In,Out](it: Iterator[In])(init: Out) (f: (Out, In) = Out): Option[Out] = { try { var result = init it foreach { x = result = f(result, x) } Some(result) } catch { case t: Throwable = None } } val it: Iterator[String] = fromFile(foo.txt).getLines val foo: Option[Int] = enum(it)(0)(_ + _.toInt) 10/29
• 11. History Iterator pattern Iteratees Discussion only to realize we’ve already had it all!? val it: Iterator[String] = fromFile(foo.txt).getLines val foo: Try[Int] = it.foldLeft(Try(0)) { // f: (Out, In) = Out case (acc, line) = acc map { x = x + line.toInt } } 11/29
• 12. History Iterator pattern Iteratees Discussion only to realize we’ve already had it all!? val it: Iterator[String] = fromFile(foo.txt).getLines val foo: Try[Int] = it.foldLeft(Try(0)) { // f: (Out, In) = Out case (acc, line) = acc map { x = x + line.toInt } } Really? What about producer and consumer talking back and forth cannot cope with inﬁnite streams resource handling (stick this into another function?) asynchronism (could have used Futures) data which are not available all at once 11/29
• 13. History Iterator pattern Iteratees Discussion only to realize we’ve already had it all!? val it: Iterator[String] = fromFile(foo.txt).getLines val foo: Try[Int] = it.foldLeft(Try(0)) { // f: (Out, In) = Out case (acc, line) = acc map { x = x + line.toInt } } Think ofWhat about Really? producer and consumer talking back and forth cannot cope with inﬁnite streams foldLeft as Enumerator resource handling (stickIteratee another function?) acc together with f as this into asynchronism (could have used Futures) data which are not available all at once 11/29
• 14. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees Enumerators . . . are stream producers push subsequent chunks of data to an Iteratee, hence folding the Iteratee over the input stream sealed trait Enumerator[F] { def apply[T](iter: Iteratee[F,T]): Iteratee[F,T] def run[T](iter: Iteratee[F,T]): T } act synchronously or asynchronously (think Futures) come with batteries included object Enumerator { def apply[T](xs: T*): Enumerator[T] def fromTextFile(fileName: String): Enumerator[String] def fromStream(s: InputStream, chunkSize: Int = 8192) : Enumerator[Array[Byte]] } 12/29
• 15. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees Enumerators communicate state to the Iteratee: sealed trait Input[+T] case class Element[T](x: T) extends Input[T] case object Empty extends Input[Nothing] case object EOF extends Input[Nothing] 13/29
• 16. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees Enumerators communicate state to the Iteratee: sealed trait Input[+T] case class Element[T](x: T) extends Input[T] case object Empty extends Input[Nothing] case object EOF extends Input[Nothing] For now, assume enumerators like this: def enum[F,T](xs: List[F])(it: Iteratee[F,T]): Iteratee[F,T] 13/29
• 17. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees Iteratees . . . stream consumers consume chunks of input (as a whole) are immutable, re-usable computations state is encoded through type sealed trait Iteratee[F,T] { def run: T } case class Done[F,T] (result: T, remainingInput: Input[F]) case class Cont[F,T] (k: Input[F] = Iteratee[F,T]) case class Error[F,T](t: Throwable) // All of Done, Cont and Error extend Iteratee[F,T] 14/29
• 18. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees So far: Enumerators fold iteratees over a stream Enumerators communicate state through Input[T] Iteratees encapsulate result, error or computation 15/29
• 19. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees So far: Enumerators fold iteratees over a stream Enumerators communicate state through Input[T] Iteratees encapsulate result, error or computation Let’s revisit our “enumerator”: def enum[F,T](xs: List[F])(it: Iteratee[F,T]): Iteratee[F,T] = (xs, it) match { case (h::t, Cont(k)) = enum(t)( k(Element(h)) ) case _ = it } // k: Input[In] = Iteratee[F,T] This is all we need to have some working examples. 15/29
• 20. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees def head[T]: Iteratee[T,T] = { def step: Input[T] = Iteratee[T,T] = { case Element(x) = Done(x, Empty) case Empty = Cont(step) case EOF = Error(new NoSuchElementException) } Cont(step) } Example: scala enum(Hello world!.toList)(head) res1: Iteratee[Char,Char] = Done(H,Empty) scala res1.run res2: Char = H 16/29
• 21. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees def length[T]: Iteratee[T, Int] = { def step(n: Int): Input[T] = Iteratee[T, Int] = { case EOF = Done(n, EOF) case Empty = Cont(step(n)) case Element(_) = Cont(step(n+1)) } Cont(step(0)) } Example: scala enum(Hello world!.toList)(length) res3: Iteratee[Char,Int] = Cont(function1) scala res3.run res4: Int = 12 17/29
• 22. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees How this works length n=0 n=1 Cont(step(n)) n=2 Cont(step(n)) El(7) Cont(step(n)) El(9) n=2 enum(List(7,9)) = enum(List(9)) = enum(Nil) = Cont(step(n)) 18/29
• 23. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees How this works Further use that iteratee or apply run to it... n=2 Cont(step(n)) EOF run = Done(2,EOF) 18/29
• 24. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees There’s a lot of pattern matching going on. Frameworks to the rescue: object Iteratee { def apply[F,T](el: Element[F] = Iteratee[F,T], empty: = Iteratee[F,T], eof: = Iteratee[F,T]): Iteratee[F,T] def fold[F,T](z: T)(f: (T,F) = T): Iteratee[F,T] } Because that’s what we need most of the time. 19/29
• 25. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees Wouldn’t it be awesome if Iteratees were composable? def drop1Keep1[T] = for { _ - drop[T](1) h - head } yield h def pair[T] = for { h1 - head[T] h2 - head } yield (h1, h2) They are indeed! Iteratees compose sequentially (and so do Enumerators). scala enum(Hello World!.toList)(drop1Keep1).run res1: Char = e scala enum(Hello World!.toList)(pair).run res2: (Char, Char) = (H,e) 20/29
• 26. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees As we know, for translates to map and flatMap: trait Iteratee[F,T] { def map[U](f: T = U): Iteratee[F,U] def flatMap[U](f: T = Iteratee[F,U]): Iteratee[F,U] } 21/29
• 27. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees As we know, for translates to map and flatMap: trait Iteratee[F,T] { def map[U](f: T = U): Iteratee[F,U] def flatMap[U](f: T = Iteratee[F,U]): Iteratee[F,U] } Adding this to Cont is straight-forward. def map[U](f: T = U): Iteratee[F,U] = Cont(elt = k(elt) map f) def flatMap[U](f: T = Iteratee[F,U]): Iteratee[F,U] = Cont(elt = k(elt) flatMap f) // k: Input[F] = Iteratee[F,T] 22/29
• 28. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees So is adding to Done. def map[U](f: T = U): Iteratee[F,U] = Done(f(x), remainingInput) def flatMap[U](f: T = Iteratee[F,U]): Iteratee[F,U] = f(x) match { case Done(xPrime, _) = Done(xPrime, remainingInput) case Cont(k) = k(remainingInput) case err @ Error(_) = err } 23/29
• 29. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees Now why is that so great? Easily compose simple (or complex) iteratees def pair[T] = for { def drop1Keep1[T] = for { h1 - head[T] _ - drop[T](1) h - head h2 - head } yield h } yield (h1, h2) No repetitive pattern-matching hell Beneﬁt from utility functions, e.g. sequence or repeat Example: scala def fivePairs[T] = sequence(List.fill(5)(pair[T])) ... scala enum((1 to 10).toList)(fivePairs).run res1: List[(Int, Int)] = List((1,2), (3,4), (5,6), (7,8), (9,10) 24/29
• 30. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees Now why is that so great? Easily compose simple (or complex) iteratees def pair[T] = for { def drop1Keep1[T] = for { h1 - head[T] _ - drop[T](1) h - head h2 - head } yield h } yield (h1, h2) No repetitive pattern-matching hell Beneﬁt from utility functions, e.g. sequence or repeat Example: scala def alternates[T] = repeat(drop1Keep1[T]) ... scala enum((1 to 10).toList)(alternates).run res2: Stream[Int] = Stream(2, ?) // res2.toList == List(2, 4, 6, 8, 10) 24/29
• 31. History Iterator pattern Iteratees Discussion Enumerators Basics Composition Enumeratees Enumeratees... are both consumer and producer link in between Enumerator and Iteratee feed transformed input from Enumerator to “inner” Iteratee, for instance filter take ... are (of course?) composable allow vertical (parallel) stacking of Iteratees (one to many) See eamelink’s play-related examples @ https://gist.github.com/eamelink/5639642 25/29
• 32. History Iterator pattern Iteratees Discussion Brief discussion: Enumerators, Iteratees, Enumeratees Easy to use Composable in sequence and in parallel Tail recursion ⇒ no stack overﬂow Thin layer ⇒ high performance Good match for asynchronuous environments, e.g. the play framework Pure form lacks functional treatment of side eﬀects 26/29
• 33. History Iterator pattern Iteratees Discussion Where to go from here Read blogs tutorials, start out with [1] to [6] Use or rather implement iteratees Check out the play framework Asynchronous iteratees Numerous factory methods, e.g. easily bridge between Rx’s Observables and Enumerators [3] All the other goodies from play Check out scalaz 7 Implementation in terms of monad transformers ⇒ your choice of IO, Future, Option, . . . All what’s missing in the standard library See examples and explanations at [1] and [2] 27/29
• 34. History Iterator pattern Iteratees Discussion Thank you for your attention Anyone trying to understand monads will inevitably run into the [...] monad, and the results are almost always the same: bewilderment, confusion, anger, and ultimately Perl. Daniel Spiewak, Dec. 2010 28/29
• 35. History Iterator pattern Iteratees Discussion References: [1] “Enumeration-based I/O With Iteratees” @ http://blog.higher-order.com/blog/2010/10/14/scalaz-tutorial-enumeration-based-io-with-iteratees/ [2] “learning Scalaz: Enumeration-Based I/O with Iteratees” @ http://eed3si9n.com/learning-scalaz/Iteratees.html [3] “RxPlay: Diving into iteratees and observables and making them place nice” @ http://bryangilbert.com/code/2013/10/22/rxPlay-making-iteratees-and-observables-play-nice/ [4] “Understanding Play2 Iteratees for Normal Humans” @ http://mandubian.com/2012/08/27/understanding-play2-iteratees-for-normal-humans/ [5] “Incremental multi-level input processing and collection enumeration” @ http://okmij.org/ftp/Streams.html [6] “Teaching an Old Fool New Tricks” @ http://themonadreader.wordpress.com/2010/05/12/issue-16/ 29/29
• 36. Running Iteratees Example Backup slides 30/29
• 37. Running Iteratees Example run makes great eﬀort to yield the ﬁnal result: def run: T = this match { case Done(rslt, _) = rslt case Error(t) = throw t // k: Input[F] = Iteratee[F,T] case Cont(k) = k(EOF).run // may loop forever } 31/29
• 38. Running Iteratees Example run makes great eﬀort to yield the ﬁnal result: def run: T = this match { case Done(rslt, _) = rslt case Error(t) = throw t // k: Input[F] = Iteratee[F,T] case Cont(k) = k(EOF) match { case Done(rslt, _) = rslt case Cont(_) = sys.error(Diverging iteratee!) case Error(t) = throw t } } 31/29
• 39. Running Iteratees Example def drop[T](n: Int): Iteratee[T, Unit] = { def step: Input[T] = Iteratee[T, Unit] = { case EOF = Done((), EOF) case Empty = Cont(step) case Element(_) = drop(n-1) } if (n = 0) Done((), Empty) else Cont(step) } Example: scala enum(Hello World!.toList)(drop(3)) res5: Iteratee[Char,Unit] = Done((),Empty) scala res5.run // blank 32/29
• 40. Running Iteratees Example How this works drop(2) n=2 n=1 Cont(step(n)) Cont(step(n)) El(1) enum(List(1,2,3)) Done((),Empty) El(2) = enum(List(2,3)) = enum(List(3)) = Done((),Empty) 33/29