Streams are often underestimated and skipped as possible solutions. In many cases, we created much more complex solutions than their streams counterpart. Why? It's hard to answer, but in this presentation, I would like to tell you a story about how we started to use FS2, without sacrificing purity and code readability. (https://github.com/functional-streams-for-scala/fs2)

1. Stream or not to Stream?
Fs2 is the answer

2. Few words about me…
•Senior Software Engineer at
•Scala developer since 2012
• @LukaszByczynski

3. Motivation

4. Pump in ML data to HDFS
• Instead of keeping files on S3 (compressed JSONs)
we would like to have them on HDFS stored with
optimized format like Parquet
• We need to support full and incremental dumps from
our data ocean (Vertica)
• Simplify the current solution and provide
“predictable” memory usage.

5. Parquet File

6. Parquet File

7. Parquet file on HDFS

8. Why to Stream?
• compositionality
• resource safety
• expressiveness
• “infinite” input sources
• speed / back pressure
• async boundaries

9. What to choose?

10. + stable
+ enterprise ready
+ documentation
+ python ready
- impure
- heavyweight
- lack of control
- YAGNI
Spark

11. + stable
+ enterprise ready
+ documentation
- impure
- complex API
- slow?
- actors?
Akka

12. + nice rx-like API
+ fast
+ stable
+/- pure
- bus factor == 1
- documentation
Monix

13. + super fast*
+ pure
- bus factor == 2
- mindshift
- documentation
- early preview
- sacrifice
half of community
ZIO
Stream

14. + fast
+ pure
+ stable
+/- documentation
+/- mindshift
- not enterprise
Fs2

15. Fs2 API

18. Chunk[+O]
Strict, finite sequence of values that allows index-
based random access of elements.

19. Stream[F, +O]
A stream producing output of type O and which may
evaluate F effects. If F is Pure, the stream evaluates
no effects

20. Pull[F[_],+O, +R]
Reads values from one or more streams, returns a
result of type R, and produces a Stream[F,O] when
calling stream method.

21. Pipe[F, I, O]
Stream[F, I] b> Stream[F, O]

22. Sink[F, I]
Pipe[F, I, Unit]

23. Who uses it?
doobie http4s

24. Show me the code…

25. compositionality
val composed = Stream.range(1, 4) ++ Stream("a", "b", "c")
println(composed.compile.toList)
val zipped = Stream
.range(1, 4)
.zip(Stream("a", "b", "c"))
.map { case (l, r) b> s"$l$r" }
println(zipped.compile.toList)

26. compositionality
val composed = Stream.range(1, 4) ++ Stream("a", "b", "c")
println(composed.compile.toList)
val zipped = Stream
.range(1, 4)
.zip(Stream("a", "b", "c"))
.map { case (l, r) b> s"$l$r" }
println(zipped.compile.toList)

27. compositionality
val composed = Stream.range(1, 4) ++ Stream("a", "b", "c")
println(composed.compile.toList)
val zipped = Stream
.range(1, 4)
.zip(Stream("a", "b", "c"))
.map { case (l, r) b> s"$l$r" }
println(zipped.compile.toList)

28. resource safety
Stream
.bracket(IO(Files.createTempFile("aaaa", "bbbb")))(f b> IO(f.toFile.delete()))
.evalMap(f b> IO(println(f)))
.compile
.drain
.as(ExitCode.Success)

29. resource safety
Stream
.bracket(IO(Files.createTempFile("aaaa", "bbbb")))(f b> IO(f.toFile.delete()))
.evalMap(f b> IO(println(f)))
.compile
.drain
.as(ExitCode.Success)

30. resource safety
Stream
.bracket(IO(Files.createTempFile("aaaa", "bbbb")))(f b> IO(f.toFile.delete()))
.evalMap(f b> IO(println(f)))
.compile
.drain
.as(ExitCode.Success)

31. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

32. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

33. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

34. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

35. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

36. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

37. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

38. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

39. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

40. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

41. expressiveness
private val blockingEc = Resource
.make(
IO(ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2)))
)(ec b> IO(ec.shutdown()))
val converter: Stream[IO, Unit] = Stream.resource(blockingExecutionContext).flatMap { blockingEC b>
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0 / 9.0)
io.file
.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blockingEC, 4096)
.through(text.utf8Decode)
.through(text.lines)
.filter(s b> !s.trim.isEmpty && !s.startsWith("//"))
.map(line b> fahrenheitToCelsius(line.toDouble).toString)
.intersperse("n")
.through(text.utf8Encode)
.through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blockingEC))
}
def run(args: List[String]): IO[ExitCode] =
converter.compile.drain.as(ExitCode.Success)

42. “infinite” input sources
val tick = System.currentTimeMillis()
Stream
.repeatEval(IO(System.currentTimeMillis()))
.take(100)
.fold(0l)((_, r) b> r)
.compile
.lastOrError
.flatMap { result b>
IO(println(result - tick))
}
.as(ExitCode.Success)

43. “infinite” input sources
val tick = System.currentTimeMillis()
Stream
.repeatEval(IO(System.currentTimeMillis()))
.take(100)
.fold(0l)((_, r) b> r)
.compile
.lastOrError
.flatMap { result b>
IO(println(result - tick))
}
.as(ExitCode.Success)

44. “infinite” input sources
val tick = System.currentTimeMillis()
Stream
.repeatEval(IO(System.currentTimeMillis()))
.take(100)
.fold(0l)((_, r) b> r)
.compile
.lastOrError
.flatMap { result b>
IO(println(result - tick))
}
.as(ExitCode.Success)

45. “infinite” input sources
val tick = System.currentTimeMillis()
Stream
.repeatEval(IO(System.currentTimeMillis()))
.take(100)
.fold(0l)((_, r) b> r)
.compile
.lastOrError
.flatMap { result b>
IO(println(result - tick))
}
.as(ExitCode.Success)

46. async boundaries
val threadPool = Executors.newSingleThreadExecutor
val singleCs = IO.contextShift(ExecutionContext.fromExecutor(threadPool))
Stream
.range(1, 10)
.covary[IO]
.parEvalMap(4)(i b> printWithThreadName(s"Value $i") >> IO.pure(i))
.fold(0)(_ + _)
.compile
.lastOrError
.flatMap(result b> IO.shift(singleCs) >> printWithThreadName(s"Result: $result"))
.flatMap(_ b> IO(threadPool.shutdown()))
.as(ExitCode.Success)

47. async boundaries
val threadPool = Executors.newSingleThreadExecutor
val singleCs = IO.contextShift(ExecutionContext.fromExecutor(threadPool))
Stream
.range(1, 10)
.covary[IO]
.parEvalMap(4)(i b> printWithThreadName(s"Value $i") >> IO.pure(i))
.fold(0)(_ + _)
.compile
.lastOrError
.flatMap(result b> IO.shift(singleCs) >> printWithThreadName(s"Result: $result"))
.flatMap(_ b> IO(threadPool.shutdown()))
.as(ExitCode.Success)

48. async boundaries
val threadPool = Executors.newSingleThreadExecutor
val singleCs = IO.contextShift(ExecutionContext.fromExecutor(threadPool))
Stream
.range(1, 10)
.covary[IO]
.parEvalMap(4)(i b> printWithThreadName(s"Value $i") >> IO.pure(i))
.fold(0)(_ + _)
.compile
.lastOrError
.flatMap(result b> IO.shift(singleCs) >> printWithThreadName(s"Result: $result"))
.flatMap(_ b> IO(threadPool.shutdown()))
.as(ExitCode.Success)

49. async boundaries
val threadPool = Executors.newSingleThreadExecutor
val singleCs = IO.contextShift(ExecutionContext.fromExecutor(threadPool))
Stream
.range(1, 10)
.covary[IO]
.parEvalMap(4)(i b> printWithThreadName(s"Value $i") >> IO.pure(i))
.fold(0)(_ + _)
.compile
.lastOrError
.flatMap(result b> IO.shift(singleCs) t> printWithThreadName(s"Result: $result"))
.flatMap(_ b> IO(threadPool.shutdown()))
.as(ExitCode.Success)

50. async boundaries
val threadPool = Executors.newSingleThreadExecutor
val singleCs = IO.contextShift(ExecutionContext.fromExecutor(threadPool))
Stream
.range(1, 10)
.covary[IO]
.parEvalMap(4)(i b> printWithThreadName(s"Value $i") >> IO.pure(i))
.fold(0)(_ + _)
.compile
.lastOrError
.flatMap(result b> IO.shift(singleCs) t> printWithThreadName(s"Result: $result"))
.flatMap(_ b> IO(threadPool.shutdown()))
.as(ExitCode.Success)

51. Demo

52. Summary

53. Best practices
Try to keep all streaming code
very close to interpreter layer

54. Best practices
There is one exception, if your
application based on Streaming solution
(ex. Http4s) you can return Stream as
the result of operation
(there are benefits !)

55. Best practices
There is one exception, if your
application based on Streaming solution
(ex. Http4s) you can return Stream as
the result of operation
(there are benefits !)

56. Best practices
Try to avoid complex pipelines
as long as possible

57. Best practices
If you need to do that,
consider using other solutions like Akka
(they integrate well)
https://github.com/krasserm/streamz

58. Best practices
Comprehension and simplicity
is the key to readability

59. Q&A