As professional software engineers, sometimes messy details of the real world stand in the way of us delivering principled software. Flaky connections, unreliable services, and bulletproof job scheduling in the presence of non-determinism and failure all tricky problems that discourage us from writing principled software. Yet sometimes the shortcuts we take to solve these problems result in downtime for the business and sleepless nights for us.
In this brand-new presentation, created exclusively for Scala in the City, John A. De Goes will show how functional programming can help bring order to even the most chaotic systems. Using ZIO, a new zero-dependency Scala library for building massively scalable asynchronous and concurrent applications, John will demonstrate how functional programming leverages reified effects and algebras to solve the trickiest of reliability and scheduling problems in a principled, composable, flexible way.
Join John for an evening of fun and functional programming as you explore fresh ways of thinking about reliability and scheduling, and come out of the talk with valuable skills for using ZIO to solve the everyday problems you encounter at work.
ZIO Schedule: Conquering Flakiness & Recurrence with Pure Functional Programming
1. ZIO Schedule John A. De Goes — @jdegoes - http://degoes.net Scala in the City
ZIO Schedule
Scala in the City, October 2018 - London, UK
John A. De Goes
@jdegoes - http://degoes.net
2. What is ZIO? Retries Repeats Schedule Composition
3. What is ZIO? Retries Repeats Schedule Composition
ZIO
ZIO lets you build high-performance,
type-safe, concurrent, asynchronous
applications that don’t leak resources,
and are easy to reason about
compositionally, test, and refactor.
5. def main: Unit = {
println("Hello, what is your name?")
val name = readLine()
println(s"Good morning, $name!")
}
Second-Class Effects
✗ Pass to functions
✗ Return from functions
✗ Store in data structures
✗ Async/sync/concurrency
✗ Async/sync resource-safety
What is ZIO? Retries Repeats Schedule Composition
6. def main: IO[IOException, Unit] = for {
_ <- putStrLn("Hello, what is your name?")
name <- getStrLn
_ <- putStrLn(s"Good morning, $name!")
} yield ()
First-Class Effects
✓ Pass to functions
✓ Return from functions
✓ Store in data structures
✓ Async/sync/concurrency
✓ Async/sync resource-safety
What is ZIO? Retries Repeats Schedule Composition
7. IO[E, A]
IO[E, A] is an immutable value that
describes an effectful program, which
may fail with a value of type E, or return
a value of type A.
What is ZIO? Retries Repeats Schedule Composition
ZIO IO
8. What is ZIO? Retries Repeats Schedule Composition
IO[Nothing, A]
Never fails
ZIO IO
9. What is ZIO? Retries Repeats Schedule Composition
IO[E, Nothing]
Never returns
ZIO IO
10. What is ZIO? Retries Repeats Schedule Composition
IO[Nothing, Nothing]
Never fails or returns
ZIO IO
18. What is ZIO? Retries Repeats Schedule Composition
App Web API
19. What is ZIO? Retries Repeats Schedule Composition
App Web API
408
20. What is ZIO? Retries Repeats Schedule Composition
App Web API
408429
21. What is ZIO? Retries Repeats Schedule Composition
App Web API
408429
423
22. What is ZIO? Retries Repeats Schedule Composition
App Web API
408429
423
500
23. What is ZIO? Retries Repeats Schedule Composition
App Web API
408429
423
500 503
24. What is ZIO? Retries Repeats Schedule Composition
App Web API
408429
423
500 503
504
25. What is ZIO? Retries Repeats Schedule Composition
App Web API
408429
423
500 503
504
26. What is ZIO? Retries Repeats Schedule Composition
AppDatabase
Cache
Local
Storage
Cloud
Storage
Streaming Logs
Analytics
Web APIs
27. def networkRequest(): A = ???
def networkRequestWithRetries(max: Int, millis: Long): A = {
var i = 0
while (i < max) {
try {
return networkRequest()
}
catch {
case _ : Exception =>
i = i + 1
Thread.sleep(millis)
}
}
}
What is ZIO? Retries Repeats Schedule Composition
Retrying Synchronously
29. What is ZIO? Retries Repeats Schedule Composition
App
Generate
Report
30. What is ZIO? Retries Repeats Schedule Composition
App
Generate
Report
Every
user
31. What is ZIO? Retries Repeats Schedule Composition
App
Generate
Report
Every
user
Every
day
32. What is ZIO? Retries Repeats Schedule Composition
App
Generate
Report
Every
user
Every
day
Every
account
33. What is ZIO? Retries Repeats Schedule Composition
App
Generate
Report
Every
user
Every
day
Every
account
Every
week
34. What is ZIO? Retries Repeats Schedule Composition
AppService
Heartbeat
Data
Retrieval
Remote
Uploading
Report
Generation
Log Rotation File Cleanup
User
Reminders
Garbage
Collection
35. What is ZIO? Retries Repeats Schedule Composition
def reportGen(): A = ???
def repeatedReportGen(max: Int, millis: Long): List[A] = {
var list = List.empty[A]
var i = 0
while (i < max) {
list = reportGen() :: list
Thread.sleep(millis)
i = i + 1
}
list
}
Repeating Synchronously
37. What is ZIO? Retries Repeats Schedule Composition
ZIO SCHEDULE
Schedule[A, B]
Schedule[A, B] is an immutable
value that describes an effectful
schedule, which after consuming an A,
produces a B and decides to halt or
continue after some delay d.
38. What is ZIO? Retries Repeats Schedule Composition
ZIO SCHEDULE
Schedule[A, B]
A
B
B
Continue?
Continue after delay d
Halt immediately
39. What is ZIO? Retries Repeats Schedule Composition
Should I repeat a
program that failed
with an E?
Should I repeat a
program that
returned an A?
Retries Repeats
40. What is ZIO? Retries Repeats Schedule Composition
Retries Repeats
Schedule[E, B] Schedule[A, B]
Retry policy
Consumes errors of type E
Emits values of type B
Repeat policy
Consumes return values of type
A
Emits values of type B
41. What is ZIO? Retries Repeats Schedule Composition
Schedule.never : Schedule[Any, Nothing]
Zero Recurrences
Accepts any input
Never recurs, never emitting a value
42. What is ZIO? Retries Repeats Schedule Composition
Schedule.once : Schedule[Any, Unit]
One Recurrence
Accepts any input
Recurs once, emitting unit
43. What is ZIO? Retries Repeats Schedule Composition
Schedule.forever : Schedule[Any, Int]
Infinite Recurrences
Consumes any input
Recurs forever without delay, emitting the number of recurrences so far
44. What is ZIO? Retries Repeats Schedule Composition
Schedule.recurs(n: Int) : Schedule[Any, Int]
Fixed Recurrences
Accepts any input
Recurs the specified number of times, emitting number of recurrences so far
45. durationtask task
What is ZIO? Retries Repeats Schedule Composition
Schedule.spaced(d: Duration) : Schedule[Any, Int]
Spaced Recurrences
Accepts any input
Recurs forever with delay, emitting the number of recurrences so far
46. duration
task task
What is ZIO? Retries Repeats Schedule Composition
Schedule.fixed(d: Duration) : Schedule[Any, Int]
Fixed Recurrences
Accepts any input
Recurs forever with a delay, emitting number of recurrences so far
duration
47. What is ZIO? Retries Repeats Schedule Composition
Schedule.exponential(d: Duration, f: Double = 2.0) : Schedule[Any, Duration]
Exponential Recurrences
Accepts any input
Recurs forever with delay, emitting the current delay between steps
Scaling factor
Starting duration
48. What is ZIO? Retries Repeats Schedule Composition
Schedule.doWhile[A](f: A => Boolean): Schedule[A, A]
Conditional Recurrences
Accepts an A
Recurs while the predicate is true without delay, emitting the same A
Recurrence condition
49. What is ZIO? Retries Repeats Schedule Composition
Schedule.doUntil[A](f: A => Boolean): Schedule[A, A]
Conditional Recurrences
Accepts an A
Recurs until the predicate is true without delay, emitting the same A
Recurrence condition
50. What is ZIO? Retries Repeats Schedule Composition
Schedule.collect: Schedule[A, List[A]]
Collecting Recurrences
Recurs forever without delay, emitting a list of all consumed A’s
Consumes an A
51. What is ZIO? Retries Repeats Schedule Composition
Schedule.identity[A]: Schedule[A, A]
Identity Recurrences
Recurs forever without delay, emitting the same A
Consumes an A
52. What is ZIO? Retries Repeats Schedule Composition
Schedule.unfold[A](a: => A)(f: A => A): Schedule[Any, A]
Unfold Recurrences
Recurs forever without delay, emitting an A by unfolding the seed through repeated application
Consumes any value
53. What is ZIO? Retries Repeats Schedule Composition
Schedule.point[A](a: => A): Schedule[Any, A]
Constant Output
Recurs forever without delay, emitting a constant
Consumes any value
54. What is ZIO? Retries Repeats Schedule Composition
Schedule.lift[A, B](f: A => B): Schedule[A, B]
Function Output
Recurs forever without delay, emitting the function applied to the input
Consumes an A
55. What is ZIO? Retries Repeats Schedule Composition
Retries Repeats
val action: IO[E, A] = ???
val policy: Schedule[E, B] = ???
val retried: IO[E, A] =
action retry policy
val action: IO[E, A] = ???
val policy: Schedule[A, B] = ???
val repeated: IO[E, B] =
action repeat policy
56. What is ZIO? Retries Repeats Schedule Composition
Retries Repeats
val action: IO[E, A] = ???
val policy: Schedule[E, B] = ???
val orElse: (E, B) => IO[E2, A] = ???
val retried: IO[E2, A] =
action retryOrElse (policy, orElse)
val action: IO[E, A] = ???
val policy: Schedule[A, B] = ???
val orElse: (E, Option[B]) => IO[E2, B] = ???
val repeated: IO[E2, B] =
action repeatOrElse (policy, onError)
57. What is ZIO? Retries Repeats Schedule Composition
Retries Repeats
val action: IO[E, A] = ???
val policy: Schedule[E, B] = ???
val orElse: (E, B) => IO[E2, B] = ???
val retried: IO[E2, Either[B, A]] =
action retryOrElse0 (policy, orElse)
val action: IO[E, A] = ???
val policy: Schedule[A, B] = ???
val orElse: (E, Option[B]) => IO[E2, C] = ???
val repeated: IO[E2, Either[C, B]] =
action repeatOrElse0 (policy, onError)
61. What is ZIO? Retries Repeats Schedule Composition
Monix Retry & Repeat
Retries Repeats
val task : Task[A] = ???
val retries: Int = ???
val retried1: Task[A] =
task onErrorRestart (retries)
val pred: Throwable => Boolean = ???
val retried2: Task[A] =
task onErrorRestartIf (pred)
// onErrorRestartLoop
val task: Task[A] = ???
val pred: A => Boolean = ???
val repeated: Task[A] =
task restartUntil (pred)
62. What is ZIO? Retries Repeats Schedule Composition
ZIO SCHEDULE
Monoid
append
zero
63. What is ZIO? Retries Repeats Schedule Composition
ZIO SCHEDULE
Monoid
append
zero
Applicative
map
point
ap
64. What is ZIO? Retries Repeats Schedule Composition
ZIO SCHEDULE
Monoid
append
zero
Applicative
map
point
ap
Category
id
compose
65. What is ZIO? Retries Repeats Schedule Composition
ZIO SCHEDULE
Monoid
append
zero
Applicative
map
point
ap
Category
id
compose
Profunctor
lmap
rmap
dimap
66. What is ZIO? Retries Repeats Schedule Composition
ZIO SCHEDULE
Monoid
append
zero
Applicative
map
point
ap
Category
id
compose
Profunctor
lmap
rmap
dimap
Strong
first
second
67. What is ZIO? Retries Repeats Schedule Composition
ZIO SCHEDULE
Monoid
append
zero
Applicative
map
point
ap
Category
id
compose
Profunctor
lmap
rmap
dimap
Strong
first
second
Choice
left
right
68. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]) && (s2 : Schedule[A, C]) : Schedule[A, (B, C)]
Intersection
s1 s2 s1 && s2
Continue : Boolean b1 b2 b1 && b2
Delay : Duration d1 d2 d1.max(d2)
Emit : (A, B) a b (a, b)
69. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]) || (s2 : Schedule[A, C]) : Schedule[A, (B, C)]
Union
s1 s2 s1 || s2
Continue : Boolean b1 b2 b1 || b2
Delay : Duration d1 d2 d1.min(d2)
Emit : (A, B) a b (a, b)
70. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]) andThen (s2 : Schedule[A, C]) : Schedule[A, Either[B, C]]
Sequence
A
A
B
C
s1
s2
s1 andThen s2A Either[B, C]
71. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]) >>> (s2 : Schedule[B, C]) : Schedule[A, C]
Compose
A
B
B
C
s1
s2
s1 >>> s2A C
72. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]).jittered : Schedule[A, B]
Jittering
A AB Bs1 s1’
Delays randomly
jittered
73. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]).collect : Schedule[A, List[B]]
Collecting
A AB List[C]s1 s1’
Emissions collected
74. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]).whileInput(f: A => Boolean) : Schedule[A, B]
Filtering By Input
A AB Bs1 s1’
Continues while/until f returns true
(s1 : Schedule[A, B]).untilInput(f: A => Boolean) : Schedule[A, B]
75. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]).whileOutput(f: B => Boolean) : Schedule[A, B]
Filtering By Output
A AB Bs1 s1’
Continues while/until f returns true
(s1 : Schedule[A, B]).untilOutput(f: B => Boolean) : Schedule[A, B]
76. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]).map(f: B => C) : Schedule[A, C]
Mapping
A AB Cs1 s1’
B mapped to C
77. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]) *> (s2 : Schedule[A, C]) : Schedule[A, C]
(s1 && s2).map(_._2)
Left / Right Ap
(s1 : Schedule[A, B]) <* (s2 : Schedule[A, C]) : Schedule[A, B]
(s1 && s2).map(_._1)
78. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]).logInput(f: A => IO[Nothing, Unit]) : Schedule[A, B]
Logging Inputs
A AB Bs1 s1’
Inputs logged to f
79. What is ZIO? Retries Repeats Schedule Composition
(s1 : Schedule[A, B]).logOutput(f: B => IO[Nothing, Unit]) : Schedule[A, B]
Logging Outputs
A AB Bs1 s1’
Outputs logged to f
80. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
81. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
82. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
Schedule.exponential(10.milliseconds)
83. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
(Schedule.exponential(10.milliseconds)
andThen ???)
84. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
(Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds)
andThen ???)
85. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
(Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds)
andThen
Schedule.fixed(60.seconds))
86. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
(Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds)
andThen
(Schedule.fixed(60.seconds) && Schedule.recurs(100))
87. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
(Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds)
andThen
(Schedule.fixed(60.seconds) && Schedule.recurs(100)).jittered
88. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
(Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds)
andThen
(Schedule.fixed(60.seconds) && Schedule.recurs(100)).jittered
*> Schedule.identity[A].collect
89. What is ZIO? Retries Repeats Schedule Composition
Crafting a ZIO Schedule
Produce a jittered schedule that first does exponential spacing (starting from 10
milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed
spacing of 60 seconds between recurrences, but will only do that for up to 100
times, and emits a list of the collected inputs.
def customSchedule[A]: Schedule[A, List[A]] = {
import Schedule._
(exponential(10.millis).whileOutput(_ < 60.secs) andThen
(fixed(60.secs) && recurs(100)).jittered *> identity[A].collect
}
91. Functional Scala by John A. De Goes
Local Remote Date
London London Time Oct 21 - 23
Paris Paris Time Oct 25 - 27
SF / SV Pacific Coast Time Nov 18 - 21
That’s A Wrap!