Traversals and Scans and CQRS, oh my!

About me
• Freelance software developer
• Doing functional programming, microservices, DevOps
• Co-organize the Underscore Scala meetup
• Hit me up if you want to work together!
Itamar Ravid - @itrvd 2

Agenda
• Why Event Sourcing and CQRS are worth your time
• How we can use functions to implement them
• How abstractions from FP help us
• (shortly) Hooking this up to a stream

Event Sourcing
CQRS

That's not the only
downside!
• No determinism
• Lowly available
• High coupling, barely autonomous

What can we do to ﬁx this?

Focus on behaviors.
The objects will follow.

Responding to queries

Materialized views

CQRS: Command/Query Responsibility
Separation
We store the events in an append only stream.
We handle reads using a specialized store, derived from the
events.
Now you know CQRS!

What do we gain?
• Faster response time
• Availability
• Determinism, take your debugging home
•
!
Event-sourced services are like pure functions!

And speaking of pure functions
Let's see some code!
The examples ahead assume:
import cats._, cats.data._, cats.implicits._

Zoom in
We'll now focus on the reservation service.
Let's event-storm this!
What commands/events are we going to use?

Commands
We might be asked to:
• Create a reservation
• Add a person to a reservation
• Cancel a reservation

Events
And these will cause the:
• Reservation to be created
• Reservation to be updated
• Reservation to be canceled

Command Deﬁnitions
I promised some code, right?
sealed trait ReservationCommand
case class Create(roomId: String, guests: Int)
extends ReservationCommand
case class ModifyGuests(reservationId: String, amount: Int)
case class Cancel(reservationId: String)

Event Deﬁnitions
And we have the corresponding events:
sealed trait ReservationEvent
case class Created(id: String, roomId: String, guests: Int)
extends ReservationEvent
case class Updated(id: String, roomId: String, guests: Int)
case class Canceled(id: String, roomId: String, guests: Int)

Breakdown
So technically, we need:
1. Validation of commands
2. State representation, state mutation
3. Event generation
4. Persistence

Validation
So we want a function, right?
def processCommand(command: ReservationCommand): ???

Validation
def processCommand(command: ReservationCommand): Either[Error, Unit]

Validation
type Error = String
type Result[A] = Either[Error, A]
def processCommand(command: ReservationCommand): Result[Unit]

Validation
Let's start with simple validation: the number of guests should be
positive.
def validateGuests(command: Create): Result[Unit] =
if (command.guests <= 0) Left("Non-positive guests")
else Right(())

Stateful validation
Next, we want a stateful validation - no duplicate reservations.

Stateful validation
Next, we want a stateful validation - no duplicate reservations.
To do that, we need some sort of state:
case class Reservation(id: String)
trait Reservations {
def byRoomId(id: String): Option[Reservation]
}

Validation
And here's our validation:
def validateDup(reservations: Reservations,
command: Create): Result[Unit] =
reservations.byRoomId(command.roomId) match {
case Some(_) => Left("Duplicate reservation")
case None => Right(())
}

Validation - continued
Our command processing now looks like this:
def processCommand(reservations: Reservations,
command: ReservationCommand): Result[Unit] =
command match {
case create: Create =>
(validateGuests(create), validateDup(create, reservations))
.tupled.void
// case modifyGuests, case cancelReservation, etc.
}

What's that (a, b).tupled function?
This will combine the result of both validations into a single
value:
val guestValidation: Result[Unit] = validateGuests(create)
val dupValidation: Result[Unit] = validateDup(create, reservations)
val tupled: Result[(Unit, Unit)] = (guestValidation, dupValidation).tupled

What's that (a, b).tupled function?
This will combine the result of both validations into a single
value:
val guestValidation: Result[Unit] = validateGuests(create)
val dupValidation: Result[Unit] = validateDup(create, reservations)
val tupled: Result[(Unit, Unit)] = (guestValidation, dupValidation).tupled
If one of them fails - both fails too.

What else can we say about Result?

It's an Applicative - so we can use tupled.

It's also a Monad, so we can use for comprehensions:
for {
a <- validate1(cmd)
b <- validate2(cmd, a)
} yield f(a, b)

A few side notes:
• Our validation unfortunately short circuits. We don't see all
errors.
Check out cats.data.Validated for a solution.
• Applicative s are essential for joining independent
computations. Deﬁnitely read up on them!
Check out the mapN, *>, <* combinators.

Validation - summary
This is all I'm going to say about validation!
• Write an Either returning function
• Give it what it needs to compute
• Combine with the Applicative combinators
• Proﬁt Validated!

Generating events
The path of least resistance is another function for generating:
def events(reservations: Reservations,
command: ReservationCommand): List[ReservationEvent]

Generating events
We return the events inside Result:
command: ReservationCommand)
: Result[List[ReservationEvent]] =
command match {
case create: Create =>
(validateGuests(create), validateDup(create, reservations))
.tupled
.map(_ => events(reservations, commands))
}
Since we're using List, we can output 0, 1 or many events.

Generating events
This is great if we can separate validation and event generation.

Generating events
This isn't always possible!

Generating events
This isn't always possible!
We'll upgrade the two functions with event generation.

Generating events

Generating events
There's a cool data type called WriterT we can use here:
case class WriterT[F[_], Log, Value](run: F[(Log, Value)])

Generating events
We'll modify the validation return type to be:
WriterT[Result, List[ReservationEvent], A]
Which wraps values of type
Result[(List[ReservationEvent], A)]

Generating events
Now, when we use the .tupled syntax, we get back:
val writer: WriterT[Result, List[ReservationEvent], Unit] =
(validateDup(reservations, command),
validateGuests(command)).tupled.void
And it actually concatenated the lists for us!

Generating events
We can "peel" this using .run:
val writer: WriterT[Result, List[ReservationEvent], Unit] =
(validateDup(reservations, command),
validateGuests(command)).tupled.void
val result: Result[(List[ReservationEvent], Unit)] =
writer.run

Generating events
Or, if we only want the log:
val result: Result[List[ReservationEvent]] =
writer.written

Generating events
So let's give this new type a name:
type EventsAnd[A] = WriterT[Result, List[ReservationEvent], A]

Actually writing those functions
Here's an example of how one of those functions would look like:
def validateGuests(command: Create): EventsAnd[Unit] =
if (command.guests <= 0)
WriterT.liftF("Non-positive guests".asLeft[Unit])
else
WriterT.putT(().asRight[Error])(List(ReservationCreated(command.id)))

Generating events - summary
We're done! Our functions now validate and generate events:
def validateGuests(command: Create): EventsAnd[Unit]
command: Create): EventsAnd[Unit]
val result = (validateGuests(c), validateDup(s, c))
.tupled.void

Generating events - summary
We're done! Our functions now validate and generate events:
def validateGuests(command: Create): EventsAnd[Unit]
command: Create): EventsAnd[Unit]
val result = (validateGuests(c), validateDup(s, c))
.tupled.void
Of course, if validation fails - everything still fails.

A few side notes on WriterT
• WriterT is also a Monad and an Applicative.
• Lots of combinators - check out the docs!
• WriterT can be quite allocation-heavy.
We'll touch a possible solution towards the end.

State mutation
Our function doesn't do anything currently:
command: ReservationCommand): EventsAnd[Unit]

State mutation
It's pretty awesome that we can reason about it just by looking at
the type:
(Reservations, Command) => WriterT[Result, List[Event], Unit]
Nothing else happens. WYSIWYG.

State mutation
The path of least resistance is to return a new state:
command: ReservationCommand): (Reservations, EventsAnd[Unit])

State mutation
The path of least resistance is to return a new state:
command: ReservationCommand): (Reservations, EventsAnd[Unit])
We're not saying anything about whether it was modiﬁed.

State mutation
However, we want to guarantee that nothing happens if
validation fails.
command: ReservationCommand): EventsAnd[Reservations]
If validation is ﬁne, we get a new state.
If it fails, we keep the previous state.

State mutation
But - how do we reconcile the two resulting states?
def validateGuests(s: Reservations, c: Create): EventsAnd[Reservations]
def validateDup(s: Reservations, c: Create): EventsAnd[Reservations]
val result: EventsAnd[(Reservations, Reservations)] =
(validateGuests(s, c),
validateDup(s, c)).tupled

State mutation
Ideally, we'd just carry the state from function to function:

State mutation
Let's modify the functions slightly:

State mutation
Let's modify the functions slightly:
def validateGuests(c: Create): Reservations => EventsAnd[Reservations]

State mutation
Functional programming saves the day again- this is StateT:
case class StateT[F[_], State, Value](run: State => F[(State, Value)])

State mutation
We add yet another type alias on top:
type CommandProcessor[A] = StateT[EventsAnd, Reservations, A]
def validateGuests(c: Create): CommandProcessor[Unit]
def validateDup(c: Create): CommandProcessor[Unit]

State mutation
We can now compose our command processors:
def validateGuests(c: Create): CommandProcessor[Unit]
def validateDup(c: Create): CommandProcessor[Unit]
val resultProcessor: CommandProcessor[Unit] =
(validateGuests(c),
validateDup(c)).tupled.void
But where did our initial state disappear to?

State mutation
We haven't run anything yet! Let's run the processor:
val result: EventsAnd[(Reservations, Unit)] = resultProcessor.run(state)
This does a lot of stuff!

State mutation
Here's how we actually write one of these functions:
def validateGuests(command: Create): CommandProcessor[Unit] =
StateT.liftF(WriterT.liftF("Non-positive guests".asLeft[Unit]))
else
for {
state <- StateT.get[EventsAnd, Reservations]
events = generateEventsSomehow(state)
_ <- StateT.liftF(WriterT.tell[Result, List[ReservationEvent]](events))
newState = generateNewStateSomehow(reservations, event)
_ <- StateT.set[EventsAnd, Reservations](newState)
} yield ()

State mutation
Here's how we actually write one of
these functions:
def validateGuests(command: Create): CommandProcessor[Unit] =
StateT.liftF(WriterT.liftF("Non-positive guests".asLeft[Unit]))
else
for {
state <- StateT.get[EventsAnd, Reservations]
events = generateEventsSomehow(state)
_ <- StateT.liftF(WriterT.tell[Result, List[ReservationEvent]](events))
newState = generateNewStateSomehow(reservations, event)
_ <- StateT.set[EventsAnd, Reservations](newState)
} yield ()

Summary
• I bet you're loving the boilerplate by now ;-)
It gets worse as you add more layers!
• Don't worry though! This is solvable.
• As before, StateT is also a Monad and an Applicative.

Recap
We now have a pretty powerful type as a building block:

Recap
If we expand everything, we get:
type CommandProcessor[A] =
Reservations => Either[Error, (List[Event], (Reservations, A))]

Is that it?
The command processor can return values, signal failures, log
events, and accumulate state.

Is that it?
The command processor can return values, signal failures, log
events, and accumulate state.
No magic! These are just functions!

Fitting this into an application
We still have some work to do:
• we need to hook this up to a source of commands,
• and we actually need to persist the state and events.

Fitting this into an application
Where can commands come from?
Treating these as inﬁnite streams makes our life easier.

Streams
We'll use fs2 for our examples.
Akka Streams can deﬁnitely work too! Check out the slides
when I publish them.

fs2
fs2 is based on one type:
Stream[Effect[_], Element]
A possibly inﬁnite stream of Element, with effect Effect.
Effect can be cats IO, scalaz IO, Monix Task, etc.

fs2
fs2 is based on one type:
Stream[Effect[_], Element]
A possibly inﬁnite stream of Element, with effect Effect.

fs2
To run our command processor, we can use mapAccumulate:
class Stream[F[_], Element] {
def mapAccumulate[State, Out](init: State)
(f: (State, Element) => (State, Out)): Stream[F, (State, Out)]
}

fs2
To run our command processor, we can use mapAccumulate:
class Stream[F[_], Element] {
def mapAccumulate[State, Out](init: State)
(f: (State, Element) => (State, Out)): Stream[F, (State, Out)]
}
We'll set:
• Element = Command, State = Reservations
• Out = (Reservations, List[ReservationEvent])

fs2
def process(commands: Stream[IO, Command], initial: Reservations)
: Stream[IO, (Reservations, List[ReservationEvent])] =
commands.mapAccumulate(initial) { (state, command) =>
processCommand(command).run(state) match {
case Left(error) => (state, (state, Nil))
case Right(nextState, (events, _)) => (nextState, (nextState, events))
}
}

Persisting events and state
def persist(reservations: Reservations): IO[Unit]
def produce(events: List[ReservationEvent]): IO[Unit]
stream.evalMap {
case (state, events) =>
(persist(state), produce(events))
.tupled
.void
}

Where to go from here
• The solution to monad transformer overhead: ﬁnally tagless
• Keep the effect abstract, interpret into Task+IORef.
• Check out the links on next slide.
• You can also hire me and I'll help you make it work ;-)

Where to go from here
Resources:
• Ben Stopford, Designing Event-Driven Systems:
https://www.conﬂuent.io/designing-event-driven-systems
• Martin Kleppman's blog:
https://martin.kleppmann.com
• The FS2 guide:
https://functional-streams-for-scala.github.io/fs2/
• Finally tagless and Free:
https://softwaremill.com/free-tagless-compared-how-not-to-commit-to-monad-too-early/

Questions?

Thank you!

BonusItamar Ravid - @itrvd 117

Digression: Folds
If you squint, our function looks like a foldLeft:
def foldLeft[S, A](init: S)
(f: (S, A ) => S ): S
def processCommand: (Reservations, ReservationCommand) => Reservations

Digression: Folds
There's also an interesting variant of foldLeft when the result
has an effect:
def foldLeftM[S, A](init: S)(f: (S, A) => IO[S]): IO[S]

Digression: Folds
Since EventsAnd is a Monad, we can use foldLeftM:
val commands: List[ReservationCommand]
val resultState: EventsAnd[Reservations] =
commands.foldLeftM(Reservations.empty)(processCommand)
Any failed validation will halt the processing,
and all the events would be accumulated.

Digression: Folds
Another interesting type of fold is a scan:
def scanLeft[S, A](init: S)(f: (S, A) => S): List[S]
Looks exactly like a fold, but you get a list of intermediate states.

Digression: Folds
Scanning commands would get us a list of intermediate
Reservations:
val commands: List[ReservationCommand]
val states = commands.scanLeft(Reservations.empty)(processCommand)
// List(reservations1, reservations2, ...)

Traversals
What's the shortest way to make processCommand work
on a List[ReservationCommand]?
def processCommand(command: ReservationCommand): CommandProcessor[Unit]

Traversals
Let's try mapping.
val cmds: List[ReservationCommand]
val mapped: List[CommandProcessor[Unit]] = cmds.map(processCommand)
We get back a list of functions; not very helpful.

Traversals
The operation we're looking for is an effectful map:

Traversals
Lucky for us, we have the sequence function:
val cmds: List[ReservationCommand]
val mapped: List[CommandProcessor[Unit]] = cmds.map(processCommand)
val sequenced: CommandProcessor[List[Unit]] = mapped.sequence
So now we get one giant processing function that runs
everything!

Traversals
As the title gave away, this can be expressed as:
val traversed: CommandProcessor[List[Unit]] =
cmds.traverse(processCommand)
And we can actually run it:
traversed.run(initState) match {
case Left(_) =>
// log the error
case Right(resultState, (events, _)) =>
// persist the events and state
}

Traversals
This is our Traversable typeclass:
trait Traversable[F[_]] extends Functor[F]
with Foldable[F] {
def traverse[G[_]: Applicative, A, B](fa: F[A])
(f: A => G[B]): G[F[B]]
}

Traversals
If you line traverse up with map, the effectful map analogy is
clearer:
def map [A, B](fa: F[A])(f: A => B ): F[B]
def traverse[G[_], A, B](fa: F[A])(f: A => G[B]): G[F[B]]

Akka Streams
With Akka Streams, we can use the scan operator to process
commands:
class Source[Element] {
def scan[State](zero: State)(f: (State, Element) => State): Source[State]
}
If we set:
• Element = ReservationCommand
• State = (Reservations, List[ReservationEvent])

Akka Streams
def commandProcessFlow(initial: Reservations):
Flow[ReservationCommand, (Reservations, List[ReservationEvent]), NotUsed] =
Flow[ReservationCommand]
.scan((initial, List[ReservationEvent]())) {
case ((state, _), command) =>
processCommand(command).run(state) match {
case Left(error) => (state, Nil)
case Right(nextState, (events, _)) => (nextState, events)
}
}

Traversals and Scans and CQRS, oh my!

Recommended

Recommended

More Related Content

Similar to Traversals and Scans and CQRS, oh my!

Similar to Traversals and Scans and CQRS, oh my! (20)

Recently uploaded

Recently uploaded (20)

Traversals and Scans and CQRS, oh my!