The document summarizes Bobby Calderwood's presentation on using Redis Streams to build event-driven microservices and user interfaces in Clojure(Script). The presentation covers how Redis Streams were used to facilitate asynchronous processing and distributed consistency for a customer project. It also discusses how Carmine, the Clojure Redis client, was updated to support Redis Streams shortly after their release. The presentation concludes with a demo of how Redis Streams can be used to retrofit an existing system with asynchronous integration.

1. PRESENTED BY
Using Redis Streams to build
Event-driven Microservices and
User Interface in Clojure(Script)
Bobby Calderwood
Evident Systems, Founder
bobby@evidentsystems.com
@bobbycalderwood

2. PRESENTED BY
• Software Engineer @Cognitect for 4+ years, Consulting and Datomic
• Distinguished Engineer @CapitalOne for 3+ years, helping redesign and move
mission-critical banking systems to the cloud
• Founder @Evident_Systems, helping smaller banks redesign and move
mission-critical banking systems to the cloud
• Meet with me! http://meet.evident.systems/founder
Hello, I’m Bobby Calderwood!

3. PRESENTED BY
1 Story of a real customer system
Event-Sourcing + Redis + Clojure(Script) = Joy
2 Aside: Memoir of an occasional Open Source contributor
How we got Clojure’s Carmine to speak Redis Streams
3 Our Story Continues: The (not so) Big Refactor
Retrofitting Redis Streams into a working system
Agenda:
4 Demo System
Let’s pretend we have to build a system, and then modify it with an asynchronous integration.

4. PRESENTED BY
• Major automobile manufacturer
• Greenfield system for new capability, political proving ground for stack and process
• Highly asynchronous image processing and significant UI display/manipulation
• Possible need for data re-processing/re-play, undo/redo, audit, and sharing with
other teams
• We decided to architect it using Event Sourcing on Streaming Data using
Redis, and implemented in Clojure(Script)
I was solving a customer problem

5. PRESENTED BY
Why Event Sourcing
& Streaming?

6. PRESENTED BY
• At very least, we knew we had to
process images asynchronously
• We suspected that we’d need to
keep this pipeline flexible, to try
different processing steps and
techniques
• We knew we’d need to convey
the results of this pipeline back
to the UI (the oft-forgotten
participant in the distributed
system)
To facilitate our asynchronous processing pipeline

7. PRESENTED BY
• Integration among microservices becomes complex and burdensome
– Balkanization of data stores
– Data “owner” determines data access pattern
– Non-determinism == impossibility of reasoning about system:
• state
• causality/time
• modes of failure
Because HTTP APIs aren’t enough

8. AWS Death Star diagram, circa 2008 as per Werner Vogels tweet

9. PRESENTED BY
• Request/response protocols face a fundamental epistemological problem
– What is happening, anything new?
– When should I check for new data?
– Who’s responsible to tell me?
Because HTTP APIs aren’t enough

10. PRESENTED BY
• Event Sourcing provides as-of
consistency across distributed
participants
• Streaming Data protocols convey new
data/events to interested participants
(subscribers)
• Any participant can simulate
synchrony and consistency by
blocking and waiting for a specific
event
Streaming Data + Event Sourcing Facilitate Distributed Consistency
Image by Alan Light CC BY-SA 3.0

11. PRESENTED BY
• “Situated Programs” a la Rich Hickey:
– interact with other systems
– often interact with humans
– remain in use for long periods of time
– are situated in a changing world
– use other people’s code
• In large enterprises
– audit/compliance/governance
– certain eventual integration with many new possible systems and data
consumers
– across many teams and organizational units who (basically) trust each other
More generally, to solve a specific type of problem

12. Nasa, Public Domain

14. PRESENTED BY
Why Redis?
(for streaming data)

15. PRESENTED BY
‘cause your last slide
definitely said “Kafka”

16. PRESENTED BY
• Redis is ubiquitous: you (or your customer) already has it in the stack
• Redis is battle-tested, and Ops is comfortable with it
• Redis is (often) easier to operate than Kafka (with some tradeoffs)
Redis is Already There, and Just Works

17. PRESENTED BY
• After Kafka Popularized, Salvatore noticed gap in Redis, as he described: http://
antirez.com/news/114
• Existing data structures got almost there
– Sorted Sets: memory hungry, clients can’t block awaiting new records, order not
stable
– Lists: linear time to scan, can’t seek to specific position, no fan-out to multiple
consumers
– Pub/Sub: doesn’t maintain history, can’t do windowed aggregation, etc.
• Redis Streams maintains range-queryable history, and efficiently conveys to
awaiting clients
Redis recently added Streams

18. PRESENTED BY
Why Clojure?

19. PRESENTED BY
• Clojure is a fantastic all-purpose programming language whose philosophy aligns
well with event sourcing, streaming data
• We needed to build all components quickly, with a small team of Clojurists:
– UI in ClojureScript
– React + Reagent + re-frame
– Access to JavaScript libraries with Clojure language semantics
– HTTP API in Clojure + Pedestal (+ GraphQL via Lacinia)
– Workers in Clojure
• Clojure’s core.async library provide in-process semantics aligned with streaming data
Clojure(Script) is powerful and full-stack

20. PRESENTED BY
• Very solid, simple Redis client
• Generated from Redis artifacts (on which more to come)
• Stays close to Redis semantics
Clojure has Carmine

21. PRESENTED BY
But Carmine didn’t have Streams…

22. PRESENTED BY
• ~1 hour after Salvatore’s blog post dropped, I filed https://github.com/
ptaoussanis/carmine/issues/208
• But I felt Open Source guilt, so I opened https://github.com/ptaoussanis/
carmine/pull/210
• But Carmine is generated from Redis Docs JSON, which didn’t yet include
Streams operations, so I filed https://github.com/antirez/redis-doc/issues/943,
and opened https://github.com/antirez/redis-doc/pull/963
…so we added support for Streams

23. PRESENTED BY
And support for all the above was
added incredibly quickly!

24. PRESENTED BY
• We had already built the HTTP API and processing pipeline
• Both components were producers and consumers of events
• Our implementation was limited in its scalability/fault-tolerance, so we needed
to re-work
• Ugh, how much re-work to support Streams?!?
…but not quickly enough to prevent re-work

25. PRESENTED BY
…including re-tooled deployment pipeline to support multiple workers
How much rework?
$ git diff --stat master
<snip>
16 files changed, 576 insertions(+), 598 deletions(-)

26. PRESENTED BY
• XADD — Produce to a Stream
• XRANGE, XREVRANGE — Query stream per range of existing offsets, non-
blocking, batch-ish
• XREAD — Optionally-blocking read (conveys novelty), realtime-ish
• XREADGROUP — Optionally-blocking read by a group of consumers, ack-ing
messages to advance consumer offset (but watch message ordering!)
But How to Redis Stream?
https://redis.io/topics/streams-intro

27. PRESENTED BY
And Now,
A Completely Unrelated
Demo App!

28. PRESENTED BY
• Always use Clojure, so that your major distributed systems refactorings can be
a net code reduction!
• Sharpen your tools: contribute to open source!
• Use Event Sourcing!
– Kafka for big/important/durable/expensive things
– Redis for smaller things https://redis.io/topics/streams-intro
• Check out my demo app! https://github.com/bobby/redisconf19-demo
Conclusion

29. Thank you!
bobby@evidentsystems.com
@bobbycalderwood

30. PRESENTED BY