The adoption and popularity of the microservices architecture continues to grow across a spectrum of enterprises in every industry. Although a consensus on an implementation standard has yet to be reached, advanced design patterns and lessons learned about the complexities and pitfalls of deploying microservices at scale have been established by thought leaders and the development community. With Redis and Kafka becoming de facto standards across most microservices architectures, we will discuss how their combination can be used to simplify the implementation of event-driven design patterns that will provide real-time performance, scalability, resiliency, traceability to ensure compliance, observability, reduced technology sprawl, and scale to thousands of services. In this discussion, we will decompose a real-time event-driven payment-processing microservices workflow to explore capturing telemetry data, event sourcing, CQRS, orchestrated SAGA workflows, inter-service communication, state machines, and more.

1. Redis and Kafka
Advanced Microservices Design Patterns Simplified
ALLEN TERLETO
SENIOR DIRECTOR, CUSTOMER SOLUTIONS AND FIELD ENGINEERING
https://www.linkedin.com/in/allenterleto/
AUGUST 2020

2. 2
Explore demand and goals for Microservices
1
Microservices
Architecture
Define the characteristics of a Microservice
2
Quick introduction of Kafka and Redis
3
Advanced Microservices Architecture Patterns
4
+

4. What is holding back organizations from improving time-to-market?
4
Some Manager: We need more communication on and between our teams
Jeff Bezos: “No. Communication is terrible”

5. Software Evolution from Monolithic to Microservices
5 Source: https://www.martinfowler.com/microservices/

6. Characteristics of Microservices Architecture
6
Source: https://www.martinfowler.com/microservices/
1. Componentization via Services
2. Organized around Business Capabilities
3. Smart endpoints & dumb pipes
4. Decentralized Governance
5. Decentralized Data Management
6. Products not Projects
7. Infrastructure Automation
8. Design for failure
9. Evolutionary Design
API Gateway
Microservice
Message Broker
Microservice Microservice

7. There’s a lot to this.. so where do we begin?
7

8. There’s a lot to this.. so where do we begin?
8

9. Open Source In-Memory Database
supporting a variety of data structures and
data models optimized for
real-time and high-throughput use-cases
Open Source Stream Processing Platform
supporting distributed messaging
publish-subscribe data feeds for near-
real-time and high-throughput use-cases
9

10. Open Source Redis Modules
10 For more info check out: https://redislabs.com/community/redis-modules-hub/

11. Beautiful Chaos known as Microservices Architecture
12

12. Design Pattern - Bounded Context | Domain Driven Design
13
RedisAIRediSearch
risk:profile
input:tensors
output:tensors
RedisBloom
Authenticate
Digital Identity
user:profile
behavioral:profile
transaction:profile
geolocation
trusted:devices
Probabilistic Fraud
Detection Checkpoint
Transaction
Risk Scoring
user:transcations
Approve | Decline
Payment
Solution
• Practice Domain-Driven Design (DDD) principles
• Simplify complex domain models by dividing them into
separate Bounded Contexts; manageable by small teams
• Choose optimal data model based on each microservice’s data
access patterns and SLAs (RediSearch, RedisBloom, RedisAI, etc.)
Challenge
• Efficient communication between software
developers and domain experts
• Smaller autonomous teams are more productive
• Common terminology to establish requirements,
acceptance criteria, test cases, etc.
Bounded Context Bounded Context

13. Design Anti-Pattern – Two-Phase Commit Transactions
15
REDIS STREAMS
API Gateway
Microservice Microservice
write write
Challenge
• Each microservice can have its own database
however some business transactions will span
multiple Business Contexts
• In a Microservices Architecture, the Two-Phase
Commit protocol is an anti-pattern, since one or
more out of the hundreds or thousands of
microservices can, at any time, become unavailable
commit commitSolution
• Embrace eventual consistency and change mental model
from conforming to traditional ACID constraints
• Apply appropriate design patterns –
Message Broker, Saga, CQRS, Message Relay, Outbox, etc.

14. Design Pattern – Publish-Subscribe Message Broker
16
Kafka Message Broker
RediSearch RedisBloom
API Gateway
Authenticate
Digital Identity
Probabilistic Fraud
Detection Checkpoint
Challenge
• Two-Phase Commit protocol is an anti-pattern
• Transactions will span multiple business contexts
• Microservices can become unavailable at anytime
Solution
• Embrace event-driven architecture principles
• Inter-service communication (events) asynchronously
flow through a publish-subscribe Message Broker
• Zero, one, or more microservices can subscribe to
events that are relevant to their Business Context;
each at their own pace
• Microservices, recently recovered from unavailability, can
catch up to all the events they missed while unavailable
publish
subscribe subscribe

15. Design Pattern – Choreography-based Saga (State Machine)
17
Kafka Message Broker
RedisAIRedisBloom
Probabilistic Fraud
Detection Checkpoint
Transaction
Risk Scoring
Approve | Decline
Payment
YesNo
YesNo
Challenge
• Transactions will span multiple Business Contexts and the Two-Phase Commit protocol is an anti-pattern
• Microservices can include decision-logic that impacts which downstream services are in the transactional scope
Solution
• Saga is a sequence of local transactions
• Choreography-based Sagas are not
centrally coordinated
• Has roots in Finite-State Machine
• Each local transaction updates its
database and publishes an event to
trigger the next local transaction
• Sagas allow for unavailability along
their Chain-of-Responsibility, rollback
compensation for failures, and can replay the entire transaction

16. Design Pattern – Transactional Outbox | Message Relay
18
RediSearch
Kafka Message Broker
Authenticate
Digital Identity
RedisGears
user:profile
behavioral:profile
transaction:profile
geolocation
trusted:devices
Challenge
• Writing atomically to a database and a message broker can add
unnecessary code complexity
• Message Broker unavailability would force decisions about each
microservice’s trade-offs between availability and consistency
Solution
• Separate the concern of managing inter-service communication
by using an event-driven Message Relay
• RedisGears can be used to easily implement the Transactional
Outbox pattern by listening for events within Redis databases and
relaying event notifications (write-behind pattern) to a Message
Broker for downstream inter-service communication
• Redis-based microservices would avoid code complexity, concerns
with Message Broker unavailability, and performance overhead

17. Design Pattern – Capturing Telemetry
19
RedisGears
RedisTimeSeries
Analytics Dashboard
RediSearch
user:profile
behavioral:profile
transaction:profile
geolocation
trusted:devices
RedisAI
risk:profile
input:tensors
output:tensors
RedisBloom
user:transcations
RedisGears RedisGears
RedisTimeSeries
Challenge
• Telemetry is built on three pillars –
metrics, logs, and tracing
• Metrics are used for holistic observability
of a distributed-system in real time
• Metrics and tracing visualization often use
an underlying data model called “time-series”
Solution
• RedisTimeSeries has a native time-series data
model supporting high-volume inserts, low-
latency reads, aggregated queries, search-by
labels, downsampling, retention, indexing, etc.
• Metrics can be collected at various levels –
KPIs, application SLAs, infrastructure utilization, etc.

18. How can developers visualize time-series telemetry data?
20 Download for free: https://redislabs.com/redisinsight/

19. How can business analysts visualize time-series telemetry data?
21 For more info check out: https://github.com/mikhailredis/redis-pop-up-store and https://github.com/RedisTimeSeries/grafana-redis-datasource

20. Design Pattern – Event Sourcing
22
RedisAI
Analytics Dashboard
Redis Streams
risk:profile
input:tensors
output:tensors
RedisBloom
user:transcations
RedisGears RedisGears
Kafka Message Broker
Challenge
• Observability, tracing, and auditing of Choreography-based
Sagas is difficult across a decoupled chain of microservices
Solution
• Event Sourcing implies recording every microservice’s state
change as an immutable event; acting as a source of truth
• It is composed of a Message Broker and Event Store
• Based on the ordered sequence of events persisted in the
event store, a microservice can replay/rebuild its system
state by reprocessing recorded events at any time
• Redis Streams is an immutable in-memory append-only log
data structure perfectly suited and used as an Event Store
• Visualizing a stream of events can provide real-time
observability of the system’s state and instant recovery/replay

21. How can we visualize a stream of events?
23

22. Design Pattern – CQRS (Command Query Responsibility Segregation)
24
API Gateway
Approve | Decline
Payment
Payment History
RedisCDC
Redis Enterprise
readwrite
Challenge
• Each microservice should choose an optimal data model
for its unique data access patterns and SLAs
• There is no single data model optimally suited for all needs
Solution
• CQRS implies maintaining separate data structures for
writing (command) and reading (query) information
• Allows a microservice with extreme durability SLAs to use a
write-optimized database, while another microservice that
requires access to the same data uses a read-optimized database
• Managing eventual consistency between models is commonly
handled using the Transaction Log Tailing pattern
(a.k.a. Change-Data-Capture) or Message Relay pattern (covered earlier)
• RedisCDC provides a seamless drop-in-solution to keep heterogenous databases in-synch with Redis

23. Design Pattern – Shared Data (Microservice Level)
25
API Gateway
Approve | Decline
Payment
Payment History
RedisCDC
Redis Enterprise
readwrite
Clearing & Settlement
read
Challenge
• Multiple microservices need to
query the same data which
could be sourced from multiple
downstream microservices
Solution
• Implement the CQRS pattern
to build a view-only database
composed of replicated state-
change events from one or more
source databases
• Redis is optimally suited as a view-only
database, or microservices-cache,
because it inherently guarantees atomicity
for each operation, across multiple clients,
and provides sub-millisecond latency at scale

24. Design Pattern – Shared Data (Global Level)
26
Redis Enterprise
user:session
oauth:token
Kafka Message Broker
RediSearch RedisBloom
API Gateway
Authenticate
Digital Identity
Probabilistic Fraud
Detection Checkpoint
Challenge
• Session state, client authentication tokens,
and/or other global data may need to
be shared across hundreds of microservices
• A shared database, or ephemeral cache,
would couple all microservices together
• A global database could become
a single-point of failure user:transcations
user:profile
behavioral:profile
transaction:profile
geolocation
trusted:devices
Solution
• API Gateway deploys its own isolated and
highly-available database to store global data
• Redis is used across thousands of
microservices architectures in production
to manage session state and client authentication tokens

25. Retrospective
27
Challenges
• There’s more to microservices than breaking off chunks of code from a monolith
• Data Management quickly becomes a pain point if best practices are not considered
• Decentralization comes with trades-offs to complexity, technology sprawl, observability, etc.
Solutions
• Microservices design patterns help avoid complexity at scale
• Redis supports multiple models optimally suitable to simplify their implementation
• Kafka and Redis help turn Microservices Architecture’s beautiful chaos into controlled chaos

26. Check out my interview with Chris Richardson from
28
https://www.youtube.com/watch?v=q5Z8-cwGBXQ

27. Free Redis Microservices e-book for Developers
29 https://redislabs.com/docs/redis-microservices-for-dummies/

28. Check out these free resources to learn more about Redis
30
• http://redis.io/
• https://university.redislabs.com/
• https://docs.redislabs.com/latest/rs/
• https://www.youtube.com/c/RedisUniversity

29. Thank you!
redislabs.com