Scaling an Event-Driven Architecture with IBM and Confluent | Antony Amanse and Anton McConville, IBM

1. Scaling Microservices
with Kafka

2. © 2021 IBM Corporation 2
Kafka
Key Components
• Brokers
• Producers
• Topic
• Partitions
• Consumers and Consumer Group
• Messages or Records
https://ibm-cloud-architecture.github.io/refarch-eda/technology/kafka-overview/

3. © 2021 IBM Corporation 3
Topics and Partition
https://ibm-cloud-architecture.github.io/refarch-eda/technology/kafka-overview/

4. Consumers and Consumer Group
© 2021 IBM Corporation 4
https://ibm-cloud-architecture.github.io/refarch-eda/technology/kafka-producers-consumers/

5. Consumer Lag
© 2021 IBM Corporation 5
https://ibm-cloud-architecture.github.io/refarch-eda/technology/kafka-producers-consumers/

6. © 2021 IBM Corporation 6
KEDA
Kubernetes Event-driven Autoscaling
KEDA is a Kubernetes-based Event Driven Autoscaler. With
KEDA, you can drive the scaling of any container in Kubernetes
based on the number of events needing to be processed.
KEDA works alongside standard Kubernetes components like
the Horizontal Pod Autoscaler.
With Kafka, KEDA scales based on consumer lag.
Install via Helm or OperatorHub on OpenShift
https://keda.sh/

7. © 2021 IBM Corporation 7
KEDA Architecture
https://keda.sh/docs/2.2/concepts/#architecture

8. © 2021 IBM Corporation 8
KEDA
Event Sources
https://keda.sh/

9. © 2021 IBM Corporation 9
KEDA
https://keda.sh

10. Example KEDA
ScaledObject
10
© 2021 IBM Corporation

11. Example KEDA
ScaledObject
© 2021 IBM Corporation 11

12. OPENSHIFT
KAFKA
API SERVICE
ORDER
CONSUMERS
COURIER
CONSUMERS
KITCHEN
CONSUMERS
REDIS
APP UI
STATUS
CONSUMERS
MONGODB MONGODB MONGODB
REALTIME DATA
POD DATA
KEDA
Example Architecture
© 2021 IBM Corporation 12

13. Demo
© 2021 IBM Corporation 13

14. Demo
© 2021 IBM Corporation 14
GitHub (WIP):
github.com/IBM/scaling-apps-with-kafka
OPENSHIFT
KAFKA
API SERVICE
ORDER
CONSUMERS
COURIER
CONSUMERS
KITCHEN
CONSUMERS
REDIS
APP UI
STATUS
CONSUMERS
MONGODB MONGODB MONGODB
REALTIME DATA
POD DATA
KEDA

15. Resources
© 2021 IBM Corporation 15
KEDA
https://keda.sh/
Kafka Overview
https://ibm-cloud-architecture.github.io/refarch-eda/technology/kafka-overview/
https://www.confluent.io/what-is-apache-kafka
Kafka on Confluent Cloud
https://www.confluent.io/

16. Future Content
© 2021 IBM Corporation 16
Code Pattern:
• Scaling Apps with Kafka using KEDA (published)
https://developer.ibm.com/patterns/scaling-an-
event-driven-architecture-with-kafka-on-openshift/
Article
• Capacity Planning

17. Thank you
© 2021 IBM Corporation 17
Anthony Amanse
Twitter @AnthonyAmanse
Anton McConville
Twitter @antonmc

18. © 2021 IBM Corporation 18