Patterns covered: Strangler pattern Sidecar pattern Outbox pattern Saga pattern Slides from Red Hat Summit 2021 talk https://events.summit.redhat.com/widget/redhat/sum21/sessioncatalog/session/1607120592928001VuOF

1. Application modernization patterns
with Apache Kafka, Debezium,
and Kubernetes
Bilgin Ibryam
@bibryam
Product Manager
Red Hat

2. 2
We build our computers
the way we build our cities -
over time, without a plan,
on top of ruins.
Ellen Ullman

3. About me
3
▸ Product Manager at Red Hat
▸ Former Principal Architect
▸ Committer at Apache Camel
▸ Author
･ Camel Design Patterns
･ Kubernetes Patterns
▸ @ bibryam
▸ https://www.ofbizian.com

4. About this talk
4
Application migration/modernization Rs
▸ Rehosting - lift-and-shift
▸ Replatorming - lift, tinker, and shift
▸ Refactor / Re-architect
･ Debezium
･ Apache Kafka
･ Kubernetes

5. Starting point
5
Challenges with monolithic applications
▸ Frequent deployment is difficult
▸ Obstacle to scaling development
▸ Scaling the application can be difficult
Expected modernization benefits
▸ Reduce time to market
▸ Greater team autonomy
▸ Improved automation
▸ Increased application performance

6. Target state
6
Cloud native microservices’ characteristics
▸ Independently deployable
▸ Modeled around a business domain
▸ Own their data
▸ Emit events (based on EDA)
▸ Built on open source technology such as
Kubernetes, Apache Kafka, Debezium
Measuring the results
▸ Lead time for change
▸ Deployment frequency
▸ Mean time to recovery

7. Migration
Challenges

8. Strangler Pattern
8
High-level steps
▸ Identify functional boundary
▸ Migrate/reimplement functionality
▸ Synchronize data
▸ Reroute traffic to new service
▸ Decomposition old service
Main benefits
▸ No big bang migration
▸ Minimal changes to the monolith
▸ Low risk with the ability to fall back

9. Functional boundary
9
Functional boundary considerations
▸ Aggregates, bounded context based on DDD
▸ Event Storming technique by A. Brandolini
▸ Data model coupling
▸ Module dependencies
Where to start from?
▸ Easy win
▸ Noticeable improvement
▸ A representative effort

10. Interception options
10
Prerequisites
▸ Ability to identify inbound calls
▸ Ability to direct calls to new service (and back)
Routing options
▸ Client redirection
▸ Transparent HTTP proxy
▸ Shared protocol translation proxy
▸ Embedded/sidecar proxy

11. Interception options
11
Prerequisites
▸ Ability to identify inbound calls
▸ Ability to direct calls to new service (and back)
Routing options
▸ Client redirection
▸ Transparent HTTP proxy
▸ Shared protocol translation proxy
▸ Embedded/sidecar custom proxy

12. Database strangulation
12
Database first Database and code together
Code first

13. Data synchronization
13
Cons
▸ Polling delay, might miss updates
▸ Polling overhead
▸ Deletes not captured
▸ Not transparent to writing application
Pros
▸ Simple Installation and configuration
Cons
▸ Varying capabilities
▸ Limited integration capabilities
▸ Not portable
▸ Not easily testable
▸ Not transparent to source database
Pros
▸ No additional application required
Cons
▸ Requires specialized tools
▸ Requires database configuration
Pros
▸ All changes (deletes) are captured
▸ No polling delay or overhead
▸ Transparent to writing application
Triggers Log readers
Queries/Polling

14. Debezium
14
What is Debezium?
▸ Open source Change Data Capture platform
▸ Snapshotting, filtering, transformations
▸ MySQL, PostgreSQL, MongoDB, SQL Server,
Db2, Oracle, Vitess, Cassandra
▸ Use cases: data replication, cache updates, search index
updates, audit logs, sync data between services

15. Strangler Pattern with Debezium
15
Strangler Pattern with Debezium
▸ Transparent to the writing application
▸ Initial bulk import through snapshotting
▸ Schema, table, column filtering
▸ Single Message Transformations as anti-corruption layer
▸ Schema Registry for schema validation and
compatibility enforcement of data models
Debezium with Apache Kafka
▸ Guaranteed ordering, message compaction
▸ Pull based - re-read from start, or new changes
▸ Kafka Connect - offset tracking, fail over

16. Release steps
16
Steps so far
▸ Identified a functional boundary
▸ Migrated it as a new service
▸ Created a new data model in a new database
▸ Automated and deployed everything
▸ Bulk imported data and kept it in-sync
▸ No traffic routed to the new service

17. Release steps
17
Next step
▸ Route READ traffic to the new service

18. Release steps
18
Next steps
▸ Route READ and WRITE traffic to the new service
▸ Two-way data synchronization in progress

19. Release steps
19
Next steps
▸ Stop WRITE traffic to legacy system
▸ Stop data synchronization from legacy system

20. Release steps
20
Final steps
▸ STOP READ traffic to the legacy application
▸ Stop data synchronization to legacy system
▸ Decommission migrated module

21. New
Challenges

22. Modernization challenges
22
Migration challenges
▸ Low-risk changes with demonstrable progress
▸ Deliver enhancements and new “business value”
▸ Simultaneously delivering new services
Distributed systems challenges
▸ Automation and operations at scale
▸ Dual-writes and long-running business transactions
▸ Analytical and reporting data needs

23. Operating event-driven services at scale
23
Microservices on Kubernetes
▸ Deployment/rollback
▸ Placement/scheduling
▸ Configuration management
▸ Resource/failure isolation
▸ Auto heal/upgrade (through operators)
Event-driven applications on Kubernetes
▸ Strimzi - operating Apache Kafka cluster
▸ KEDA - event-driven workload autoscaling
▸ Debezium - change data capture platform
▸ Knative Eventing - event-driven primitives

24. Outbox Pattern
24
Offers an approach for services to update their data store and
notify other services in a reliable and eventually consistent
manner.
▸ Addresses the dual-write problem
▸ Offers “read your own writes" semantics
▸ “at-least-once” semantics for consumers

25. Saga Pattern
25
Splits up long-running business transactions into a series of
multiple local transactions. When implemented using the Outbox
Pattern, it offers the benefits of orchestration and asynchronous
communication.
▸ Orchestration based coordination
▸ Asynchronous communication with Apache Kafka

26. Summary
26
Modern software systems are like cities - they evolve over time, on top of legacy systems. Using
proven patterns and standardized tools help create long-lasting systems that grow with your needs.
▸ Strangler Pattern - a low-risk application migration technique
▸ Outbox/Saga Patterns - reliable inter-service communication approach
▸ Debezium - a non-intrusive toolking for change data capture
▸ Kubernetes / Strimzi / Apache Kafka - de facto standards in their fields

27. Resources
27
▸ Monolith To Microservices
▸ Building Event-Driven Microservices
▸ Designing Data-Intensive Applications
▸ Kubernetes Patterns
▸ Debezium.io
▸ Keda.sh
▸ Strimzi.io
▸ github.com/windup

28. 28
Red Hat OpenShift Streams
for Apache Kafka
a fully managed Apache Kafka service by Red Hat
http://red.ht/TryKafka
Try Apache Kafka in seconds