Video recording of this presentation: https://youtu.be/upWzamacOVQ Blog post with more details: https://www.kai-waehner.de/blog/2020/04/24/mainframe-offloading-replacement-apache-kafka-connect-ibm-db2-mq-cdc-cobol/ Mainframes are still hard at work, processing over 70 percent of the world’s most essential computing transactions every day. Very high cost, monolithic architectures, and missing experts are the key challenges for mainframe applications. Time to get more innovative, even with the mainframe! Mainframe offloading with Apache Kafka and its ecosystem can be used to keep a more modern data store in real-time sync with the mainframe. At the same time, it is persisting the event data on the bus to enable microservices, and deliver the data to other systems such as data warehouses and search indexes. But the final goal and ultimate vision are to replace the mainframe by new applications using modern and less costly technologies. Stand up to the dinosaur, but keep in mind that legacy migration is a journey! Kai will guide you to the next step of your company’s evolution! You will learn: - how to not only reduce operational expenses but provide a path for architecture modernization, agility and eventually mainframe replacement - what steps some of Confluent’s customers already took, leveraging technologies like Change Data Capture (CDC) or MQ for mainframe offloading - how an event streaming platform enables cost reduction, architecture modernization, and a combination of a mainframe with new technologies

1. Mainframe Integration, Offloading
and Replacement with Apache Kafka
Stand up to the Dinosaur!
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

2. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Event Streaming
3. Apache Kafka as Cloud-native and Hybrid
Infrastructure
4. Mainframe Integration, Offloading and
Replacement
5. Case Study
Agend

3. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Event Streaming
3. Apache Kafka as Cloud-native and Hybrid
Infrastructure
4. Mainframe Integration, Offloading and
Replacement
5. Case Study
Agend

4. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
The Mainframe is here to stay!
“Mainframes are still hard at work,
processing over 70 percent of the world’s
most important computing transactions
every day. Organizations like banks, credit
card companies, airlines, medical facilities,
insurance companies, and others that can
absolutely not afford downtime and errors
depend on the mainframe to get the job
done. Nearly three-quarters of all Fortune
500 companies still turn to the mainframe to
get the critical processing work completed”
https://www.bmc.com/blogs/mainframe-mips-an-introduction/

5. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
What is a Mainframe?
Modern mainframe design is characterized less by raw
computational speed and more by:
• High reliability and security
• Extensive input-output ("I/O") facilities with the ability to
offload to separate engines
• Strict backward compatibility with older software
• High hardware and computational utilization rates through
virtualization to support massive throughput
• Hot-swapping of hardware, such as processors and memory
Vendors: “IBM and the Seven Dwarfs”
The IBM z15, announced in 2019,
with up to 40TB RAM and 190 Cores,
typically costs millions $$$
(variable software costs not included)

6. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
COBOL (COmmon Business Oriented Language)
6
• Available since the 1950s. Latest update: 2014.
• Compiled English-like computer programming language designed for business use
• From column-based punch-card-image format to object-oriented programming
• Concerns: Lack of structure, compatibility issues, verbose syntax, isolation from computer science community

7. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Neobanks and FinTechs Hunting the Traditional Banks
Monolithic
Proprietary
Complex
Inflexible

8. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
How long does it take do deploy a new feature
to the mainframe in your organization?
8
“N26 is a relatively young company, so we
take advantage of a modern tech stack…
We typically practice continuous
deployment, so every merge goes
through rigorous automated testing and
gets deployed to live”
https://www.infoq.com/news/2020/01/scaling-infrastructure-code/

9. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Not just the FinTechs Modernize their Architecture!
9
https://www.confluent.io/kafka-summit-london18/distributing-computing-key-player-in-corebanking-platforms/
Sberbank re-implemented their Core Banking Platform
around Kafka and Event Streaming to be ready for the future!

10. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
The mainframe supports
up-to-date technologies such as
DB2, MQ, WebSphere, Java, Linux,
Web Services, Kubernetes, Ansible!
https://www.zazzle.com.au/cobol+dinosaur+gifts

11. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Deploy Kafka and its Ecosystem on the IBM z15 Mainframe…

12. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
MIPS (million instructions per second)
to normalize CPU usage across CPU types and models or hardware configs
MSU (million service units)
hardware and software metrics calculated directly by the operating system
12
… and what about
hiring mainframe
experts?

13. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Huge demand to build an open, flexible, scalable platform
• Real time
• Scalability
• High availability
• Decoupling
• Cost reduction
• Flexibility
• Standards-based
• Extensibility
• Security
• Infrastructure-independent
• Multi-region / global

14. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Event Streaming
3. Apache Kafka as Cloud-native and Hybrid
Infrastructure
4. Mainframe Integration, Offloading and
Replacement
5. Case Study
Agend

15. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Event Streaming in the Finance Industry
Check past Kafka Summit videos for details about the use cases:
https://kafka-summit.org/past-events/

16. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Event Streaming for
Traditional and New Innovative Use Cases
in the Finance Industry
Real Time Processing Digital Transformation Strategic Goals
Short-Sale Risk
Calculation / Trade
Approval
Mainframe Offloading
and Replacement
Credit Card
Fraud Detection
Next-Best
Offer
Robot Process Automation
(e.g. Know Your Customer,
KYC)
Customer Service
(e.g. Chat Bots)
IT
ModernizationRegulatory
Reporting
Account Login
Fraud Detection
Anomaly Detection
Across Assets and Locations
Derivatives
Pricing Compliance
Trading Post-
Processing
Strategic
Planning and
Simulations

17. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
https://www.confluent.io/customers/rbc/ “… rescue data off of the mainframe, in a cloud native,
microservice-based fashion … [to] … significantly reduce the
reads on the mainframe, saving RBC fixed infrastructure
costs (OPEX). RBC stayed compliant with bank regulations
and business logic, and is now able to create new applications
using the same event-based architecture.”

18. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Event Streaming
3. Apache Kafka as Cloud-native and Hybrid
Infrastructure
4. Mainframe Integration, Offloading and
Replacement
5. Case Study
Agend

19. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Event Streaming Platform –
The Commit Log
Time
P
C1 C2
C3

20. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Event Streaming Platform –
A Distributed System for 24/7 and Zero Data Loss
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

21. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
An Event Streaming Platform
is the Underpinning of an Event-driven Architecture
Microservices
Mainframes
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservices
events
SaaS
data
Customer
experience
s
Streams of real time events
Stream processing
apps
Connectors
Connectors
Stream processing
apps

22. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Apache Kafka at Scale at Tech Giants
> 7 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data

23. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Kafka Connect
Kafka Cluster
CRM
Integration
Domain-Driven Design and Decoupled Applications
Legacy
Integration
Custom
Application
Mainframe
Connector
Java / C++ /
Go / Python /
KSQL
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Payment Domain Fraud Domain
Audit Logs,
RBAC, etc.

24. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Mission-Critical
How to
deploy this 24/7,
including
Disaster Recovery?

25. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Disaster Recovery – RPO and RTO
RPO = Recovery Point Objective
RTO = Recovery Time Objective

26. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
IBM GDPS
● Synchronous and asynchronous mirroring
● Different RTO / RPO setups
● Built for resiliency and disaster recovery
● Supports multiple sites
● Uses concepts like XCF (Cross-system Coupling
Facility), Parallel Sysplex, Disk Mirroring,
HyperSwap, etc.
● Independent of transaction manager (e.g. CICS,
IMS) or database manager (e.g. DB2 , IMS, VSAM)
https://www.ibm.com/it-infrastructure/z/technologies/gdps
https://ibmsystemsmag.com/IBM-Z/07/2019/resiliency-gdps-solutions

27. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Kafka Clusters
can Stretch over Regions
Zero Downtime + Zero Data loss
(RPO=0 and RTO=0)
e.g. Stretched over US East + Mid + West
Automate Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without
Custom Code
Multi-Region Cluster
(Only available in Confluent Platform)

28. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Example of a Multi-Region Cluster in a Bank
Large FinServ Customer
Payment
Log
Payment
Log
Location Location
synchronous
asynchronous
● ‘Payment’ transactions enter
from us-east and us-west with
fully synchronous replication
● ‘Log’ and ‘Location’ information
in the same cluster use async -
optimized for latency
● Automated disaster recovery
(zero downtime, zero data loss)
Result: Clearing time from ‘deposit’ to
‘available’ goes from 5 days to 5
seconds (including security checks)
(Only available in Confluent Platform)

29. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Global Event Streaming
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO=0
Stream Data Globally with
Replication and Cluster Linking

30. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Event Streaming
3. Apache Kafka as Cloud-native and Hybrid
Infrastructure
4. Mainframe Integration, Offloading and
Replacement
5. Case Study
Agend

31. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
What to do to stay competitive and be innovative?
Big Bang
Replacements
Usually Fail!

32. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Mainframe
Offloading
Journey from Mainframe
to Hybrid* and Cloud
PHASE
3
Hybrid
Replication
Mainframe
Replacement
PHASE
2
PHASE
1 * with or without the mainframe

33. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Mainframe Offloading
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
MIPS / MSU
MIPS / MSU
MIPS / MSU
Read
No MIPS / MSU

34. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Mainframe Replacement
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
MIPS / MSU
MIPS / MSU
MIPS / MSU
Read
No MIPS / MSU

35. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Mainframe Replacement (without Writing New Apps)
https://learnworthy.net/could-java-be-the-next-cobol/
https://medium.com/@FranzRoses/the-enterprise-journey-to-decompose-the-cobol-banking-core-into-java-the-developer-perspective-2e8a53bb528e
https://bs2manuals.ts.fujitsu.com/psBEANCONNECTV65en/beanconnect-user-guide-user-guide-13821/cobol2java-bc-ug-499/mapping-cobol-data-types-to-java-classes-bc-ug-500
What about
the 5% of
(complex) code
that cannot
be migrated
automatically?
Showstopper?

36. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Orders Customers
Payments
Stock
WebSockets / SSE
JMS
ESB
REST
Java
Connect
RPC
MQ
Integration between Kafka and Mainframes
MQ
C++ File
???

37. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Change Data Capture (CDC)
Transaction log-based CDC pushes data changes (insert, update, delete) from the mainframe
to Kafka in real time.
+ Real time push updates to Kafka
+ Eliminate disruptive full loads, i.e. minimize production impact
+ Reduce MIPS consumption
+ Integrate with any mainframe technology
(DB2 z/OS, VSAM, IMS/DB, CICS, etc.)
+ Full support
- High licensing costs

38. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Integration Options between Kafka and Mainframe
• IBM InfoSphere Data Replication (IIDR) Change Data Capture (CDC) solution for Mainframe
• 3rd Party commercial CDC solution (e.g. Attunity, DBS-H, B.O.S. Software, …)
• Open-source CDC solution (e.g. Debezium) – but you still need an IIDR license, this is the same
challenge as with Oracle and GoldenGate CDC)
• DB2 SQL Integration: Create interface tables + Kafka Connect + JDBC connector
• IBM MQ interface + Kafka Connect’s IBM MQ connector
• VSAM File Integration (e.g. B.O.S, Luminex, Syncsort, Qlik Replicate, …)
• Confluent REST Proxy and HTTP(S) calls from the mainframe
• Kafka Client APIs on the mainframe
Evaluate the performance / scalability / feature set / cost
of the tools and the footprint on the mainframe!
Don’t underestimate politics…

39. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Event Streaming
3. Apache Kafka as Cloud-native and Hybrid
Infrastructure
4. Mainframe Integration, Offloading and
Replacement
5. Case Study
Agend

40. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Cloud
Adoption
Journey from Mainframe
to Hybrid and Cloud
PHASE
3
Hybrid
Cloud
Cloud-First
Development
PHASE
2
PHASE
1
https://www.accenture.com/_acnmedia/pdf-70/accenture-moving-to-the-cloud-strategy-for-banks-in-north-america.pdf
Case Study - Bank CEO
“This is the last 5-year $20M IBM contract.
Get rid off the mainframe!”

41. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Strangler Design Pattern
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/
https://martinfowler.com/bliki/StranglerFigApplication.html
“The most important
reason to consider a
strangler fig application
over a cut-over rewrite
is reduced risk.”
Martin Fowler

42. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 0: Direct Communication between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)

43. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 1: Kafka for Decoupling between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool

44. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 2 to 4: New Projects and Applications
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project
(Elastic, Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool

45. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 5: Mainframe Replacement
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project
(Elastic, Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
Core Banking ‘2020’
(Modern Technology)
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14

46. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
What about Transactions?
48

47. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
IBM Mainframe Database and Transaction Managers
49
IMS
• Hierarchical Database
• Transaction Manager
• Supports Cobol, Assembler, PL/1, Java
• IMS Connect for Integration with WebSphere MQ, SOAP, …
DB2
• Relational Database
CICS
• Transaction Manager
• Database “Lite” (VSAM Datasets)
• Integration and Application Programming Capabilities similar to IMS,
but much easier to use
• Advanced Features like Transaction Prioritization
The Heart
of your
Business
App

48. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
“Transactions” in Apache Kafka
50
Exactly-Once Semantics (EOS)
available since Kafka 0.11:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
https://www.confluent.io/kafka-summit-london18/dont-repeat-yourself-introducing-exactly-once-semantics-in-apache-kafka/

49. Mainframe Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Bi-Directional Integration with Referential Integrity
51
Java
App
Python
App
Mainframe
Transactions
Bi-Directional Integration
Secured Referential Integrity
End-to-End “Transactions”
Low Latency
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Kafka
Exactly-Once
Semantics
using librdkafkaIMS
DB
Cobol
App
Scope of the Middleware

50. Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!