(Benny Lee + Christopher Arthur, Bank of Australia) Kafka Summit SF 2018 Commonwealth Bank of Australia (CBA) is Australia’s largest bank with over 15m customers, 50,000 employees and over USD700 billion in assets. We started the journey two years ago to transform our existing enterprise architecture into an “event driven” architecture. Since then, Kafka has become a mission critical platform in the Bank and it is the core component in our “event driven” architecture strategy. In this talk, we will walk you through the journey of how we stood up the initial Kafka clusters, the challenges we encountered (both technical and organisational) and how we overcame those challenges. We will also deep dive into one of the use cases for Kafka (with Kafka Streams and Connectors) in our new real time payment system that was introduced in Australia early this year. We will discuss why we think Kafka was the perfect solution for this use case, and the lessons learned. Key Takeaways: -Lessons learned from our experiences (that we think other companies could be able to benefit from) -Our use cases for Kafka with a particular focus on the new real time payment systems (NPP) initiative in Australia

1. Kafka in the Enterprise -
Our Two-Year Journey to Build a Data Streaming Platform
Christopher Arthur, Product Owner / Lead Architect
Benny Lee, Solutions Architect

2. 1
2
3
4
5
6
Commonwealth Bank of Australia
Agenda
About us
Where it all started?
Use Cases
Where we are now
Lessons we have learnt (so
far…..)
2
Questions

3. About us
• Founded in 1911
• Australia's leading provider of financial services.
• Total assets – over $700 billion in USD
3
Source: Based on the Banks Daily 2018 ranking as at May 31, 2018

4. Where it all started?
• Current Integration Patterns
4
• Digital growth
• Analytical workload demand
• Real time fraud monitoring and detection
• Regulatory and compliance demand

5. Where it all started?
• Low Cost Reads Project
5

6. Adoption Rate
6
0
20
40
60
80
2016 2017 2018
Number of Projects
Over 400% YTY
increase in demand

7. 7
Event Hub
WealthRetail
Banking
Business &
Private Banking
Online
Brokerage
Institutional
Banking &
Markets

8. 8

9. Use Cases
• Customer notifications
• Workflow initiation
• Account fulfilment
• Customer on boarding notifications
• Auditing and Fraud detection for online transactions
• Real time payment registration and maintenance
• Payment enrichment
• Task replication
9

10. New Payments Platform (NPP)
• Launch in February 2018
• Make or receive payments in real-time, 24/7
• Addressing Service (PayID)
› Initiate payments using mobile number, email address or Australian Business
Number (ABN)
10

11. NPP – PayID Registration
11

12. NPP – Payment Enrichment
12

13. Task Replication
13

14. Where we are now
• Event Driven Architecture
14

15. Top Lessons we have learnt (so far…..)
1. Governance and Self Service
2. Message duplication
15

16. Governance and Self Service
16

17. 17
Topic
Creation
Topic
Discovery
Data
Lineage
Viewing
Message
Topic
Ownership
Topic
Monitoring
When supporting multiple teams

18. 18
Topic management

19. 19
Topic management

20. 20
Topic Viewer

21. 21
Topic Viewer

22. 22
Data Lineage
• How do we keep track where
messages are coming from?
• How do we keep track how
messages are moving across
multiple topics?
• Multiple applications publish messages into Kafka

23. 23
Data Lineage

24. Message duplication ( and message duplication…)
24
• Messages persist in Kafka
• Consuming Application might ‘accidentally’ reprocess all the messages again
› Default offsets retention (offsets.retention.minutes)!
• Consequences: Negative impacts on the Platform
› Reduce adoption
› Bad reputation

25. Message duplication
25
• Consider……..
• Topic retention period
• Change offsets.retention.minutes (and auto.offset.reset)
• Use Kafka ”exactly once” feature
• Ensure downstream applications are idempotent.
• Deduplicate messages before sending to downstream applications.
• Duplication needs to be considered in every use case end to end
• Understanding the impact if duplication occurs
• Mitigation procedure
• Testing, testing and testing!

26. Thank you

27. Questions?