From Trickle to Flood with Kafka@ING (Filip Yonov & Timor Timuri, ING) Kafka Summit SF 2019
Kafka at ING has a long history. It all started in 2014 when Kafka was introduced to support our use-cases for fraud detection. The following years saw Kafka growing until 2018 when it took the spotlight at ING as the #1 searched-for technology with an unprecedented adoption curve. Suddenly what was a small trickle of niche use-cases, became a flood of customers on-boarding for every imaginable usage pattern. Even more astonishing was that our single cluster configuration saw almost 700% load increase just in 2018. During the explosion our team was so busy with on-boarding, maintenance and ops that it wasn't clear whether we were supporting or sabotaging our long-term success. It was clear that we needed to challenge our view of how we used Kafka. We asked ourselves: given the demand, how can our clients easily manage their streams while not caring about scaling, clusters or technologies. How to provide Kafka to all 40 ING markets with a single experience regardless of country, app or use-case? To answer that we had a to undertake a paradigm shift - from a single-cluster on premise to multi-cluster hybrid cloud, from ops to self-service, from cluster-based to event-first architecture. In this talk we share our journey of how we completely re-imagined Kafka at ING while serving our clients at lightning speed. We would like to discuss our experience of running a single cluster with more than 1000 topics and show how we made Kafka truly self-service via our Streaming Marketplace. Last, but not least, how the past years of success with Kafka have given us the courage to go all-in on the event-first thinking and never think back, putting ING once again one step ahead. Co-presented by Filip Yonov, Product Owner Kafka @ ING
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to From Trickle to Flood with Kafka@ING (Filip Yonov & Timor Timuri, ING) Kafka Summit SF 2019
Similar to From Trickle to Flood with Kafka@ING (Filip Yonov & Timor Timuri, ING) Kafka Summit SF 2019 (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
From Trickle to Flood with Kafka@ING (Filip Yonov & Timor Timuri, ING) Kafka Summit SF 2019
- 1. Max. width Collaborate. Innovate. Scale. From trickle to flood with Kafka @ ING Timor Timuri, IT Engineer Filip Yonov, Product Owner
- 2. 2 1. Earn the primary relationship 2. Develop analytics skills to understand our customers better 3. Increase the pace of innovation to serve changing customer needs 4. Think beyond traditional banking to develop new services and business models Empowering people to stay a step ahead in life and in business. Simplify & Streamline Operational Excellence Performance Culture Lending Capabilities Purpose Customer Promise Strategic Priorities Enablers Creating a differentiating customer experience Clear and Easy Anytime, Anywhere Empower Keep Getting Better
- 3. Market Leaders Growth Markets Wholesale Banking activities only Challengers World map 3
- 4. Kafka@ING is serving more than 300 teams Eventbus Business events Logging Any other Use- case 1 Cluster, 12 Brokers Traffic grew 800% last year 420 564 665 758 870 1113 0 200 400 600 800 1000 1200 2018-11 2019-01 2019-03 2019-05 2019-07 2019-09 Topics Count 13.2 22.2 55.2 65.3 74.6 112.5 0 20 40 60 80 100 120 2018-11 2019-01 2019-03 2019-05 2019-07 2019-09 MessagesInPerSec (x1000)
- 6. Success comes at a cost 6 • Increased responsibility and availability • Exotic problems • Strain to operate and on-board
- 7. 7 • Amount of requests growing exponentially • Repetitive manual work • Error prone • Managing metadata • Longer waiting time for clients Operational concerns
- 8. 8 Automate as much as possible • Store and present all meta-data • Manage requests • Automated request execution • Less operational work!
- 9. Focus on value - open the platform to everyone@ING 9 Business Users •Topic design? •Access control? •Encryption decision? •Lifecycle management?
- 10. PROS vs CONS 10 • Self service • Business in control • Transparency • Custom built to fit in the ING landscape • Less operations! • Need to build it ourselves • Need to Protect the core Kafka components • Need to fool-proof the platform
- 11. Lets build Kafka self-service for ING 11 • ING look and feel • Single sign-on • CLI • One place to explore the metadata • Give topic ownership to the…owners!
- 12. Demo 12
- 13. 13 • Schema Management • Ownership • Unified language • Reuse of data • Data lineage Next Steps
- 14. Can we handle the flood? 14 •Multiple clusters? •Cloud, hybrid, global? •All Markets? Empower our customers to focus on their business in all environments, across clusters and in all markets
- 15. Questions?
- 16. Thank you
- 17. Follow us to stay a step ahead ING.com YouTube.com/ING SlideShare.net/ING@ING_News LinkedIn.com/company/ING Flickr.com/INGGroupFacebook.com/ING