Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime. Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka


Published on

Slides of our talk at: Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka.

By Tim van Baarsen & Marcos Maia

Nowadays our customers are expecting real time interactions with investments products and services we deliver at ING Bank especially because financial market fluctuations can have a direct impact on the investments performance of our customers.

In this talk, we present how we deal with the massive stream of price updates on stocks and how we leverage many features of Apache Kafka and Spring Kafka to improve and simplify our solutions. From simple producer / consumer up to advanced usage of streams we strongly rely on Kafka to solve our functional and non functional requirements delivering high quality, real time stream processing software that is reliable, precise and extremely valuable to our customers.

Main challenges of our project One of the main challenges of this project was to deal with a massive continuous stream of stock price updates and matching them against thousands of alerts created by our customers and still keep up with sending real time alerts to them.

Published in: Technology
  • @Wayne Heth thanks for your message and reaching out. Unfortunately we can't share the complete source-code with you at this point in time. Both Marcos and myself are preparing a couple of blog post on the topics we presented at in Barcelona including coding examples. We will publish the blog posts in the near future at: For us it will be interesting to know if your looking to learn more about specific topics so we can prioritize our blog posts. Feel free to reach out on twitter: @TimvanBaarsen or @thegroo.
    Are you sure you want to  Yes  No
    Your message goes here
  • Excellent talk! Is the code for the demo available for download anywhere?
    Are you sure you want to  Yes  No
    Your message goes here Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka

  1. 1. Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka Marcos Maia & Tim van Baarsen SPRING.IO BARCELONA – MAY 2019
  2. 2. Introduction Tim van Baarsen Software Engineer at ING @TimvanBaarsen Marcos Maia Software Engineer at ING @thegroo
  3. 3. ING is active in more than 40 counties 3
  4. 4. Agenda 4 • Use case @ ING • Mifid II • Mobile Investments App • Overall Architecture • Demo • Metrics / Monitoring • Kafka streams • Local development / Code walk through • Lessons learned • Questions
  5. 5. Disclaimer 5 Both the code and high level architecture we show here today is not the actual product we developed within ING! This example is inspired by our work
  6. 6. Domain Lingo 6 Tick Price update on stock=
  7. 7. Mifid II: “Inform customers when a stock drops -10% in price at the end of the day!” 7 Mifid II Regulations ! At the end of the day? "No, in real time!Markets in Financial Instruments Directive II = Regulatory reform financial markets European Union came into effect on January 3, 2018
  8. 8. Screens of our mobile Investments app 8 price going up! specific price -10% drop in price Recurring alerts
  9. 9. 2.000 – 12.000 stock updates a second! 9 ! "
  10. 10. The perfect “match” 10 ❤
  11. 11. Overall architecture 11 Alert service A P I Matching Service Stocks update receiver Internet “SMS Platform” A P I Receive stock price updates ~ 12.000/sec Publish stock price updates to Kafka Create alerts Delete alerts List alerts Customer receives investment alert Consume updates & match against alerts created by customers In case of a match send a sms to customer Customer alert(s) stored in database Notification Service Schema Registry Zookeeper “Push Platform” A P I !
  12. 12. Quick Demo 12
  13. 13. Monitoring - Metrics 13 • Micrometer • Think SLF4J, but for metrics (Jon Schneider) • Application metrics • Out of the box metrics • Business metrics • Vendor Neutral
  14. 14. Monitoring – Real-time? 14
  15. 15. Kafka Streams Join 15 Matching Service raw-stock-quote stock-quote-tick stock-quote-meta log-compaction Raw-tick stock-quote-ticker RocksDB Key value ING Raw-tick GOOGL Raw-tick AAPL Raw-tick Key value ING Meta-data GOOGL Meta-data AAPL Meta-data GlobalKTableKStream
  16. 16. Local development 16 • Docker • Kafka • Zookeeper • Kafka manager • All our Spring Boot applications • Producers for development • Custom Kafka command line producer • Lab project
  17. 17. Spring Kafka 17 • Producing: • ProducerFactory • KafkaTemplate • “The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances.” • Consuming • Kafka consumers are not thread safe • @KafkaListener • MessageListener (many different interfaces) • Access to ConsumerRecord • Access to manual control to commit the offset • Scale up multiple consumer (threads) within the application
  18. 18. Spring Kafka 18 • Integration tests • Junit rule: EmbeddedKafkaRule • @EmbeddedKafka • Metrics • Kafka consumer metrics out of the box (Micrometer)
  19. 19. Lessons learned 19 • Use Spring Kafka after you understand the Kafka APIs • Error handling • (De)serialization -> Apache Avro • Think about # of partitions per topic upfront if possible (routing) • Can your consumers keep up? Can you scale up? • Monitor your topics / streams in production • Based on metrics -> Micrometer
  20. 20. Lessons learned 20 • Kafka clients are backward compatible with Kafka brokers • Kafka Streams • KTables • GlobalKTables • Interactive Queries • RocksDB build for ssd • Describe • processing.guarantee=exactly_once • Kafka topology • Plan ahead • small & multiple topologies
  21. 21. Questions 21 !
  22. 22. Contact Tim van Baarsen Software Engineer @ ING @TimvanBaarsen Marcos Maia Software Engineer @ ING @thegroo