In this presentation you can see some of the design choices that we made developing the Business Event Bus within a Rabobank context. More information on the meeting can be found here: https://www.meetup.com/Kafka-Meetup-Utrecht/events/232443728/

1. The Business Event Bus
Joris Meijer
Dizzit

2. Dizzit &A little about me
-Married, father of Imke, Felix
-Living in Voorschoten
-+/- 8y exp. in SME (php, matlab),
> 4 y exp. in large enterprise (Java)
-Joined Dizzit december 2015
-Scrum master / developer
-Interested in Scrum, Kafka, Docker,
Rancher, much more

3. Dizzit &Our starting point
-Enterprise context
-BEB will become core architecture
-100s of connecting applications
-Versioned data contracts
-Movement towards devops
-Infrastructure independence

4. Dizzit &Application classification
3-2-3
3-2-3

5. Dizzit &Application classification
3-2-3
3-2-3availability

6. Dizzit &Application classification
3-2-3
3-2-3
integrity

7. Dizzit &Application classification
3-2-3
3-2-3confidentiality

8. Dizzit &Design choices
3-2-3
-Multi datacenter setup
-Configuration provider
-Schema registry
-Secure setup
-BEB client library
-Dockerize all components

9. Dizzit &Multi datacenter setup
3-2-3
Best Boxtel
- Active-active setup
-Datacenters in Best and Boxtel
-Using Mirror Maker
- Challenge: prevent feedback

10. Dizzit &Multi datacenter setup
Active-active setup
Challenge: prevent infinite loop
A
Message x
B
sync
sync
-Prevent feedback

11. Dizzit &Multi datacenter setup
Active-active setup
Challenge: prevent infinite loop
A
Message x
B
sync
sync
Message x
-Prevent feedback

12. Dizzit &Multi datacenter setup
Active-active setup
Challenge: prevent infinite loop
A
Message x
B
sync
sync
Message x
Message x
-Prevent feedback

13. Dizzit &Multi datacenter setup
Custom BEB header
Magic Major Minor
Info
size
Info
ORIGINAL
MESSAGE
0B EB 00 01 04 00 00 00 01
-Custom BEB header

14. Dizzit &Multi datacenter setup
-Custom messagehandler

15. Dizzit &Configuration provider
-Find a cluster to connect to
-Dynamic configuration
-Interruptions (maintenance)
-Spread the load
Client
Configuration provider
1
2
Cluster X
3

16. Dizzit &Configuration provider
-Find a cluster to connect to
-http GET:
http://cp.rabobank.nl?clientTopic=distribution-outboundpushmessage
- Result:

17. Dizzit &Configuration provider
-Spread the load
-http GET:
http://cp.rabobank.nl?clientTopic=distribution-outboundsmsmessage
- Result:

18. Dizzit &Configuration provider
-Dynamic configuration
-Allow maintenance
-http POST:
http://cp.rabobank.nl/admin/setswitch?switch=clusterA
- result:

19. Dizzit &BEB client library
-Simplify setup for connecting clients
-Obtain and refresh configuration from
configuration provider
-Support multiple strategies

20. Dizzit &BEB client library
3-2-3
Configuration
Provider
Client
Kafka Java Library
BEB
SerDe
BEB Client Library
Producer
factory
Consumer
factory

21. Dizzit &BEB client library
-Simplify setup for connecting clients
-Obtain configuration from configuration
provider

22. Dizzit &BEB client library
-Support multiple strategies

23. Dizzit &BEB client library
-Support multiple strategies
AT_MOST_ONCE vs AT_LEAST_ONCE

24. Dizzit &BEB client library
-Support multiple strategies
AT_MOST_ONCE vs AT_LEAST_ONCE
- Delivery not guaranteed
- No duplication
- Fast
- Delivery guaranteed
- Duplications can occur
- Slightly slower

25. Dizzit &BEB client library
-Support multiple strategies

26. Dizzit &BEB client library
-Support multiple strategies
-Custom retry policy

27. Dizzit &BEB client library
-Reload configuration from configuration
provider
-Maintenance window
Client
Configuration provider
1
2
Cluster X
3

28. Dizzit &Schema registry
-Uses Apache avro
-Contract between producers and
consumers
-Backwards compatible
-More control over schema
governance
-POST only allowed by
BEB devops team

29. Dizzit &Schema registry
-Schema ID is present in every
message
-Call to schema registry is made prior
to (de)serialization
-Examples (DEMO)

30. Dizzit &Topic catalogue

31. Dizzit &Secure setup
-Authentication & authorization
- Mutual SSL
- ACLs per topic, per environment

32. Dizzit &Dockerize all components
-Docker image per component
-Based on Confluent platform
-Externalized properties
-Docker registry

33. Dizzit &Dockerize all components
-Docker image per component
-Based on Confluent platform
-Externalized properties
-Docker registry
-Rancher for deployment

34. Dizzit &Enough talking
http://static.tumblr.com/52679116680c197db232c3116ce9853
7/y07rdrp/1XJmt370e/tumblr_static_demotimeheader2.png

35. Dizzit &DEMO
Account
Entry
Producer
Alert
Producer
Dashboard
Backend
Dashboard
Frontend
Account EntriesAlerts

36. Dizzit &Future
-BEB in the cloud
-Configurable mirror maker
-Dynamic scaling
-Automatic failover
-Easier topic management
-EXACTLY_ONCE strategy

37. Dizzit &Questions?