Kafka in Space | Kafka Summit London 2022
•
0 likes•123 views
Polar stations offer a unique and strategic position on earth when dealing with satellite missions. Getting data from network isolated scientific stations poses a series of unique challenges. The only way to ensure a trusted data transit is through secured geostationary satellites 36 thousand kilometers above our heads. Sending and deduplicating high value information through a non conventional worldwide network with limited bandwidth and high latency brings interesting challenges. We tried to solve this challenge leveraging dynamic routing with BGP, the protocol that rules the internet. However BGP and Kafka alone could not resolve the whole problem, especially when keeping in mind the financial impact of secured satellite telecommunications for ensuring data transit. As a global Kafka stretch cluster was clearly out of question, we needed to be more creative and answer specific questions.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kafka in Space | Kafka Summit London 2022
Similar to Kafka in Space | Kafka Summit London 2022 (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
Kafka in Space | Kafka Summit London 2022
- 2. 25/04/2022 Kafka in space 2 Who are we?
- 3. Speakers 3 25/04/2022 Kafka in space Michael DEBOUVER Senior cloud architect Airbus Olivier LAGARDE Senior cloud expert Airbus Nils BOUCHARDON Senior solutions architect Confluent
- 4. 4 25/04/2022 Kafka in space
- 5. Airbus: A European company 5 AIRBUS Commercial Aircraft Helicopter Defence and Space 25/04/2022 Kafka in space
- 6. Context: Space missions 6 25/04/2022 Kafka in space Telecommunications Earth observation Images Radar Altimetry Cloud Infra for massive processing Navigation
- 7. Our mission at Airbus Defence and Space 7 DESIGN INTEGRATE DELIVER INFORMATION SYSTEM 25/04/2022 Kafka in space
- 8. 25/04/2022 Kafka in space 8 Let’s look at a use case!
- 9. Let’s take a use case 9 In case of a natural disaster, observation space systems can: • Map the area • Guide the first responders • Evaluate the disaster • Analyze the disaster evolution è Data needs to be ”fresh”, human lives are at stake, time is essential. 25/04/2022 Kafka in space A tsunami hits somewhere on earth
- 10. 25/04/2022 Kafka in space 10 What do we need from our space observation system?
- 11. Observation satellites: characteristics 11 25/04/2022 Kafka in space Earth observation satellites are on polar orbits: • Low earth orbit (≈ 700 km) • Consistently covers the entire globe • Frequently goes over the poles (<1h)
- 12. Polar stations 12 25/04/2022 Kafka in space Some stations have dedicated submarine cables Svalbard (Norway) Other don’t Troll Satellite Station (Norway) source: https://submarinecablemap.com
- 13. Data exchanges: context 13 Kafka in space 25/04/2022
- 14. 25/04/2022 Kafka in space 14 What are the requirements for exchanging critical data?
- 15. Data exchanges: objectives Kafka in space Resilient Flexible High performance Opened Secured Efficient 15 25/04/2022
- 16. 25/04/2022 Kafka in space 16 What are the characteristics of our environment?
- 17. WAN 100 ms RTT 1 Gbps WAN 600 ms RTT 10 Mbps Satellite communications Network overview: WAN characteristics 17 25/04/2022 Kafka in space Kubernetes Cluster Polar station Central 1 Kubernetes Cluster Central 2 Kubernetes Cluster No Internet Access
- 18. Border Gateway Protocol over Site to site IPSec VPN Network overview: network configuration 18 25/04/2022 Kafka in space Kubernetes Cluster Polar station Central 1 Kubernetes Cluster Central 2 Kubernetes Cluster AS1 AS2 AS3
- 19. 25/04/2022 Kafka in space 19 How can we optimize network usage?
- 20. Data exchanges: reminder 20 Kafka in space 25/04/2022
- 21. Data exchanges: network optimization 21 Kafka in space 25/04/2022
- 22. 25/04/2022 Kafka in space 22 How do we use the Kafka ecosystem to exchange data?
- 23. Overall Architecture 23 25/04/2022 Kafka in space Producer Polar station Central 1 Confluent Operator Confluent Replicator Central 2 Consumer Control Center Kafka Brokers Kafka Brokers Control Center Confluent Operator Confluent Replicator Consumer Kafka Brokers Control Center Confluent Operator
- 24. Data exchanges: nominal case 24 25/04/2022 Kafka in space Kafka Brokers Producer Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer
- 25. 25/04/2022 Kafka in space 25 It’s time for a demo!
- 26. 25/04/2022 Kafka in space 26 What if we lose the connectivity between the polar station and central1?
- 27. Connection lost between the station and Central 1 27 25/04/2022 Kafka in space Kafka Brokers Producer Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer
- 28. 28 25/04/2022 Kafka in space Kafka Brokers Producer Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer Connection lost between the station and Central 1
- 29. 25/04/2022 Kafka in space 29 It’s time for a demo!
- 30. Connection lost between the station and Central 1 30 25/04/2022 Kafka in space Kafka Brokers Producer Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer Action: None Resolution: BGP Time to resolve: <1min
- 31. 25/04/2022 Kafka in space 31 What if central 1 is isolated?
- 32. Isolation of Central 1 32 25/04/2022 Kafka in space Kafka Brokers Producer Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer
- 33. 25/04/2022 Kafka in space 33 How can we dynamically update the replicator configuration?
- 34. Step1: Heartbeat Monitoring 34 25/04/2022 Kafka in space Kafka Brokers Polar station Kafka Brokers Central 1 Confluent Replicator Kafka Brokers Central 2 Heartbeat Producer Heartbeat station à central 1 Heartbeat station à central 2 Heartbeat station à central 1 à central 2 Confluent Replicator
- 35. Step1: Heartbeat Monitoring 35 25/04/2022 Kafka in space Kafka Brokers Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer Heartbeat Producer 1,2,3,4 1,2,3,4 1,2,3,4 1,2,3,4
- 36. Step1: Heartbeat Monitoring 36 25/04/2022 Kafka in space Kafka Brokers Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer Heartbeat Producer 1,2,3,4,5 1,2,3,4 1,2,3,4 1,2,3,4,5
- 37. Step2: Replicator Controller on Central 2 37 25/04/2022 Kafka in space Control Heartbeat delta Replicator Controller Nominal Mode: • Data source: Central 1 Failover Mode: • Data from Polar Station • Switch Replicator configuration • Reset offsets
- 38. Replicator Controller - Components 38 25/04/2022 Kafka in space Producer Polar station Central 1 Confluent Operator Confluent Replicator Central 2 Consumer Heart Beat Kafka Brokers Kafka Brokers Confluent Operator Confluent Replicator Consumer Kafka Brokers Confluent Operator Replicator Controller
- 39. Summary 39 25/04/2022 Kafka in space Kafka Brokers Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Replicator Controller Producer Heartbeat Producer Consumer
- 40. 25/04/2022 Kafka in space 40 It’s time for a demo!
- 41. Failback 41 25/04/2022 Kafka in space Kafka Brokers Producer Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer Action: Automated Resolution: Controller Time to resolve: <2min Heartbeat Producer
- 42. 25/04/2022 Kafka in space 42 What happens when network returns to normal?
- 43. Failback 43 25/04/2022 Kafka in space Kafka Brokers Producer Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer Heartbeat Producer
- 44. 25/04/2022 Kafka in space 44 It’s time for a demo!
- 45. Failback 45 25/04/2022 Kafka in space Kafka Brokers Producer Polar station Kafka Brokers Central 1 Confluent Replicator Consumer Kafka Brokers Central 2 Confluent Replicator Consumer Action: Automated Resolution: BGP + Controller Time to resolve: <1min Heartbeat Producer
- 46. Conclusion 46 25/04/2022 Kafka in space Resilient Flexible Opened Secured Efficient No data loss Highly available Kafka deployed with TLS and authentication High throughput communications Scalable Reversible Low latency Automatic recovery Bandwidth optimization Modular approach Open Source based High performance
- 47. Way forward / Enhancements 47 25/04/2022 Kafka in space • Manage Replicator configuration through CRDs • Optimize duration of Failback / Failover • Explore Cluster Linking capabilities to replace Replicator • Smarter offset reset strategy
- 48. 48 25/04/2022 Kafka in space Questions?