The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

1. Cloud-Based
Event Stream
Processing
Architectures and
Patterns with
Apache Kafka
KAFKA SUMMIT EUROPE 2021
Israel Ekpo

2. @IzzyAcademy
IzzyAcademy.com

3. Outline
Tips for success
Patterns for the Cloud
Resources
Follow up and Next Steps

4. Tips for Success

5. Fundamentals
•People
•Process
•Product

6. Restaurant
Experience

7. Where, Why, How, What, Who

8. Home Construction

9. Where, Why, How, What, Who

10. Where, Why, How, What, Who

11. Cloud-Based Event
Stream Processing with
Apache Kafka

12. • Brokers
• Schema Registry
• Kafka Streams, KSQLDB
• Kafka Connect Ecosystem
• Kafka Client Ecosystem

13. • Operations, Data Engineers, Data Scientists, Solutions Architects
• System Architecture, DevOps Pipeline, Agile Methodologies
• Software Product or Experience

14. Horseless-
Carriage
Syndrome

15. •Architectural Complexity
•Operating Complexity
•Lack of Trust and Desire for Control

16. Patterns for the Cloud

17. Deployment Options

18. Deployment Options
Fully-
Managed
Self-
Managed

19. Fully-Managed vs Self-Managed

20. Fully-Managed on Azure
Confluent Cloud Integration on Azure
Azure Event Hubs

21. Self-Managed On Azure
Confluent Platform on Virtual Machines
Confluent Platform on Kubernetes (via Confluent Operator)
Apache Kafka on Azure HDInsight

22. Hybrid
Scenarios
Linking
On-Premise to Cloud
Environments

23. Hybrid Scenarios

24. Hybrid
Scenarios
Mixing Fully-Managed
with Self-Managed

25. Hybrid Scenarios for Flexibility
Dynamic Capacity Needs (Kafka Streams, KSQLDB, etc)
Some Components not available in Fully-Managed Offering
Custom Scenarios

26. Stretch Clusters

27. Tiered Storage

28. Orchestration vs Choreography

29. Event Stream Meshes

30. Integration
Options

31. Integration Options via
Connect Framework
Source Connectors
Sink Connectors
https://www.confluent.io/hub/#azure

32. Summary
Event Stream Meshes via KSQLDB
Orchestration vs Choreography
Dynamic Capacity
Stretch Clusters
Tiered Storage

33. Summary
Mindset Change
People
Deployment Options
Awareness of Environment and Integration Options

34. Conclusion
Get the right team in place
Understand your environment and what is possible
Good process in place
Clear goals and objectives
Reach out

35. izzyacademy.com/kafka-in-the-cloud

36. Thank you

37. @IzzyAcademy
IzzyAcademy.com