In the past few years, many businesses started do understand the potential of real-time data analytics. And many of those invested time, energy and finances to make it happen, with weaker outcomes than expected. Reasons are few for this: too ambitious plans by leadership regarding leveraging data, not enough discipline defining goals and MVP for initial use cases, a plethora of tools and vendors available who claim that can solve all the problems, etc. So, how can we get the most value with reasonable costs out of fast (real-time) data? We will try to answer this question and give actionable advice.

1. Reality and Traps of Real-time
Data Engineering
Miloš Solujić

2. Let me introduce myself
Programming since 1990
IoT enthusiast and
contributor since 2007
DS and Data Engineering since
2012
Consultant and advisor

3. And you...
❏ Data Scientist
❏ Data Engineer
❏ Neither

4. What will we talk about
❏ Why data and ML
❏ Why real-time
❏ History of approaches
❏ Available options ATM
❏ How Kafka can help
❏ Conclusions, Q&A

5. ❏Why data and ML?
Inform our decisions?

8. ❏Why real-time
☑ The cost went way down
☑ Tooling drastically improved
☑ It can change the game
☑ It’s fun

9. ❏Kinds of fast data
☑ Stream processing
☑ Complex Event Processing (CEP)
☑ Real-time processing

11. ❏History of approaches

12. ❏History of approaches
Lambda architecture

13. ❏History of approaches
Kappa architecture

14. ❏Available options

15. ★ Operations
★ Topology updates tricky
★ Simply: other tools got
lessons from it and took over

16. ★ Latency, micro batches
★ State management fragility
★ Expensive (in memory)
★ Window criteria limitations
★ Back pressure handling

17. ★ Configuration files store
jobs
★ Serialization and
transformation order
★ Yarn is mandatory for
cluster. Big overkill

18. ★ Horizontal scaling
★ Job management in production
★ Support for interactive data
analysis

19. Anything more
approachable?

20. Isn’t Kafka
just
dump
pipe?

21. What Kafka offers?
☑ Reliable message broker
☑ Data connectors
☑ Support for stream processing
☑ KSQL

22. Why Kafka streaming
☑ Elastic, highly scalable,
fault-tolerant
☑ Deploy to containers, VMs,
bare metal, cloud
☑ Equally viable for small,
medium, & large use cases
☑ Fully integrated with Kafka
security

23. Why Kafka streaming
☑ Write standard Java and Scala
applications
☑ Exactly-once processing
semantics
☑ No separate processing cluster
required
☑ Develop on Mac, Linux, Windows

24. Sample composition

26. What we talked about
☑Why data and ML
☑Why real-time
☑History of approaches
☑Available options ATM
☑How Kafka can help

27. Questions?

28. @msolujic