Life doesn't happen in batch mode which is why application engineers and data architects need to closely cooperate to get the best out of streaming platforms like Apache Kafka and NoSQL data stores such as MongoDB. This session explores ways and means to integrate both worlds in a streaming fashion.

1. @
#MDBlocal
Hans-Peter Grahsl, NETCONOMY
Streaming Data on the Shoulders of Giants
hpgrahsl
LONDON

2. Speed & Agility
Among Top Tech Risks
For businesses to stay
relevant they must
deliver value at a
breakneck pace and
be constantly seeking
new sources of value…

3. #MDBLocal
Managing,
Processing &
Analyzing Data
We all use Data
to unlock insights
and drive value
mongodb.com/local/london

4. #MDBLocal
Diminishing Value of Data
mongodb.com/local/london
near
real time
seconds minutes hours days months
VALUEofdatatodecision-making
Traditional “batch” business intelligenceTime critical decisions
actionable reactive historical
preventive/predictive
Source: Perishable Insights, Mike Gualtieri, Forrester

5. #MDBLocal
Historic ETL can be very painful
• batch-driven
• brittle & error prone
• (too) late answers
Speed & Agility
Antipattern
mongodb.com/local/london

6. #MDBLocal
Alleviate ETL Pain with Streaming
Speed & Agility
Enabler
mongodb.com/local/london
• event-centric & stream-oriented
• loosely coupled fabric
• quick insights & fast answers

7. #MDBLocal
Streams Processors
Connected Apps
Architecture of a Modern Data Platform?
mongodb.com/local/london
Streaming Data Fabric
Data Sources
Connected Apps
Data Sources

8. #MDBLocal
On the Shoulders of Giants
mongodb.com/local/london
Kafka®MongoDB®

9. #MDBLocal
Modern Database
mongodb.com/local/london

10. #MDBLocal
Modern Database
Document Model
Run Anywhere
Distributed & Scalable
Resilient & Performant
mongodb.com/local/london

11. Apache Kafka®
Minimum Viable Introduction

12. #MDBLocal
Streaming Platform
mongodb.com/local/london

13. #MDBLocal
Streaming Platform
ü distributed
ü horizontally scalable
ü highly fault-tolerant
mongodb.com/local/london

14. #MDBLocal
What is Streaming?
“… a type of data processing
that is designed with
infinite data sets in mind …”
– Tyler Akidau
mongodb.com/local/london

15. "…everything that happens in a company
– every customer interaction, every API request, every
database change – can be represented as
real-time stream that anything else can tap into,
process or react to."

16. "…Kafka and the whole category of stream
processing represents a fundamental
paradigm shift in how the digital part of a
company is built, how data is used, and how
applications are built. This is actually a pretty
rare thing…"
– Jay Kreps

17. #MDBLocal
KStreams
App
Data
Sources
Data
Sinks
KSQL
App
Streams API
KSQL
Consumer API
Connect API
App
Apps
App
Apps
Connect API
Producer API
mongodb.com/local/london

18. #MDBLocal
Kafka APIs in a Nutshell…
• Producer & Consumer API
à publish-subscribe scenarios
• Connect API
à streaming data integration scenarios
• Streams API & KSQL
à code or SQL-based streaming scenarios
mongodb.com/local/london

19. Kafka® Connect
What’s it about?

20. #MDBLocal
Kafka Connect Basics
ANY
sink
Connect Connect
ANY
sink
ANY à e.g. file systems, data stores, REST endpoints, …
mongodb.com/local/london

21. #MDBLocal
Kafka Connect Basics
often about data stores
Connect ConnectSOURCE SINK
mongodb.com/local/london

22. #MDBLocal
Kafka Connect Basics
or more concretely
Connect Connect
https://hub.confluent.io à many many more
mongodb.com/local/london

23. #MDBLocal
Kafka Connect Basics
or more concretely
Connect Connect
mongodb.com/local/london
https://hub.confluent.io à many many more

24. Kafka® Connect
How do connectors operate?

25. #MDBLocal
Kafka Source Connectors
Source
Connector
Converter
Serialize
S
M
T
1 … N
Single Message Transforms
for basic in-flight manipulations
…
S
M
T
mongodb.com/local/london

26. #MDBLocal
Kafka Sink Connectors
Converter
Deserialize
Sink
Connector
S
M
T
1 … N
Single Message Transforms
for basic in-flight manipulations
…
S
M
T
mongodb.com/local/london

27. Official Connector
Announced at #MDBW19

28. #MDBLocal
MongoDB Connector for Apache Kafka
mongodb.com/local/london
Map and persist events
from Kafka topics directly
to MongoDB
Publish data changes
from MongoDB into
Kafka topics

29. #MDBLocal
MongoDB
Connector for
Apache Kafka
ü officially supported
by MongoDB
ü Verified Gold certified
by Confluent
mongodb.com/local/london
Available for Download on the Confluent Hub
https://www.conflent.io/hub/mongodb/kafka-connect-mongodb

30. Use Cases
MongoDB Connector for Apache Kafka

31. #MDBLocal
Single Customer View for eCommerce
MongoDB
Sinks
Single Source
of Truth
Source
Connectors

32. #MDBLocal
Data Synchronization between Microservices
Service 1
Service N
MongoDB
Sinks
.
.
.

33. #MDBLocal
Recommendation Engine for Opinion Mining
Surveys & Polls
Data
MongoDB
Source
Change
Streams
Change
Streams
User
Recommendation
Engine

34. Demo Scenario
Let’s see it in action!

35. #MDBLocal
Producer APIdata
generation
Stream
Processor
MongoDB
Sink Connector
MongoDB
Source Connector
data serving
REST
Change
Streams
device
management
SSE
Demo Scenario
!
mongodb.com/local/london

36. #MDBlocal
Streaming Data on the
Shoulders of Giants
[NETCONOMY]
Hans-Peter Grahsl
https://www.surveymonkey.com/r/KXC6DB6

37. THANK YOU!