IoT Data Streaming: Why MQTT and Kafka are a match made in heaven

IoT Data
Streaming
Why MQTT and Kafka are a match made in heaven
Margaretha Erber & Dominik Obermaier

Insert
Picture
Copyright © by HiveMQ. All Rights Reserved.
Insert
Picture
Dominik Obermaier
• Strong background in
distributed and large scale
systems architecture
• OASIS MQTT TC Member
• Author of „The Technical
Foundations of IoT“
• Conference Speaker and
Author
• Program committee member
for German and international
IoT conferences
HiveMQ CTO & Co-Founder
Magi Erber
WELCOME
Product Manager
@HiveMQ
@dobermai
linkedin.com/in/dobermai/
@ErberMagi
linkedin.com/in/margaretha-erber/
• Conference Speaker
• Author
• Expert for cloud native
technologies and Apache
Kafka

People on the Internet
The Internet of Things is huge

Technical IoT
Challenges
Confidential and Proprietary. Copyright © by HiveMQ. All Rights Reserved.

• Enterprise IT infrastructure is not
suitable for IoT
• Massive scalability required for
millions of devices
Challenge 1 - Scalability

• End customers are used to instant user
experiences like instant messaging with
WhatsApp
• Critical systems need reliable and
instant data transfer like manufacturing
systems
• Bidirectional communication required
Challenge 2 - Instant Data Delivery required

• Customer experience for IoT apps and
devices must be excellent even when
internet connectivity is flaky
Especially for moving “devices” like cars
• Devices and apps must be easy to program and
maintain, complexity should be in the cloud not
on the device
Cloud is easier to update than physical devices
Challenge 3 - Unreliable Networks

Technologies built for
the Internet of Humans
are not suitable for the
Internet of Things
9

• (I)IoT Messaging Protocol
• Created for extreme scale and instant data
exchange
• Publish/Subscribe based architecture
• Easy on the device side, pushes all
implementation complexity to the server
• Built for machines and constrained devices
(binary, data agnostic)
• Designed for reliable communication over
unreliable channels
What Is MQTT?

Publish / Subscribe Pattern

Use Cases
13
Connected Car
IIoT /
Industry 4.0
Logistics Telecommunication
IoT Messaging
Middleware

Push Communication
MQTT
Use
Cases
Reliable Communication over unreliable
networks
Constrained Devices
Low Bandwidth and High Latency
Industrial Message Bus

• Optimized to stream data between systems and
applications in a scalable manner
• Scale-out with multiple partitions for a topic and
multiple nodes
• Perfect for inter-system communication
• inside trusted network
• with stable IP addresses and connections
and
• limited number of producers and
consumers
Apache Kafka Strengths

Apache Kafka
for IoT
How does it fit in?

Kafka brokers need to be addressed directly by the clients
Kafka does not support large amounts of topics
Kafka clients are reasonably complex and resource intensive compared to client
libraries for IoT protocols
Kafka clients require a stable TCP connection for best results
It’s unusual (and very often not even possible at all) to have tens of thousands or
even millions of clients connected to a single Kafka cluster
Kafka is missing some key IoT features
Challenges for Apache Kafka in IoT

How to use the best of
both worlds?

💛
A Love Story Made in Heaven

Fast, reliable communication
Ideal for communication over
unreliable networks
Connects millions of IoT
Devices
Optimized to stream data
between systems and
applications
Communication inside trusted
networks
Connects limited number of
communication partners
MQTT and Kafka

How to combine both worlds?
Kafka Connect MQTT Proxy Custom Bridge MQTT Broker
Extension

Kafka Connect
sensor-data
alert
Kafka Connect
MQTT Broker
MQTT
devices
Mobile Application

- Doesn’t scale well as Kafka Connect
has to act as MQTT Client
- MQTT Clients are not designed for
extremely large amounts of MQTT
messages
- Centralizes business and message
transformation logic
Kafka Connect
+ Ideal if you don’t control the
MQTT Broker or use a third party
MQTT Broker

MQTT Proxy
MQTT
devices
MQTT Proxy
Broker
Kafka Consumer

- Does not implement full MQTT ISO Standard
- Does not support features unique to MQTT
which were designed for IoT use cases
(LWT, retained messages,...)
- Client must ensure that no MQTT features
beside plain publishing are used
- Tightly coupled with Kafka, does not allow
the Pub/ Sub features of MQTT
MQTT Proxy
+ Does not require an MQTT broker
+ Usually stateless, which makes scaling
easier

MQTT Custom Bridge
Mobile Application
MQTT
devices
Custom Bridge
MQTT Broker
Broker

- Extremely hard to avoid message loss
- Developers must take care of fault tolerance
themselves
- Another system which adds complexity and
adds no value
MQTT Custom Bridge
+ Application used for transposing MQTT
to Kafka and vice versa built by
developers themselves, so no external
component

Mobile
Application
sensor-data
Stream processing
using KSQL
Other
Backend
Systems
MQTT to Kafka
Kafka to MQTT
MQTT
devices
Confluent Schema Registry
Get Schema
alert
MQTT Broker Extension

- Needs an MQTT broker that can be
extended
- Out of the Box MQTT Broker Extension to
connect with Kafka only available for
HiveMQ
MQTT Broker Extension
+ MQTT Broker uses native Kafka protocol
+ All MQTT Features can be used
+ Bi-directional producing and consumption
possible
+ High Scalability and resilience.
+ Extreme throughput. Can write hundreds of
thousands of MQTT messages per second
to Kafka
+ Ideal for aggregating MQTT topics to Kafka
Firehose Topics
+ Can write to multiple Kafka Deployments at
the same time

What are the next steps?
+

IoT Data Streaming: Why MQTT and Kafka are a match made in heaven

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to IoT Data Streaming: Why MQTT and Kafka are a match made in heaven

Similar to IoT Data Streaming: Why MQTT and Kafka are a match made in heaven (20)

More from Dominik Obermaier

More from Dominik Obermaier (18)

Recently uploaded

Recently uploaded (20)

IoT Data Streaming: Why MQTT and Kafka are a match made in heaven