"In this session, we have a look at a real-world IoT project in which hundreds of residential building complexes are equipped with thousands of sensors and actuators that communicate via Kafka to an optimization system to reduce energy consumption and eventually help to protect our planet. MQTT is a commonly used protocol in the IIoT world, especially in conjunction with the widely used Sparkplug standard. With the available MQTT source and sink connectors for Kafka Connect, one would think integrating such IoT systems with Kafka should be straightforward. But is it? We will learn the core concepts of MQTT and the Sparkplug protocol, Kafka Connect, and Confluent’s MQTT Source and Sink connectors. Having the basics covered, we will follow a message from an IoT device to Kafka and back. We dive deeper to identify issues and see for example how to guarantee message ordering in multi-partitioned topics with a custom partitioner or single message transform (SMT). If you want to learn about the MQTT and Sparkplug protocol, how to connect Sparkplug-based IoT devices to Kafka, or are just curious about our IoT use case then this session is perfect for you!"

1. Integrating Sparkplug IoT EoN Nodes with
Kafka - A Real-world Example
Yves Kurz – PAUL Tech AG

2. Outline
16.05.23 3
What does PAUL do? 3
IoT Architecture @ PAUL 9
MQTT Primer 12
Introduction to Sparkplug 13
Connecting IoT and Kafka 15
Do‘s and Don‘ts 24

3. 4
In 2023 PAUL will save ~ 21‘500 Tons of CO₂!
To compensate that amount of CO₂ it would need an area of 10’000 sqm of forest or…
… ~80% the size of Airport Heathrow! … ~3x the size of Central Park! … ~4x the size of Disneyland California!

4. What does PAUL do?
PAUL implements the adaptive hydraulic balancing in drinking water systems and heating.
16.05.23 5

5. PAUL
Smart Valve
PAUL
Gateway
PAUL
Temperature Sensor
PAUL
Energy Valve
What does PAUL do?
PAUL implements the adaptive hydraulic balancing in drinking water systems and heating.
16.05.23 6

6. The optimization through PAUL's AI and its advantages
AI technology analyzes the current state of the entire system and optimizes it
Consumption
Behaviour
Vacancy
Weather
Sun side
orientation
Day time /
Season
Combines billions of data
points from other houses.
PAUL AI
ADVANTAGES
Up to 40% energy savings!
Optimization of ESG ratings!
More attractive interest rates
for refinancing!
Sizing of new heating systems
based on consumption data
instead of assumptions!
Perfect preparation for the use
of alternative heat generators!
16.05.23 7

7. PAUL Tech AG in Numbers
16.05.23 8
4 Offices: Mannheim, Nürnberg, Jacksonville, Bern
>130 Employees
2017 Year of Establishment
>150‘000 Residential Units
>500‘000‘000 Data Points
https://www.paul.tech/en/career

8. PAUL’s Legacy Architecture 2017-2022
16.05.23 9
l Traditional Industrial Automation Architecture
l Expensive
l Does not scale
l Hard to maintain
Input Output
PLC
SCADA/Supervisory
Reporting
AWS
Field
Level

9. New Architecture 2022-
16.05.23 10
AWS
Field
Level
Control Center &
Dashboarding Services
Kafka
IoT Services
Kafka Connect
MQTT Broker
l Concept and architecture validation from Oct ‘21 –
March’22
l Implementation May ’22-
l In productive use since Dec ‘22
l No PLC and SCADA system
l Router is Sparkplug EoN node
l From Polling to Report by Exception

10. Covered in this Session
16.05.23 11
AWS
Field
Level
Control Center &
Dashboarding Services
Kafka
IoT Services
Kafka Connect
MQTT Broker
l MQTT - Transport Protocol between Edge Nodes
l Sparkplug – MQTT Topic and Payload Definition
l Kafka Connect – Bridging MQTT and Kafka

11. MQTT Primer
16.05.23 12
l Message Queuing Telemetry Transport
l Lightweight open protocol
l Used in (I)IoT in a constrained environment
l Low Bandwidth, High Latency
l Low Power Equipment
l Publish-Subscriber Model with Central Broker
Topics & Subscriptions
l Topics names form a hierarchical tree
l Messages published on a topic
l Subscriptions on a specific topic or with wildcards to a topic
sub-tree
l MQTT5: Shared Subscription
Brocker
Client 1
Client 2
Client 3
Messages
l Quality of Service – Message Delivery guarantees
l QoS 0: At most once
l QoS 1: At least once
l QoS 2: Exactly once
l Retained Messages
l LWT – Last Will Telegram
✉
→
m
y/t1
my/t1 → ✉
my/t2 ← ✉
✉
←
my/t1
✉
←
my/t2
https://mqtt.org

12. Sparkplug
MQTT + Sparkplug = 'Plug & Play' IIoT
16.05.23 13
l Specification originated by Cirrus Link Solutions now
maintained by the Eclipse Foundation
l Designed to allow interoperability between IIoT devices
l Allows realtime communication between devices in the
system
l Ensures devices and applications are aware of state and
configuration
Extends MQTT
l Defines a Topic Namespace
l Specifies Message payload content and serialization formats
l Sparkplug A: JSON
l Sparkplug B: Protobuf
l Defines device state management
l Broker failover
https://sparkplug.eclipse.org

13. Sparkplug
Architecture
16.05.23 14

14. Sparkplug
MQTT Topic Namespace
16.05.23 15
Message Format
l spAv1.0: JSON
l spBv1.0: Protobuf
namespace/group_id/message_type/edge_node_id/[device_id]
Type of Message
l NBIRTH: Sent by edge node when it gets online.
l NDEATH: Sent by broker (LWT) when an edge node goes
offline.
l DBIRTH, DDEATH: Similar to NBIRTH, NDEATH but sent by
edge node when a connected device gets online/offline
l NDATA: Sent by edge node when the state of the edge node
changes.
l DDATA: Sent by edge node when the state of a device
changes
l NCMD: Contains a command to an edge node
l DCMD: Contains a command to a device
Group ID allows dividing the
namespace into arbitrary
groups. E.g by region.
Unique identifier of an edge
node
Unique (within edge node)
identifier of a device

15. Sparkplug
Payload content
16.05.23 16
Payload Fields
l Timestamp
l Array of Metrics
l Name
l Data type
l Value
l Timestamp
l Optional:
l Alias
l Historical, Transient, Null
l Properties (property set)
l Quality Code,…
l Metadata (seq, filename, filetype md5sum, description
…)
l Optional
l UUID
l Body (byte data)

16. Connecting IoT with Kafka
16.05.23 17
MQTT Broker
Topics to which
Edge Nodes
publish
Topics to which
Edge Nodes
subscribe
Kafka
iot-mqtt-source
iot-mqtt-sink
MQTT Source Connector
Kafka Connect
MQTT Sink Connector
IoT Services

17. Connecting IoT with Kafka
MQTT Source Connector
16.05.23 18
l We use Confluent’s MQTT Source Connector
l Copy messages from topics written by Edge Nodes to
Kafka iot-mqtt-source topic
l /spBv1.0/+/NBIRTH/#
l /spBv1.0/+/DBIRTH/#
l /spBv1.0/+/NDEATH/#
l /spBv1.0/+/DDEATH/#
l /spBv1.0/+/NDATA/#
l /spBv1.0/+/DDATA/#
l Don’t use Clean Session, to not loose messages when
connector temporarely goes offline
l Kafka Message
l Key: MQTT Topic name to which payload was published
l Data: MQTT Raw message (Protobuf, Sparkplug b)
https://docs.confluent.io/kafka-connectors/mqtt/current/mqtt-source-connector
l What about the ordering of messages in Kafkas partitions?

18. Connecting IoT with Kafka
MQTT Source Connector – With Custom Partitioner
16.05.23 19
l We have a requirement that message ordering is guaranteed per Edge Node
l Default Partitioner is hashing message key to determine the partition
l Custom partitioner guarantee that all messages from one Edge Node are store in the same partition
namespace/group_id/message_type/edge_node_id/[device_id]
Partition number: #(full_key) MOD num_partition
namespace/group_id/message_type/edge_node_id/[device_id]
Partition number: #(edge_node_id) MOD num_partition

19. Custom SMT
16.05.23 20
No easy way to access the # partitions of
destination Kafka topic within an SMT

20. Better Solution - Custom Partitioner
16.05.23 21
In a partitioner we have access to the a Cluster
object, this gives us access to the # of partitions

21. Connecting IoT with Kafka
MQTT Sink Connector
16.05.23 22
l We use Confluent’s MQTT Sink Connector
l Source Topic: iot-mqtt-sink
l Kafka Message
l Key: MQTT Topic name to which message shall be
published
l Data: MQTT Raw message (Protobuf, Sparkplug b)
l By default the connector publishes into the MQTT topic
that has the same name as the Kafka topic.
l Change the behaviour by using ExtractTopic SMT
provided by Confluent, to publish to the correct MQTT
topic
https://docs.confluent.io/kafka-connectors/mqtt/current/mqtt-sink-connector/

22. Do’s & Don’t’s
16.05.23 23
l Select the right architecture depending on your needs. Fat vs
Thin Edge Devices. Pushing Kafka to the Edge may not make
sense.
l When working with Sparkplug, start with JSON, when the
system is more mature, switch to Protobuf
l Use good tools. For MQTT use MQTT Explorer, and build from
source to get Protobuf deserializer support.
l Write defensive code in SMTs/Partitioners
l Test your system, esp. for performance, and outages of
brokers and connectors.
l Don’t invent your own IoT protocol
l Don’t use SMTs to do the work of a partitioner
l Try avoiding using custom SMTs and/or partitioners if you
plan to use cloud hosted Kafka Connect

23. 16.05.23 24
Questions

24. Thank You
©2023. ALL RIGHTS RESERVED.
Principal Engineer
l Email: yves.kurz@paul.tech
l Web: https://paul.tech/
l Twitter: @yveskurz
l Sources: https://github.com/yveskurz/kafka-summit-2023-
sparkplug-kafka
Yves Kurz