Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. What makes time series datastreams challenging is that they often have orders of magnitude more data than other workloads, with millions of time series datapoints being quite common. Given its ability to ingest high volumes of data, Kafka is a natural part of any data architecture handling large volumes of time series telemetry, specifically as an intermediate buffer before that data is persisted in InfluxDB for processing, analysis, and use in other applications. In this session, we will show you how you can stream time series data to your IoT application using Kafka queues and InfluxDB, drawing upon deployments done at Hulu and Wayfair that allow both to ingest 1 million metrics per second. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.

1. Al Sargent
Sr. Director, Product
InfluxData
April 2021
Stream processing IoT time
series data with Kafka &
InfluxDB
Mirek Malecha
Product Manager
Bonitoo
April 2021

2. © 2021 InfluxData. All rights reserved.
3
What is time series
data?

3. © 2021 InfluxData. All rights reserved.
4
Time series data is
‒ Sequence of data points.
‒ Successive measurements made from same source over a time
interval.
‒ Plot the points on a graph and your X axis is time.

4. © 2021 InfluxData. All rights reserved.
5
Example: Weather conditions

5. © 2021 InfluxData. All rights reserved.
6
Example: Server monitoring

6. © 2021 InfluxData. All rights reserved.
7
Example: Healthcare

7. © 2021 InfluxData. All rights reserved.
8
What is a time series
database?

8. © 2021 InfluxData. All rights reserved.
9
Characteristics of the
Data
– All Time-stamped data
– Generated in regular (Metric) and
irregular (Event) time periods
– Huge volumes of data
– Real-time and time sensitive

9. © 2021 InfluxData. All rights reserved.
10
Time Series databases:
optimized for collecting,
storing, retrieving &
processing of time
series data
Compare this to:
– Document databases
– Optimized for JSON documents
– Search databases
– Optimized for full-text searches
– Traditional relational
– Databases optimized for the tabular
storage of related data in rows &
columns

10. © 2021 InfluxData. All rights reserved.
11
Time-series use
cases

11. © 2021 InfluxData. All rights reserved.
12
IoT DevOps Monitoring Operations Analytics
Industrial devices: factories,
energy, vehicles, buildings,
agriculture, etc.
Consumer sensors: wearables,
home appliances, health
monitors
Monitor the performance of
servers, virtual machines,
containers, and applications
Track business, customer,
financial and scientific metrics
Primary Use Cases for time series data

12. Delivered an Industrial IoT data historian
that
• provides automation and control of
their oil drilling platforms
• provides edge and centralized insight
into emerging patterns
by storing and analyzing all the sensor data
derived from energy production sensors
Internet of Things (IoT)
IoT monitoring

13. © 2020 InfluxData. All rights reserved.
15
InfluxData Time Series Data Platform
Accumulate Act
Analyze
Flux joins
MySQL, Postgres, RDS, etc.
Telegraf plugins
200+ plugins
Client libraries
Python, Javascript, Java, Go...
Prometheus
scrapers
FluentD
700+ plugins
AWS, Google, Azure
InfluxDB Cloud InfluxDB
Enterprise
InfluxDB
Purpose-built Time Series Database
Realtime Data Stream Processing
Visualization & Dashboarding
Data Analysis & Anomaly Detection
Alerting & Notifications
Metering & Billing
InfluxDB
Open
Source
(OSS)
On-prem, self-managed cloud
Alerting Systems
PagerDuty
Slack
Webhooks
Grafana
Client Libraries
CLI
REST API

14. © 2021 InfluxData. All rights reserved.
16
Simple IoT Solution

15. © 2021 InfluxData. All rights reserved.
17
IoT Center
• Simple demo application
• Register, configure and monitor IoT devices
• How to integrate InfluxDB
• Write measurements
• Query database
• Manage InfluxDB
• Based on Node.js and React
• Use Kafka to write measurements
• Virtual Device - emulates real devices
• Help with links to source code - GIT
• InfluxDB Cloud - No DB installation

16. © 2021 InfluxData. All rights reserved.
18
IoT Center Functions
‒ Device Registration - register new IoT Devices
‒ Virtual Device - emulate IoT device
‒ Dashboard - show measured data from the IoT Devices

17. © 2021 InfluxData. All rights reserved.
19
IoT Devices Registration
Left menu
● Devices
Registration

18. © 2021 InfluxData. All rights reserved.
20
Virtual Device - Generate demo data
Left menu
● Virtual Device
Top screen
● Button with pencil

19. © 2021 InfluxData. All rights reserved.
21
Visualisation
Left Menu
● Dashboard
Filters
● Device
● Time

20. © 2021 InfluxData. All rights reserved.
22
IoT Center Architecture
Configuration
InfluxDB Cloud
Time Series Database
(or standalone InfluxDB)
IoT Center
IoT
Devices
InfluxDB
JS
Client
Virtual
Device
Web
Browser
Measurements (temp, humidity, pressure, GPS, …)
UI
Kafka
topic: iot
Telegraf
IoT Devices registration, Queries
kafkajs

21. © 2021 InfluxData. All rights reserved.
23
Arduino
Python
Java
.NET
Kafka
Scalable component decoupling IoT layer from DB/Analytics layer
IoT Center - virtual device
‒ KafkaJS client
IoT devices - data flow
const producer = kafka.producer()
await producer.connect()
await producer.send({
key: 'mykey',
topic: KAFKA_TOPIC,
messages: [{value: influxLineProtocolData}],
compression: CompressionTypes.GZIP,
})
IoT
Device
Kafka Telegraf
MQTT Broker
.
.
.
1
2
3

22. © 2021 InfluxData. All rights reserved.
24
Telegraf
A plugin-driven server agent collecting messages from Kafka into
InfluxDB.
1. Download telegraf (Docker, Linux, Windows, macOS, ARM, ...)
https://portal.influxdata.com/downloads/
2. Update configuration file telegraf.conf (see the next slide)
3. Start it
# telegraf --config telegraf.conf
Kafka Telegraf
IoT Devices

23. © 2021 InfluxData. All rights reserved.
25
Telegraf.conf file
Read data from Kafka
https://github.com/influxdata/telegraf/tree/master/plugins/inputs/kafka_consumer
Write to InfluxDB (replace fields by your account)
Format options
https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
‒ Values, CSV, JSON, XML, InfluxDB Line Protocol, Collectd, Dropwizard, Graphite, Grok,
Logfmt, Nagios, Prometheus, PrometheusRemoteWrite, Wavefront
[[outputs.influxdb_v2]]
urls = ["https://us-west-2-1.aws.cloud2.influxdata.com"]
token = "heRbZX2n4kc8Q_jYPddwjkv3dAZRorNQnN67pMwKs1lGgbMW8vWRjAi7VvkUitQMii2XwJM9qX3cnK4oAZDIjg=="
organization = "iot-center@bonitoo.io"
bucket = "iot_center"
[[inputs.kafka_consumer]]
brokers = ["localhost:9092"]
topics = ["iot"]
data_format = "influx"

24. © 2021 InfluxData. All rights reserved.
26
InfluxDB
FREE Account
https://cloud2.influxdata.com
‒ Data Explorer
‒ Dashboards
‒ Alerts
‒ Configuration
Kafka Telegraf
IoT Devices

25. © 2021 InfluxData. All rights reserved.
27
Demo Kafka Telegraf
Virtual Device

26. © 2021 InfluxData. All rights reserved.
28
IoT Center - Source code
https://github.com/bonitoo-io/iot-center-v2 ESP8266 and ESP32
devices
Arduino
Raspberry Pi
Python
Mobile application
Flutter/Dart

27. © 2021 InfluxData. All rights reserved.
29
Summary
‒ Time series data
‒ InfluxDB
‒ Highly scalable database
‒ Store & Analyze data
‒ CLI, API, Dashboards, Tasks, etc.
‒ Free cloud time-unlimited account
‒ Telegraf
‒ Reads Kafka messages
‒ Stores data into InfluxDB
‒ 200+ data inputs
‒ Configuration file, only 9 lines

28. © 2021 InfluxData. All rights reserved.
30
influxdata.com/cloud

29. Thank You