Gene Hynson [InfluxData] | How We Built the MQTT Native Collector | InfluxDays 2022

InfluxData
InfluxDataInfluxData
Gene Hynson [InfluxData] | How We Built the MQTT Native Collector | InfluxDays 2022
Native MQTT Collector
Gene Hynson - Senior Software Engineer, InfluxData
The MQTT Native Collector is an all new way to natively ingest MQTT
data from your MQTT Broker into InfluxDB Cloud. This zero code
solution makes it easier than ever to monitor your IoT devices in
InfluxDB. This session provides an overview of the MQTT Native
Collector, describes when and how to use it, and explains how it works
behind the scenes. Lastly, Gene will demo his recent IoT project that
utilizes the MQTT Native Collector.
Gene Hynson
Senior Software Engineer,
InfluxData
Gene is a Senior Software Engineer at InfluxData who is passionate
about building thrilling product features for InfluxDB. Gene has 7 years
of experience as a full-stack engineer building modern, cloud-first
software at VMware, Pivotal, and Boeing. Outside of work, Gene
spends his free time skiing the slopes in the winter or playing volleyball
on the beach in the summer.
MQTT Native Collector
Agenda
1. What is the Native MQTT Collector?
2. How does Native MQTT Work?
3. How did we build Native MQTT?
4. Demo
What is Native Data
Collection?
The ability to directly ingest data from
popular data sources without the
need to do any transformation to Line
Protocol or run a Telegraf agent
6
Native Collection
7
MQTT Topic Subscriptions
• Subscribe to topics being published by
their own devices and sent through a
message broker directly to the InfluxDB
Cloud service
• No code or installing any additional
software required
MQTT Ingestion: Telegraf vs Native MQTT
How does Native Data
Collection work?
Simple 3 Step Configuration
Define Parsing Rules
• Configure Rules for finding
timestamps, measurements,
tags, and field information in
the MQTT messages
Configure Broker
• Broker URL
• Broker TCP/IP Port Number
• Broker Authentication Type
• Broker Authentication
Parameters
Configure Topic
• Topic Name
• InfluxDB Bucket
Connect to Broker
Gene Hynson [InfluxData] | How We Built the MQTT Native Collector | InfluxDays 2022
Subscribe to Topic
Gene Hynson [InfluxData] | How We Built the MQTT Native Collector | InfluxDays 2022
Define Data Parsing Rules
Gene Hynson [InfluxData] | How We Built the MQTT Native Collector | InfluxDays 2022
View the Data
Gene Hynson [InfluxData] | How We Built the MQTT Native Collector | InfluxDays 2022
Zero Code Data Ingestion
How did we build Native
Data Collection?
How did we build Native MQTT?
How did we build Native MQTT?
Apache NiFi automates the flow of data
between systems.
●Open source
●Data source agnostic
●Data type, format, & size agnostic
●End-to-end data encryption
●Guaranteed delivery
●Operates at a massive scale
Apache NiFi
How did we build Native MQTT?
Demo: Monitoring car
OBD2 data over time
Monitoring car OBD2 data over time
https://github.com/genehynson/arduino-mqtt-influxdb-obd2
Monitoring car OBD2 data over time
Arduino MKR1010 (WiFi)
HC-05 Bluetooth
Module
NEO-6M GPS
Module
Monitoring car OBD2 data over time
Arduino
#include <WiFiNINA.h>
while (WiFi.status() != WL_CONNECTED) {
WiFi.begin(ssid, pass);
delay(10000);
}
Serial.println("Connected to WiFi");
Serial.print("IP Address: ");
Serial.println(WiFi.localIP());
Arduino
#include <ArduinoMqttClient.h>
char host[] = "broker.hivemq.com";
int port = 1883;
if (mqttClient.connect(host, port)) {
Serial.print("Connected to broker");
}
Connect to WiFi Connect to MQTT broker
Monitoring car OBD2 data over time
Arduino
#include <ELMduino.h>
ELM327 elm;
elm.begin(Serial1, true, 5000);
...
float value = elm.rpm();
if (elm.nb_rx_state == ELM_SUCCESS) {
mqttClient.beginMessage("obd/rpm");
mqttClient.print(value);
mqttClient.endMessage();
}
Arduino
#include <TinyGPS++.h>
gps.encode(Serial2.read());
if (gps.location.isUpdated()) {
lat = gps.location.lat();
lng = gps.location.lng();
String json =
"{"lat":"+lat+","long":"+lng+"}";
mqttClient.beginMessage("odb/gps");
mqttClient.print(json);
mqttClient.endMessage();
}
Publish ELM327 as MQTT Publish GPS as MQTT
Monitoring car OBD2 data over time
Our data is being published to the
HiveMQ MQTT broker!
Now for the easy part: writing the
data to InfluxDB
Monitoring car OBD2 data over time
Connect to HiveMQ broker
Monitoring car OBD2 data over time
Specify the RPM topic
Monitoring car OBD2 data over time
Use Regex (.*) to capture the entire message as _value
“rpm” will be our _field name
“obd” measurement name can be reused in each subscription
Leave this empty to allow InfluxDB to assign a timestamp for us
import "experimental/geo"
from(bucket: "iot")
|> range(start: -1h)
|> filter(fn: (r) => r["_measurement"] == "obd")
|> filter(fn: (r) => r["_field"] == "lat" or r["_field"] == "long")
|> geo.shapeData(latField: "lat", lonField: "long", level: 20)
Monitoring car OBD2 data over time
Check out my shared Notebook!
https://tinyurl.com/influxdb-obd
from(bucket: "iot")
|> range(start: -1h)
|> filter(fn: (r) => r["_measurement"] == "obd")
|> filter(fn: (r) => r["_field"] == "mph")
|> toFloat()
Query MPH
Query GPS
Flux
Flux
Monitoring car OBD2 data over time
Check out my shared Notebook!
https://tinyurl.com/influxdb-obd
Monitoring car OBD2 data over time
https://github.com/genehynson/arduino-mqtt-influxdb-obd2
Monitoring car OBD2 data over time
If you want to skip the Arduino DIY…
● WiFi / 4G LTE Support
● Connects to any MQTT broker
● Includes support for EVs
● No coding required
● See https://community.autopi.io/
for fun extensions and project
ideas!
I have one on the way… xD
Additional Resources
Free InfluxDB: OSS or Cloud - influxdata.com/cloud
Forums: community.influxdata.com
Slack: influxcommunity.slack.com
Reddit: r/InfluxData
Influx Community (GH): github.com/InfluxCommunity
Book: awesome.influxdata.com
Docs: docs.influxdata.com
Blogs: influxdata.com/blog
InfluxDB University: influxdata.com/university
How-to guides: docs.influxdata.com/resources/how-to-guides/
T H A N K Y O U
1 of 39

More Related Content

Similar to Gene Hynson [InfluxData] | How We Built the MQTT Native Collector | InfluxDays 2022(20)

Home automation using IoT and AWS Cloud technologyHome automation using IoT and AWS Cloud technology
Home automation using IoT and AWS Cloud technology
ratthaslip ranokphanuwat601 views
Using Node-RED for building IoT workflowsUsing Node-RED for building IoT workflows
Using Node-RED for building IoT workflows
Aniruddha Chakrabarti2.2K views
IoT on Raspberry PiIoT on Raspberry Pi
IoT on Raspberry Pi
John Staveley224 views
Monkey ServerMonkey Server
Monkey Server
Eduardo Silva Pereira1.7K views

More from InfluxData(20)

Announcing InfluxDB ClusteredAnnouncing InfluxDB Clustered
Announcing InfluxDB Clustered
InfluxData100 views

Recently uploaded(20)

The Research Portal of Catalonia: Growing more (information) & more (services)The Research Portal of Catalonia: Growing more (information) & more (services)
The Research Portal of Catalonia: Growing more (information) & more (services)
CSUC - Consorci de Serveis Universitaris de Catalunya59 views
ThroughputThroughput
Throughput
Moisés Armani Ramírez31 views
Green Leaf Consulting: Capabilities DeckGreen Leaf Consulting: Capabilities Deck
Green Leaf Consulting: Capabilities Deck
GreenLeafConsulting177 views
[2023] Putting the R! in R&D.pdf[2023] Putting the R! in R&D.pdf
[2023] Putting the R! in R&D.pdf
Eleanor McHugh36 views

Gene Hynson [InfluxData] | How We Built the MQTT Native Collector | InfluxDays 2022

  • 2. Native MQTT Collector Gene Hynson - Senior Software Engineer, InfluxData
  • 3. The MQTT Native Collector is an all new way to natively ingest MQTT data from your MQTT Broker into InfluxDB Cloud. This zero code solution makes it easier than ever to monitor your IoT devices in InfluxDB. This session provides an overview of the MQTT Native Collector, describes when and how to use it, and explains how it works behind the scenes. Lastly, Gene will demo his recent IoT project that utilizes the MQTT Native Collector. Gene Hynson Senior Software Engineer, InfluxData Gene is a Senior Software Engineer at InfluxData who is passionate about building thrilling product features for InfluxDB. Gene has 7 years of experience as a full-stack engineer building modern, cloud-first software at VMware, Pivotal, and Boeing. Outside of work, Gene spends his free time skiing the slopes in the winter or playing volleyball on the beach in the summer. MQTT Native Collector
  • 4. Agenda 1. What is the Native MQTT Collector? 2. How does Native MQTT Work? 3. How did we build Native MQTT? 4. Demo
  • 5. What is Native Data Collection?
  • 6. The ability to directly ingest data from popular data sources without the need to do any transformation to Line Protocol or run a Telegraf agent 6 Native Collection
  • 7. 7 MQTT Topic Subscriptions • Subscribe to topics being published by their own devices and sent through a message broker directly to the InfluxDB Cloud service • No code or installing any additional software required
  • 8. MQTT Ingestion: Telegraf vs Native MQTT
  • 9. How does Native Data Collection work?
  • 10. Simple 3 Step Configuration Define Parsing Rules • Configure Rules for finding timestamps, measurements, tags, and field information in the MQTT messages Configure Broker • Broker URL • Broker TCP/IP Port Number • Broker Authentication Type • Broker Authentication Parameters Configure Topic • Topic Name • InfluxDB Bucket
  • 19. Zero Code Data Ingestion
  • 20. How did we build Native Data Collection?
  • 21. How did we build Native MQTT?
  • 22. How did we build Native MQTT?
  • 23. Apache NiFi automates the flow of data between systems. ●Open source ●Data source agnostic ●Data type, format, & size agnostic ●End-to-end data encryption ●Guaranteed delivery ●Operates at a massive scale Apache NiFi
  • 24. How did we build Native MQTT?
  • 25. Demo: Monitoring car OBD2 data over time
  • 26. Monitoring car OBD2 data over time https://github.com/genehynson/arduino-mqtt-influxdb-obd2
  • 27. Monitoring car OBD2 data over time Arduino MKR1010 (WiFi) HC-05 Bluetooth Module NEO-6M GPS Module
  • 28. Monitoring car OBD2 data over time Arduino #include <WiFiNINA.h> while (WiFi.status() != WL_CONNECTED) { WiFi.begin(ssid, pass); delay(10000); } Serial.println("Connected to WiFi"); Serial.print("IP Address: "); Serial.println(WiFi.localIP()); Arduino #include <ArduinoMqttClient.h> char host[] = "broker.hivemq.com"; int port = 1883; if (mqttClient.connect(host, port)) { Serial.print("Connected to broker"); } Connect to WiFi Connect to MQTT broker
  • 29. Monitoring car OBD2 data over time Arduino #include <ELMduino.h> ELM327 elm; elm.begin(Serial1, true, 5000); ... float value = elm.rpm(); if (elm.nb_rx_state == ELM_SUCCESS) { mqttClient.beginMessage("obd/rpm"); mqttClient.print(value); mqttClient.endMessage(); } Arduino #include <TinyGPS++.h> gps.encode(Serial2.read()); if (gps.location.isUpdated()) { lat = gps.location.lat(); lng = gps.location.lng(); String json = "{"lat":"+lat+","long":"+lng+"}"; mqttClient.beginMessage("odb/gps"); mqttClient.print(json); mqttClient.endMessage(); } Publish ELM327 as MQTT Publish GPS as MQTT
  • 30. Monitoring car OBD2 data over time Our data is being published to the HiveMQ MQTT broker! Now for the easy part: writing the data to InfluxDB
  • 31. Monitoring car OBD2 data over time Connect to HiveMQ broker
  • 32. Monitoring car OBD2 data over time Specify the RPM topic
  • 33. Monitoring car OBD2 data over time Use Regex (.*) to capture the entire message as _value “rpm” will be our _field name “obd” measurement name can be reused in each subscription Leave this empty to allow InfluxDB to assign a timestamp for us
  • 34. import "experimental/geo" from(bucket: "iot") |> range(start: -1h) |> filter(fn: (r) => r["_measurement"] == "obd") |> filter(fn: (r) => r["_field"] == "lat" or r["_field"] == "long") |> geo.shapeData(latField: "lat", lonField: "long", level: 20) Monitoring car OBD2 data over time Check out my shared Notebook! https://tinyurl.com/influxdb-obd from(bucket: "iot") |> range(start: -1h) |> filter(fn: (r) => r["_measurement"] == "obd") |> filter(fn: (r) => r["_field"] == "mph") |> toFloat() Query MPH Query GPS Flux Flux
  • 35. Monitoring car OBD2 data over time Check out my shared Notebook! https://tinyurl.com/influxdb-obd
  • 36. Monitoring car OBD2 data over time https://github.com/genehynson/arduino-mqtt-influxdb-obd2
  • 37. Monitoring car OBD2 data over time If you want to skip the Arduino DIY… ● WiFi / 4G LTE Support ● Connects to any MQTT broker ● Includes support for EVs ● No coding required ● See https://community.autopi.io/ for fun extensions and project ideas! I have one on the way… xD
  • 38. Additional Resources Free InfluxDB: OSS or Cloud - influxdata.com/cloud Forums: community.influxdata.com Slack: influxcommunity.slack.com Reddit: r/InfluxData Influx Community (GH): github.com/InfluxCommunity Book: awesome.influxdata.com Docs: docs.influxdata.com Blogs: influxdata.com/blog InfluxDB University: influxdata.com/university How-to guides: docs.influxdata.com/resources/how-to-guides/
  • 39. T H A N K Y O U