RTAS 2023: Building a Real-Time IoT Application https://rtasummit.com/ Apache Pulsar, Apache Pinot, Apache Flink, Apache Kafka, Apache NiFi, FLaNK Stack, IoT https://rtasummit.com/session/building-a-real-time-iot-application-with-apache-pulsar-and-apache-pinot/ Building a Real-Time IoT Application with Apache Pulsar and Apache Pinot Timothy Spann Cloudera Time: Wednesday, April 26, 11:00 am Location: Nikko Ballroom I & II, 3rd Floor We will walk step-by-step with live code and demos on how to build a real-time IoT application with Pinot + Pulsar. First, we stream sensor data from an edge device monitoring location conditions to Pulsar via a Python application. We have our Apache Pinot “realtime” table connected to Pulsar via the pinot-pulsar stream ingestion connector. Our data streams into the stream, and we visualize it with Superset. https://medium.com/@tspann/building-a-real-time-iot-application-with-apache-pulsar-and-apache-pinot-1e3baf8c1824 Source Code https://github.com/tspannhw/pulsar-thermal-pinot Reference https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/apache-pulsar https://dev.startree.ai/docs/pinot/recipes/pulsar Create Topic in Pulsar bin/pulsar-admin topics delete persistent://public/default/thermalsensors bin/pulsar-admin topics create persistent://public/default/thermalsensors bin/pulsar-admin topics create-partitioned-topic --partitions 1 persistent://public/default/thermalsensors Consume Data in Pulsar bin/pulsar-client consume "persistent://public/default/thermalsensors" -s "thrmlsnosconsumer" -n 0 DevOps Pulsar curl http://localhost:8080/admin/v2/persistent/public/default curl http://localhost:8080/admin/v2/persistent/public/default/thermalsensors-partition-0/stats http://localhost:8080/admin/v2/persistent/public/default/thermalsensors/partitions?createLocalTopicOnly=false Data { "uuid": "thrml_qsx_20221121215610", "ipaddress": "192.168.1.179", "cputempf": 115, "runtime": 0, "host": "thermal", "hostname": "thermal", "macaddress": "e4:5f:01:7c:3f:34", "endtime": "1669067770.6400402", "te": "0.0005550384521484375", "cpu": 4.5, "diskusage": "102676.2 MB", "memory": 9.7, "rowid": "20221121215610_8e753591-cb7c-4e1c-886d-85cb3dba6c50", "systemtime": "11/21/2022 16:56:15", "ts": 1669067775, "starttime": "11/21/2022 16:56:10", "datetimestamp": "2022-11-21 21:56:14.404291+00:00", "temperature": 27.9069, "humidity": 24.89, "co2": 698.0, "totalvocppb": 0.0, "equivalentco2ppm": 65535.0, "pressure": 102048.65, "temperatureicp": 82.0 } Continuous Analytics with Flink SQL (Pulsar-Flink 1.15+ Connector) Reference: https://github.com/tspannhw/pulsar-transit-function CREATE CATALOG pulsar WITH ( 'type' = 'pulsar-catalog', 'catalog-service-url' = 'pulsar://localhost:6650', 'catalog-admin-url' = 'http://localhost:8080' ); SHOW CURRENT DATABASE; SHOW DATABASES; USE CATALOG pulsar; set table.dynamic-table-options.enabled = true;