Using FLiP with InfluxDB for EdgeAI IoT at Scale 2022
•1 like•87 views
Report
Share
Download to read offline
Using FLiP with InfluxDB for EdgeAI IoT at Scale 2022 https://adtmag.com/webcasts/2021/12/influxdata-february-10.aspx?tc=page0 Using FLiP with InfluxDB for EdgeAI IoT at Scale Date: Thursday, February 10th at 11am PT / 2pm ET Join this webcast as Timothy from StreamNative takes you on a hands-on deep-dive using Pulsar, Apache NiFi + Edge Flow Manager + MiniFi Agents with Apache MXNet, OpenVino, TensorFlow Lite, and other Deep Learning Libraries on the actual edge devices including Raspberry Pi with Movidius 2, Google Coral TPU and NVidia Jetson Nano. The team runs deep learning models on the edge devices, sends images, and captures real-time GPS and sensor data. Their low-coding IoT applications provide easy edge routing, transformation, data acquisition and alerting before they decide what data to stream in real-time to their data space. These edge applications classify images and sensor readings in real-time at the edge and then send Deep Learning results to Flink SQL and Apache NiFi for transformation, parsing, enrichment, querying, filtering and merging data to InfluxDB. In this session you will learn how to: Build an end-to-end streaming edge app Pull messages from Pulsar topics and persists the messages to InfluxDB Build a data stream for IoT with NiFi and InfluxDB Use Apache Flink + Apache Pulsar Timothy Spann, Developer Advocate, StreamNative Tim Spann is a Developer Advocate at StreamNative where he works with Apache NiFi, MiniFi, Kafka, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
Recommended
![[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines](https://cdn.slidesharecdn.com/ss_thumbnails/endss23tspannintegratingllmwithstreamingdatapipelines-231115161007-49510320-thumbnail.jpg?width=384&fit=bounds)
More Related Content
More from Timothy Spann
More from Timothy Spann(20)
Recently uploaded
Recently uploaded(20)
Using FLiP with InfluxDB for EdgeAI IoT at Scale 2022
- 1. Using FLiP with InfluxDB for Edge AI IoT at Scale Tim Spann | Developer Advocate +
- 2. streamnative.io + Tim Spann Developer Advocate Tim Spann, Developer Advocate at StreamNative ● FLiP(N) Stack = Flink, Pulsar and NiFI Stack ● Streaming Systems & Data Architecture Expert ● Experience: ○ 15+ years of experience with streaming technologies including Pulsar, Flink, Spark, NiFi, Kafka, Big Data, Cloud, MXNet, IoT and more. ○ Today, he helps to grow the Pulsar community sharing rich technical knowledge and experience at both global conferences and through individual conversations.
- 3. streamnative.io ● Learn how to build an end-to-end streaming edge app ● How to pull messages from Pulsar topics and persists the messages to InfluxDB ● Building a data stream for IoT with NiFi and InfluxDB ● Using Apache Flink + Apache Pulsar ● Using Apache Spark + Apache Pulsar Agenda +
- 4. streamnative.io FLiP Stack Weekly This week in Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark and open source friends. https://bit.ly/32dAJft +
- 5. streamnative.io ● Apache Flink ● Apache Pulsar ● Pulsar Functions ● StreamNative's Flink Connector for Pulsar ● Apache NiFi ● Apache Spark ● Python, Java, Golang FLiP(N)(S) Stack +
- 6. streamnative.io Apache Pulsar is a Cloud-Native Messaging and Event-Streaming Platform. +
- 7. Unified Messaging Model Simplify your data infrastructure and enable new use cases with queuing and streaming capabilities in one platform. Multi-tenancy Enable multiple user groups to share the same cluster, either via access control, or in entirely different namespaces. Scalability Decoupled data computing and storage enable horizontal scaling to handle data scale and management complexity. Geo-replication Support for multi-datacenter replication with both asynchronous and synchronous replication for built-in disaster recovery. Tiered storage Enable historical data to be offloaded to cloud-native storage and store event streams for indefinite periods of time. Pulsar Benefits
- 8. Top Pulsar Use Cases #1 Message Queuing #2 Data Streaming ✓ Not built for the cloud ✓ Single tenant systems ✓ Monolithic architecture couples compute with storage ✓ Lack of geo replication support
- 9. Messaging Ideal for work queues that do not require tasks to be performed in a particular order—for example, sending one email message to many recipients. RabbitMQ and Amazon SQS are examples of popular queue-based message systems. Pulsar: Unified Messaging + Data Streaming
- 10. Messaging Ideal for work queues that do not require tasks to be performed in a particular order—for example, sending one email message to many recipients. RabbitMQ and Amazon SQS are examples of popular queue-based message systems. Pulsar: Unified Messaging + Data Streaming .. and Streaming Works best in situations where the order of messages is important—for example, data ingestion. Kafka and Amazon Kinesis are examples of messaging systems that use streaming semantics for consuming messages.
- 11. Unified Messaging Model Streaming Messaging Producer 1 Producer 2 Pulsar Topic/Partition m0 m1 m2 m3 m4 Consumer D-1 Consumer D-2 Consumer D-3 Subscription D < k 2 , v 1 > < k 2 , v 3 > <k3 ,v2 > < k 1 , v 0 > < k 1 , v 4 > Key-Shared Consumer C-1 Consumer C-2 Consumer C-3 Subscription C m1 m2 m3 m4 m0 Shared Failover Consumer B-1 Consumer B-0 Subscription B m1 m2 m3 m4 m0 In case of failure in Consumer B-0 Consumer A-1 Consumer A-0 Subscription A m1 m2 m3 m4 m0 Exclusive X +
- 12. streamnative.io Use Cases Multi-Tenant Data Infrastructure AdTech Fraud Detection Connected Car IoT Analytics Microservices Development
- 14. Adopted Pulsar to replace Kafka in their DSP (Data Streaming Platform). ● 1.5-2x lower in capex cost ● 5-50x improvement in latency ● 2-3x lower in opex due Adopted Pulsar to power their billing platform, Midas, which processing hundreds of billions of financial transactions daily. Adoption then expanded to Tencent’s Federated Learning Platform and Tencent Gaming. Applied Materials is one of the biggest semiconductor hardware and software supplier in the industry. They adopted Pulsar to enable them to build a message bus to tie all of their data together. They previously used Tibco. Pulsar Adoption Use Cases
- 15. streamnative.io Passionate and dedicated team. Founded by the original developers of Apache Pulsar. StreamNative helps teams to capture, manage, and leverage data using Pulsar’s unified messaging and streaming platform.
- 16. Founded By The Creators Of Apache Pulsar Sijie Guo ASF Member Pulsar/BookKeeper PMC Founder and CEO Jia Zhai Pulsar/BookKeeper PMC Co-Founder ✓ Original creators of Apache Pulsar & BookKeeper ✓ Operated the largest Pulsar/BookKeeper cluster ✓ Data veterans with extensive industry experience Matteo Merli ASF Member Pulsar/BookKeeper PMC CTO
- 17. Powered by Apache Pulsar, StreamNative provides a cloud-native, real-time messaging and streaming platform to support multi-cloud and hybrid cloud strategies. StreamNative Cloud Flink SQL Cloud-Native
- 18. Apache Pulsar ● Serverless computing framework. ● Unbounded storage, multi-tiered architecture, and tiered-storage. ● Streaming & Pub/Sub messaging semantics. ● Multi-protocol support. +
- 19. streamnative.io Why Apache Pulsar? Unified Messaging Platform Guaranteed Message Delivery Resiliency Infinite Scalability +
- 20. streamnative.io Connectivity • Functions - Lightweight Stream Processing (Java, Python, Go) • Connectors - Sources & Sinks (InfluxDB, Kafka, S3, Kinesis, Lambda, …) • Protocol Handlers - AoP (AMQP), KoP (Kafka), MoP (MQTT) • Processing Engines - Flink, Spark, Presto/Trino via Pulsar SQL • Data Offloaders - Tiered Storage - (S3) hub.streamnative.io +
- 21. Use Cases ● Unified Messaging Platform ● AdTech ● Fraud Detection ● Connected Car ● IoT Analytics ● Microservices Development +
- 22. <-> Sensors <-> Streaming Edge Apps StreamNative Hub StreamNative Cloud Unified Batch and Stream COMPUTING Batch (Batch + Stream) Unified Batch and Stream STORAGE Offload (Queuing + Streaming) Tiered Storage Pulsar --- KoP --- MoP --- Websocket Pulsar Sink Streaming Edge Gateway Protocols <-> Sensors <-> Apps +
- 23. StreamNative Hub StreamNative Cloud Unified Batch and Stream COMPUTING Batch (Batch + Stream) Unified Batch and Stream STORAGE Offload (Queuing + Streaming) Apache Flink - Apache Pulsar - Apache NiFi <-> Devices <-> influxdb Tiered Storage Pulsar --- KoP --- MoP --- Websocket --- HTTP Pulsar Sink Pulsar Sink Streaming Edge Gateway Protocols End-to-End Streaming Edge App +
- 24. Why +
- 25. Analytics +
- 27. Analytics +
- 28. Analytics +
- 29. streamnative.io Apache Pulsar InfluxDB Sink https://pulsar.apache.org/docs/en/io-influxdb-sink/ +
- 30. streamnative.io Apache Pulsar InfluxDB Sink pulsar-admin sinks create --archive ./connectors/pulsar-io-influxdb-2.9.1.nar --name inflx-jtsn --sink-config-file conf/influxcloud.yml --inputs jtsninflx pulsar-admin sinks get --name inflx-jtsn pulsar-admin sinks status --name inflx-jtsn +
- 31. streamnative.io Apache Pulsar InfluxDB Sink +
- 32. streamnative.io InfluxDB Telegraf MQTT https://www.influxdata.com/integration/mqtt-monitoring/ https:/ /github.com/streamnative/mop MQTT on Pulsar (MoP) +
- 36. streamnative.io InfluxDB Telegraf RabbitMQ https://www.influxdata.com/integration/mqtt-monitoring/ + https://www.influxdata.com/integration/mqtt-monitoring/ https:/ /github.com/streamnative/aop AMQP on Pulsar (AoP) https:/ /hub.streamnative.io/connectors/amqp-1-0-sink/v2.7.1.1 https:/ /hub.streamnative.io/connectors/amqp-1-0-source/v2.7.1.1
- 37. streamnative.io Why Apache NiFi? https://www.influxdata.com/integration/mqtt-monitoring/ + https://www.influxdata.com/integration/mqtt-monitoring/ • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Hundreds of processors • Visual command and control • Over a 300 sources • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering • Version Control
- 38. streamnative.io Apache NiFi Pulsar Connector https://github.com/david-streamlio/pulsar-nifi-bundle +
- 39. streamnative.io Apache NiFi Pulsar Connector
- 40. streamnative.io Apache NiFi Pulsar Connector +
- 41. streamnative.io Apache NiFi Pulsar Connector +
- 42. streamnative.io InfluxDB via Apache NiFi + https://github.com/influxdata/nifi-influxdb-bundle https://www.influxdata.com/blog/building-a-data-stream-for-iot-with-nifi-and-influxdb/
- 43. streamnative.io ● Unified computing engine ● Batch processing is a special case of stream processing ● Stateful processing ● Massive Scalability ● Flink SQL for queries, inserts against Pulsar Topics ● Streaming Analytics ● Continuous SQL ● Continuous ETL ● Complex Event Processing ● Standard SQL Powered by Apache Calcite Why Apache Flink? +
- 58. ● https://github.com/tspannhw/FLiP-InfluxDB ● https://pulsar.apache.org/docs/en/io-influxdb-sink/ ● https://pulsar.apache.org/docs/en/io-overview/ ● https://github.com/tspannhw/Flip-solr ● https://github.com/tspannhw/Flip-transit ● https://github.com/tspannhw/Flip-iot ● https://www.datainmotion.dev/2020/10/flank-strea ming-edgeai-on-new-nvidia.html ● https://github.com/tspannhw/minifi-jetson-nano ● https://github.com/tspannhw/SpeakerProfile/ ● https://hub.streamnative.io/connectors/influxdb-sin k/2.5.1/ ● https://github.com/tspannhw/minifi-xaviernx/
- 59. Resources Now Available On-Demand Pulsar Training Academy.StreamNative Live 3-day Developers Training Save Your Spot! Free ebook Pulsar in Action Read it now
- 60. Let’s Keep in Touch! Timothy Spann Developer Advocate @PaasDev https://www.linkedin.com/in/timothyspann https://github.com/tspannhw +
- 61. Questions?