SlideShare a Scribd company logo

CoC23_ Let’s Monitor The Conditions at the Conference

CoC23_ Let’s Monitor The Conditions at the Conference At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around my desk.Let’s bring this to the different spots around the conference including lunch tables, vendor booths, hotel rooms, and more. I need to know about these readings now, not when I get back home from the conference. We need to get these sensor readings immediately in case we need to turn on a fan or move to another area. We will also see if my talk produces a lot of hot air!?!??My setup is pretty simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I can carry. All code and parameters will be available ASF-licensed in github.The software stack is Python and Java, MQTT, HTML, JQuery, and Apache NiFi, Apache Kafka and Apache Flink.

1 of 17
Download to read offline
Let’s Monitor The Conditions at the
Conference
Tim Spann
Principal Developer Advocate
Tuesday October 10, 2023
11:20 AM - 12:00 PM
Room 102
© 2023 Cloudera, Inc. All rights reserved. 2
At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around
my desk.Let’s bring this to the different spots around the conference including lunch tables, vendor
booths, hotel rooms, and more. I need to know about these readings now, not when I get back home
from the conference. We need to get these sensor readings immediately in case we need to turn on a
fan or move to another area. We will also see if my talk produces a lot of hot air!?!??My setup is pretty
simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I can carry. All code
and parameters will be available ASF-licensed in github.The software stack is Python and Java, MQTT,
HTML, JQuery, and Apache NiFi, Apache Kafka and Apache Flink.
3
FLaNK Stack
Tim Spann
@PaasDev // Blog: www.datainmotion.dev
Principal Developer Advocate.
Princeton Future of Data Meetup.
ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-PwC
https://medium.com/@tspann
https://github.com/tspannhw
Apache NiFi x Apache Kafka x Apache Flink
Slides, Code, Articles and More…
© 2023 Cloudera, Inc. All rights reserved. 5
Future of Data - New York + Princeton + Virtual
@PaasDev
https://www.meetup.com/futureofdata-princeton/
https://www.meetup.com/futureofdata-newyork/
From Big Data to AI to Streaming to Containers to Cloud
to Analytics to Cloud Storage to Fast Data to Machine
Learning to Microservices to ...
FLaNK Stack Weekly
This week in Apache NiFi, Apache Flink, Apache
Kafka, Apache Spark, Apache Iceberg, Python, Java,
AI, ML, LLM and Open Source friends.
https://bit.ly/32dAJft

Recommended

Pivotal Cloud Foundry 2.3: A First Look
Pivotal Cloud Foundry 2.3: A First LookPivotal Cloud Foundry 2.3: A First Look
Pivotal Cloud Foundry 2.3: A First LookVMware Tanzu
 
Zephyr Introduction - Nordic Webinar - Sept. 24.pdf
Zephyr Introduction - Nordic Webinar - Sept. 24.pdfZephyr Introduction - Nordic Webinar - Sept. 24.pdf
Zephyr Introduction - Nordic Webinar - Sept. 24.pdfAswathRangaraj1
 
How Aporeto Secures Cloud-native Across Public, Private, & Hybrid Clouds with...
How Aporeto Secures Cloud-native Across Public, Private, & Hybrid Clouds with...How Aporeto Secures Cloud-native Across Public, Private, & Hybrid Clouds with...
How Aporeto Secures Cloud-native Across Public, Private, & Hybrid Clouds with...DevOps.com
 
HashiTalks 2020 - Chef Tools & Terraform: Better Together
HashiTalks 2020 - Chef Tools & Terraform: Better TogetherHashiTalks 2020 - Chef Tools & Terraform: Better Together
HashiTalks 2020 - Chef Tools & Terraform: Better TogetherMatt Ray
 
Using Data Science & Serverless Python to find apartment in Toronto
Using Data Science & Serverless Python to find apartment in TorontoUsing Data Science & Serverless Python to find apartment in Toronto
Using Data Science & Serverless Python to find apartment in TorontoDaniel Zivkovic
 
ApacheCon 2021: Cracking the nut with Apache Pulsar (FLiP)
ApacheCon 2021:  Cracking the nut with Apache Pulsar (FLiP)ApacheCon 2021:  Cracking the nut with Apache Pulsar (FLiP)
ApacheCon 2021: Cracking the nut with Apache Pulsar (FLiP)Timothy Spann
 
Introduction to Apache NiFi 1.10
Introduction to Apache NiFi 1.10Introduction to Apache NiFi 1.10
Introduction to Apache NiFi 1.10Timothy Spann
 
TIAD - DYI: A simple orchestrator built step by step
TIAD - DYI: A simple orchestrator built step by stepTIAD - DYI: A simple orchestrator built step by step
TIAD - DYI: A simple orchestrator built step by stepThe Incredible Automation Day
 

More Related Content

Similar to CoC23_ Let’s Monitor The Conditions at the Conference

"Wie passen Serverless & Autonomous zusammen?"
"Wie passen Serverless & Autonomous zusammen?""Wie passen Serverless & Autonomous zusammen?"
"Wie passen Serverless & Autonomous zusammen?"Volker Linz
 
“Khronos Standard APIs for Accelerating Vision and Inferencing,” a Presentati...
“Khronos Standard APIs for Accelerating Vision and Inferencing,” a Presentati...“Khronos Standard APIs for Accelerating Vision and Inferencing,” a Presentati...
“Khronos Standard APIs for Accelerating Vision and Inferencing,” a Presentati...Edge AI and Vision Alliance
 
CENTRAL MANAGEMENT OF NETWORK AND CALL SERVICES
CENTRAL MANAGEMENT OF NETWORK AND CALL SERVICESCENTRAL MANAGEMENT OF NETWORK AND CALL SERVICES
CENTRAL MANAGEMENT OF NETWORK AND CALL SERVICESNazmul Hossain Rakib
 
Chicago Docker Meetup Presentation - Mediafly
Chicago Docker Meetup Presentation - MediaflyChicago Docker Meetup Presentation - Mediafly
Chicago Docker Meetup Presentation - MediaflyMediafly
 
Python on Cloud Foundry
Python on Cloud FoundryPython on Cloud Foundry
Python on Cloud FoundryIan Huston
 
IoT Edge Data Processing with NVidia Jetson Nano oct 3 2019
IoT  Edge Data Processing with NVidia Jetson Nano oct 3 2019IoT  Edge Data Processing with NVidia Jetson Nano oct 3 2019
IoT Edge Data Processing with NVidia Jetson Nano oct 3 2019Timothy Spann
 
DevOPS training - Day 1/2
DevOPS training - Day 1/2DevOPS training - Day 1/2
DevOPS training - Day 1/2Vincent Mercier
 
FIWARE Wednesday Webinars - How to Debug IoT Agents
FIWARE Wednesday Webinars - How to Debug IoT AgentsFIWARE Wednesday Webinars - How to Debug IoT Agents
FIWARE Wednesday Webinars - How to Debug IoT AgentsFIWARE
 
Codecoon - A technical Case Study
Codecoon - A technical Case StudyCodecoon - A technical Case Study
Codecoon - A technical Case StudyMichael Lihs
 
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFiTracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFiDataWorks Summit
 
.NET Cloud-Native Bootcamp
.NET Cloud-Native Bootcamp.NET Cloud-Native Bootcamp
.NET Cloud-Native BootcampVMware Tanzu
 
Industrial transformation-simplified-with-mqtt-and-sparkplug
Industrial transformation-simplified-with-mqtt-and-sparkplugIndustrial transformation-simplified-with-mqtt-and-sparkplug
Industrial transformation-simplified-with-mqtt-and-sparkplugHugoMller5
 
Zephyr-Overview-20230124.pdf
Zephyr-Overview-20230124.pdfZephyr-Overview-20230124.pdf
Zephyr-Overview-20230124.pdfibramax
 
Kubernetes One-Click Deployment: Hands-on Workshop (Mainz)
Kubernetes One-Click Deployment: Hands-on Workshop (Mainz)Kubernetes One-Click Deployment: Hands-on Workshop (Mainz)
Kubernetes One-Click Deployment: Hands-on Workshop (Mainz)QAware GmbH
 
BKK16-406 Ubuntu Core - a snappy platform for Embedded, IoT and 96boards!
BKK16-406 Ubuntu Core - a snappy platform for Embedded, IoT and 96boards!BKK16-406 Ubuntu Core - a snappy platform for Embedded, IoT and 96boards!
BKK16-406 Ubuntu Core - a snappy platform for Embedded, IoT and 96boards!Linaro
 
Rome 2017: Building advanced voice assistants and chat bots
Rome 2017: Building advanced voice assistants and chat botsRome 2017: Building advanced voice assistants and chat bots
Rome 2017: Building advanced voice assistants and chat botsCisco DevNet
 
Meeting rooms are talking! are you listening?
Meeting rooms are talking! are you listening?Meeting rooms are talking! are you listening?
Meeting rooms are talking! are you listening?Cisco DevNet
 
OSMC 2009 | NConf - Enterprise Nagios configurator by Angelo Gargiulo
OSMC 2009 | NConf - Enterprise Nagios configurator by Angelo GargiuloOSMC 2009 | NConf - Enterprise Nagios configurator by Angelo Gargiulo
OSMC 2009 | NConf - Enterprise Nagios configurator by Angelo GargiuloNETWAYS
 
The internet of $h1t
The internet of $h1tThe internet of $h1t
The internet of $h1tAmit Serper
 

Similar to CoC23_ Let’s Monitor The Conditions at the Conference (20)

"Wie passen Serverless & Autonomous zusammen?"
"Wie passen Serverless & Autonomous zusammen?""Wie passen Serverless & Autonomous zusammen?"
"Wie passen Serverless & Autonomous zusammen?"
 
“Khronos Standard APIs for Accelerating Vision and Inferencing,” a Presentati...
“Khronos Standard APIs for Accelerating Vision and Inferencing,” a Presentati...“Khronos Standard APIs for Accelerating Vision and Inferencing,” a Presentati...
“Khronos Standard APIs for Accelerating Vision and Inferencing,” a Presentati...
 
CENTRAL MANAGEMENT OF NETWORK AND CALL SERVICES
CENTRAL MANAGEMENT OF NETWORK AND CALL SERVICESCENTRAL MANAGEMENT OF NETWORK AND CALL SERVICES
CENTRAL MANAGEMENT OF NETWORK AND CALL SERVICES
 
Chicago Docker Meetup Presentation - Mediafly
Chicago Docker Meetup Presentation - MediaflyChicago Docker Meetup Presentation - Mediafly
Chicago Docker Meetup Presentation - Mediafly
 
Python on Cloud Foundry
Python on Cloud FoundryPython on Cloud Foundry
Python on Cloud Foundry
 
IoT Edge Data Processing with NVidia Jetson Nano oct 3 2019
IoT  Edge Data Processing with NVidia Jetson Nano oct 3 2019IoT  Edge Data Processing with NVidia Jetson Nano oct 3 2019
IoT Edge Data Processing with NVidia Jetson Nano oct 3 2019
 
DevOPS training - Day 1/2
DevOPS training - Day 1/2DevOPS training - Day 1/2
DevOPS training - Day 1/2
 
FIWARE Wednesday Webinars - How to Debug IoT Agents
FIWARE Wednesday Webinars - How to Debug IoT AgentsFIWARE Wednesday Webinars - How to Debug IoT Agents
FIWARE Wednesday Webinars - How to Debug IoT Agents
 
Codecoon - A technical Case Study
Codecoon - A technical Case StudyCodecoon - A technical Case Study
Codecoon - A technical Case Study
 
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFiTracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
 
Fish Cam.pptx
Fish Cam.pptxFish Cam.pptx
Fish Cam.pptx
 
.NET Cloud-Native Bootcamp
.NET Cloud-Native Bootcamp.NET Cloud-Native Bootcamp
.NET Cloud-Native Bootcamp
 
Industrial transformation-simplified-with-mqtt-and-sparkplug
Industrial transformation-simplified-with-mqtt-and-sparkplugIndustrial transformation-simplified-with-mqtt-and-sparkplug
Industrial transformation-simplified-with-mqtt-and-sparkplug
 
Zephyr-Overview-20230124.pdf
Zephyr-Overview-20230124.pdfZephyr-Overview-20230124.pdf
Zephyr-Overview-20230124.pdf
 
Kubernetes One-Click Deployment: Hands-on Workshop (Mainz)
Kubernetes One-Click Deployment: Hands-on Workshop (Mainz)Kubernetes One-Click Deployment: Hands-on Workshop (Mainz)
Kubernetes One-Click Deployment: Hands-on Workshop (Mainz)
 
BKK16-406 Ubuntu Core - a snappy platform for Embedded, IoT and 96boards!
BKK16-406 Ubuntu Core - a snappy platform for Embedded, IoT and 96boards!BKK16-406 Ubuntu Core - a snappy platform for Embedded, IoT and 96boards!
BKK16-406 Ubuntu Core - a snappy platform for Embedded, IoT and 96boards!
 
Rome 2017: Building advanced voice assistants and chat bots
Rome 2017: Building advanced voice assistants and chat botsRome 2017: Building advanced voice assistants and chat bots
Rome 2017: Building advanced voice assistants and chat bots
 
Meeting rooms are talking! are you listening?
Meeting rooms are talking! are you listening?Meeting rooms are talking! are you listening?
Meeting rooms are talking! are you listening?
 
OSMC 2009 | NConf - Enterprise Nagios configurator by Angelo Gargiulo
OSMC 2009 | NConf - Enterprise Nagios configurator by Angelo GargiuloOSMC 2009 | NConf - Enterprise Nagios configurator by Angelo Gargiulo
OSMC 2009 | NConf - Enterprise Nagios configurator by Angelo Gargiulo
 
The internet of $h1t
The internet of $h1tThe internet of $h1t
The internet of $h1t
 

More from Timothy Spann

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...Timothy Spann
 
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and FlinkDBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and FlinkTimothy Spann
 
NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...
NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...
NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...Timothy Spann
 
Building Real-Time Travel Alerts
Building Real-Time Travel AlertsBuilding Real-Time Travel Alerts
Building Real-Time Travel AlertsTimothy Spann
 
JConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and FlinkJConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and FlinkTimothy Spann
 
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
[EN]DSS23_tspann_Integrating LLM with Streaming Data PipelinesTimothy Spann
 
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines DemoEvolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines DemoTimothy Spann
 
CoC23_ Looking at the New Features of Apache NiFi
CoC23_ Looking at the New Features of Apache NiFiCoC23_ Looking at the New Features of Apache NiFi
CoC23_ Looking at the New Features of Apache NiFiTimothy Spann
 
OSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdf
OSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdfOSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdf
OSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdfTimothy Spann
 
CoC23_Utilizing Real-Time Transit Data for Travel Optimization
CoC23_Utilizing Real-Time Transit Data for Travel OptimizationCoC23_Utilizing Real-Time Transit Data for Travel Optimization
CoC23_Utilizing Real-Time Transit Data for Travel OptimizationTimothy Spann
 
The Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingThe Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingTimothy Spann
 
Meetup - Brasil - Data In Motion - 2023 September 19
Meetup - Brasil - Data In Motion - 2023 September 19Meetup - Brasil - Data In Motion - 2023 September 19
Meetup - Brasil - Data In Motion - 2023 September 19Timothy Spann
 
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Timothy Spann
 
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit DataBuilding Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit DataTimothy Spann
 
big data fest building modern data streaming apps
big data fest building modern data streaming appsbig data fest building modern data streaming apps
big data fest building modern data streaming appsTimothy Spann
 
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Using Apache NiFi with Apache Pulsar for Fast Data On-RampUsing Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Using Apache NiFi with Apache Pulsar for Fast Data On-RampTimothy Spann
 
OSSNA Building Modern Data Streaming Apps
OSSNA Building Modern Data Streaming AppsOSSNA Building Modern Data Streaming Apps
OSSNA Building Modern Data Streaming AppsTimothy Spann
 
GSJUG: Mastering Data Streaming Pipelines 09May2023
GSJUG: Mastering Data Streaming Pipelines 09May2023GSJUG: Mastering Data Streaming Pipelines 09May2023
GSJUG: Mastering Data Streaming Pipelines 09May2023Timothy Spann
 
BestInFlowCompetitionTutorials03May2023
BestInFlowCompetitionTutorials03May2023BestInFlowCompetitionTutorials03May2023
BestInFlowCompetitionTutorials03May2023Timothy Spann
 
CloudToolGuidance03May2023
CloudToolGuidance03May2023CloudToolGuidance03May2023
CloudToolGuidance03May2023Timothy Spann
 

More from Timothy Spann (20)

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
 
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and FlinkDBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
 
NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...
NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...
NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...
 
Building Real-Time Travel Alerts
Building Real-Time Travel AlertsBuilding Real-Time Travel Alerts
Building Real-Time Travel Alerts
 
JConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and FlinkJConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and Flink
 
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
 
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines DemoEvolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
 
CoC23_ Looking at the New Features of Apache NiFi
CoC23_ Looking at the New Features of Apache NiFiCoC23_ Looking at the New Features of Apache NiFi
CoC23_ Looking at the New Features of Apache NiFi
 
OSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdf
OSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdfOSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdf
OSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdf
 
CoC23_Utilizing Real-Time Transit Data for Travel Optimization
CoC23_Utilizing Real-Time Transit Data for Travel OptimizationCoC23_Utilizing Real-Time Transit Data for Travel Optimization
CoC23_Utilizing Real-Time Transit Data for Travel Optimization
 
The Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingThe Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and Streaming
 
Meetup - Brasil - Data In Motion - 2023 September 19
Meetup - Brasil - Data In Motion - 2023 September 19Meetup - Brasil - Data In Motion - 2023 September 19
Meetup - Brasil - Data In Motion - 2023 September 19
 
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
 
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit DataBuilding Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
 
big data fest building modern data streaming apps
big data fest building modern data streaming appsbig data fest building modern data streaming apps
big data fest building modern data streaming apps
 
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Using Apache NiFi with Apache Pulsar for Fast Data On-RampUsing Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp
 
OSSNA Building Modern Data Streaming Apps
OSSNA Building Modern Data Streaming AppsOSSNA Building Modern Data Streaming Apps
OSSNA Building Modern Data Streaming Apps
 
GSJUG: Mastering Data Streaming Pipelines 09May2023
GSJUG: Mastering Data Streaming Pipelines 09May2023GSJUG: Mastering Data Streaming Pipelines 09May2023
GSJUG: Mastering Data Streaming Pipelines 09May2023
 
BestInFlowCompetitionTutorials03May2023
BestInFlowCompetitionTutorials03May2023BestInFlowCompetitionTutorials03May2023
BestInFlowCompetitionTutorials03May2023
 
CloudToolGuidance03May2023
CloudToolGuidance03May2023CloudToolGuidance03May2023
CloudToolGuidance03May2023
 

Recently uploaded

Open Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsOpen Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsSprings
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flinkconfluent
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowNaoki (Neo) SATO
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureHironori Washizaki
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleAmir Moghimi
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Fermin Galan
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCIOWomenMagazine
 
killingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdfkillingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdfssuser82c38d
 
Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesGlobus
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System AdministratorsGlobus
 
Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019VICTOR MAESTRE RAMIREZ
 
Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)Jeffrey Haguewood
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with GlobusGlobus
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارsohilww
 
Managing multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerManaging multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerThierry Gayet
 
Agile & Scrum, Certified Scrum Master! Crash Course
Agile & Scrum,  Certified Scrum Master! Crash CourseAgile & Scrum,  Certified Scrum Master! Crash Course
Agile & Scrum, Certified Scrum Master! Crash CourseRohan Chandane
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowGlobus
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaSGlobus
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration TopicsGlobus
 

Recently uploaded (20)

Open Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsOpen Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and Cons
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flow
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about Architecture
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scale
 
2024 Trends Transforming Enterprise Resource Planning
2024 Trends Transforming Enterprise Resource Planning2024 Trends Transforming Enterprise Resource Planning
2024 Trends Transforming Enterprise Resource Planning
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdf
 
killingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdfkillingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdf
 
Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All Scales
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
 
Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019
 
Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with Globus
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
 
Managing multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerManaging multicast/igmp stream on Docker
Managing multicast/igmp stream on Docker
 
Agile & Scrum, Certified Scrum Master! Crash Course
Agile & Scrum,  Certified Scrum Master! Crash CourseAgile & Scrum,  Certified Scrum Master! Crash Course
Agile & Scrum, Certified Scrum Master! Crash Course
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a Flow
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaS
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration Topics
 

CoC23_ Let’s Monitor The Conditions at the Conference

  • 1. Let’s Monitor The Conditions at the Conference Tim Spann Principal Developer Advocate Tuesday October 10, 2023 11:20 AM - 12:00 PM Room 102
  • 2. © 2023 Cloudera, Inc. All rights reserved. 2 At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around my desk.Let’s bring this to the different spots around the conference including lunch tables, vendor booths, hotel rooms, and more. I need to know about these readings now, not when I get back home from the conference. We need to get these sensor readings immediately in case we need to turn on a fan or move to another area. We will also see if my talk produces a lot of hot air!?!??My setup is pretty simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I can carry. All code and parameters will be available ASF-licensed in github.The software stack is Python and Java, MQTT, HTML, JQuery, and Apache NiFi, Apache Kafka and Apache Flink.
  • 3. 3 FLaNK Stack Tim Spann @PaasDev // Blog: www.datainmotion.dev Principal Developer Advocate. Princeton Future of Data Meetup. ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-PwC https://medium.com/@tspann https://github.com/tspannhw Apache NiFi x Apache Kafka x Apache Flink
  • 5. © 2023 Cloudera, Inc. All rights reserved. 5 Future of Data - New York + Princeton + Virtual @PaasDev https://www.meetup.com/futureofdata-princeton/ https://www.meetup.com/futureofdata-newyork/ From Big Data to AI to Streaming to Containers to Cloud to Analytics to Cloud Storage to Fast Data to Machine Learning to Microservices to ...
  • 6. FLaNK Stack Weekly This week in Apache NiFi, Apache Flink, Apache Kafka, Apache Spark, Apache Iceberg, Python, Java, AI, ML, LLM and Open Source friends. https://bit.ly/32dAJft
  • 7. My Talk List Utilizing Real-Time Transit Data for Travel Optimization Looking at the New Features of Apache NiFi
  • 9. APACHE NiFi - MiNiFi Agents NiFi For Ants
  • 10. © 2023 Cloudera, Inc. All rights reserved. 10 Apache NiFi - MiNiFi Agent Edge device data collection and processing with easy to use central command and control • Small footprint agent with MiNiFi • Java and C++ agents • Rich edge processors (edge collection & processing) • End to end lineage and security • Kubernetes support • Central Command and Control (C2) • Design and deploy to millions of agents • Edge Applications lifecycle management • Multitenancy with Agent classes • Native integration with other CDF services + A lightweight edge agent that implements the core features of Apache NiFi, focusing on data collection and processing at the edge
  • 11. © 2023 Cloudera, Inc. All rights reserved. 11
  • 13. Hardware List ● Pimoroni Breakout Garden ● Raspberry Pi 400 ● MLX90640 Thermal Camera Standard ● SGP30 Air Quality Sensor Breakout (TVOC/eCO2)
  • 14. Actions ● MiNiFi: Call Python script and send to Kafka and NiFi ● Consume Kafka Streams in Apache Flink SQL & Apache NiFi ● Real-Time Analytics ● Produce HTML Page Listing all the images as uploaded ● NiFi extensions: Send to slack, websockets, html page, email, …
  • 15. DEMO
  • 16. Aggregate all data from sensors, drones, logs, geo-location devices, images from cameras, results from running predictions on pre-trained models. Collect: Bring Together Mediate point-to-point and bi-directional data flows, distribute, delivering data reliably to Apache Iceberg, S3, Apache Kudu, Apache HBase, JDBC, Slack and Email. Conduct: Mediate the Data Flow Orchestrate, parse, merge, aggregate, filter, join, transform, fork, query, sort, dissect, store, enrich with weather, location, sentiment analysis, image analysis, object detection, image recognition and more with Apache Tika, GenAI REST, Apache OpenNLP and Machine Learning. Curate: Gain Insights