SlideShare a Scribd company logo
1 of 12
Download to read offline
Fast Data for Fitness
Real-time Health Sensor Ingest
Timothy Spann - Principal DataFlow Field Engineer
Ā© 2020 Cloudera, Inc. All rights reserved. 3
Timothy Spann
Data in Motion Principal Field Engineer
@PaasDev
DZone Zone Leader and Big Data MVB
Princeton NJ Future of Data Meetup
https://github.com/tspannhw https://www.datainmotion.dev/
https://meetup.com/futureofdata-princeton/
Ā© 2020 Cloudera, Inc. All rights reserved. 4
Data Sources - From Images to Sub-Second Data Events
Heart Rate
Temperature
Blood Sugar
Altimeter
Blood Pressure
Weight
GPS
Live Video/Picture
AI for Pose Estimation
Improvement to Technique
Ā© 2020 Cloudera, Inc. All rights reserved. 5
Data for Fitness+ Pipeline
Weather
Weather
Climate
Aggregates
Sensors
SQL
Analytics
Sources
Pollution
Wearables
REST
Ā© 2020 Cloudera, Inc. All rights reserved. 6
Fitness+ Data Streaming Pipeline
Device Data
SensorsLogs
Weather
Sensors
Aggregates
Energy
SQL
Analytics
MiNiFi
Agent
Deep Learning
Classiļ¬cation
Edge Private
Cloud
Multi-Public
Cloud
Ā© 2020 Cloudera, Inc. All rights reserved. 7
Fast Fitness+ Data
Ā© 2020 Cloudera, Inc. All rights reserved. 8
CLOUDERA FLOW AND EDGE MANAGEMENT
Enable easy ingestion, routing, management and delivery of any data anywhere (Edge, cloud, data
center) to any downstream system with built in end-to-end security and provenance
Advanced tooling to industrialize
ļ¬‚ow development (Flow Development
Life Cycle)
ACQUIRE
ā€¢ Over 300 Prebuilt Processors
ā€¢ Easy to build your own
ā€¢ Parse, Enrich & Apply Schema
ā€¢ Filter, Split, Merger & Route
ā€¢ Throttle & Backpressure
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLOG
PROCESS
HASH
MERGE
EXTRACT
DUPLICATE
SPLIT
ENCRYPT
TALL
EVALUATE
EXECUTE
GEOENRICH
SCAN
REPLACE
TRANSLATE
CONVERT
ROUTE TEXT
ROUTE CONTENT
ROUTE CONTEXT
ROUTE RATE
DISTRIBUTE LOAD
DELIVER
ā€¢ Guaranteed Delivery
ā€¢ Full data provenance from
acquisition to delivery
ā€¢ Diverse, Non-Traditional Sources
ā€¢ Eco-system integration
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLOG
Ā© 2019 Cloudera, Inc. All rights reserved. 9
Apache Kafka
ā€¢ Highly reliable distributed
messaging system
ā€¢ Decouple applications, enables
many-to-many patterns
ā€¢ Publish-Subscribe semantics
ā€¢ Horizontal scalability
ā€¢ Eļ¬ƒcient implementation to
operate at speed with big data
volumes
ā€¢ Organized by topic to support
several use cases
Source
System
Source
System
Source
System
Kafka
Fraud
Detection
Security
Systems
Real-Time
Monitoring
Source
System
Source
System
Source
System
Fraud
Detection
Security
Systems
Real-Time
Monitoring
Many-To-Many
Publish-Subscribe
Point-To-Point
Request-Response
Ā© 2020 Cloudera, Inc. All rights reserved. 10
New features delivered with Cloudera Streaming Analytics (CSA) 1.2
Next Generation Streaming Analytics
Flink SQL Support
Agile Streaming App
Development using SQL
New Flink Atlas Hook
Capture operational Flink
app metadata and lineage
Single View of Flink Yarn Jobs
Improve Developer Experience
& operational visibility
Ā© 2020 Cloudera, Inc. All rights reserved. 11
ā— https://www.cloudera.com/tutorials/building-a-sentiment-analysis-application/3.html
ā— https://blog.cloudera.com/benchmarking-niļ¬-performance-and-scalability/
ā— https://www.datainmotion.dev/2020/06/no-more-spaghetti-ļ¬‚ows.html
ā— https://www.datainmotion.dev/2020/06/the-rise-of-mega-edge-ļ¬‚ank.html
ā— https://www.datainmotion.dev/2020/01/cloudera-edge2ai-miniļ¬-java-agent-with.html
ā— https://www.datainmotion.dev/2019/08/rapid-iot-development-with-cloudera.html
ā— https://www.datainmotion.dev/2019/09/powering-edge-ai-for-sensor-reading.html
ā— https://www.datainmotion.dev/2019/05/dataworks-summit-dc-2019-report.html
ā— https://www.datainmotion.dev/2019/03/using-raspberry-pi-3b-with-apache-niļ¬.html
ā— https://www.datainmotion.dev/2020/06/unboxing-most-amazing-edge-ai-device.html
ā— https://www.datainmotion.dev/2019/08/edge-processing-with-jetson-nano-part-3.html
ā— https://www.datainmotion.dev/2020/04/predicting-sensor-readings-with-time.html
REFERENCES
Ā© 2020 Cloudera, Inc. All rights reserved. 12
TH N Y U

More Related Content

What's hot

Streaming Visualization
Streaming VisualizationStreaming Visualization
Streaming Visualization
Guido Schmutz
Ā 
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQLIngesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
Guido Schmutz
Ā 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
Guido Schmutz
Ā 

What's hot (20)

PCAP Graphs for Cybersecurity and System Tuning
PCAP Graphs for Cybersecurity and System TuningPCAP Graphs for Cybersecurity and System Tuning
PCAP Graphs for Cybersecurity and System Tuning
Ā 
Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...
Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...
Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...
Ā 
Enterprise Metadata Integration
Enterprise Metadata IntegrationEnterprise Metadata Integration
Enterprise Metadata Integration
Ā 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaSolutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Ā 
Streaming Visualization
Streaming VisualizationStreaming Visualization
Streaming Visualization
Ā 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Ā 
Rediscovering the Value of Apache KafkaĀ® in Modern Data Architecture
Rediscovering the Value of Apache KafkaĀ® in Modern Data ArchitectureRediscovering the Value of Apache KafkaĀ® in Modern Data Architecture
Rediscovering the Value of Apache KafkaĀ® in Modern Data Architecture
Ā 
Streaming Visualization
Streaming VisualizationStreaming Visualization
Streaming Visualization
Ā 
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Ā 
Apache Spark in Scientific Applciations
Apache Spark in Scientific ApplciationsApache Spark in Scientific Applciations
Apache Spark in Scientific Applciations
Ā 
Build a Bridge to Cloud with Apache KafkaĀ® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache KafkaĀ® for Data Analytics Cloud ServicesBuild a Bridge to Cloud with Apache KafkaĀ® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache KafkaĀ® for Data Analytics Cloud Services
Ā 
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQLIngesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
Ā 
Building Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaBuilding Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache Kafka
Ā 
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemBuilding event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Ā 
Time Series Analysis Using an Event Streaming Platform
 Time Series Analysis Using an Event Streaming Platform Time Series Analysis Using an Event Streaming Platform
Time Series Analysis Using an Event Streaming Platform
Ā 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
Ā 
Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !
Ā 
Streaming and Social Media
Streaming and Social MediaStreaming and Social Media
Streaming and Social Media
Ā 
Cloud Connect 2012, Big Data @ Netflix
Cloud Connect 2012, Big Data @ NetflixCloud Connect 2012, Big Data @ Netflix
Cloud Connect 2012, Big Data @ Netflix
Ā 
Building Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaBuilding Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache Kafka
Ā 

Similar to Fast data for fitness 10 nov 2020

Meetup: Streaming Data Pipeline Development
Meetup:  Streaming Data Pipeline DevelopmentMeetup:  Streaming Data Pipeline Development
Meetup: Streaming Data Pipeline Development
Timothy Spann
Ā 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Timothy Spann
Ā 
Unconference Round Table Notes
Unconference Round Table NotesUnconference Round Table Notes
Unconference Round Table Notes
Timothy Spann
Ā 
Big Data LDN 2016: When Big Data Meets Fast Data
Big Data LDN 2016: When Big Data Meets Fast DataBig Data LDN 2016: When Big Data Meets Fast Data
Big Data LDN 2016: When Big Data Meets Fast Data
Matt Stubbs
Ā 

Similar to Fast data for fitness 10 nov 2020 (20)

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFiTracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Ā 
Meetup: Streaming Data Pipeline Development
Meetup:  Streaming Data Pipeline DevelopmentMeetup:  Streaming Data Pipeline Development
Meetup: Streaming Data Pipeline Development
Ā 
IBM Aspera overview
IBM Aspera overview IBM Aspera overview
IBM Aspera overview
Ā 
Simplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache KuduSimplifying Real-Time Architectures for IoT with Apache Kudu
Simplifying Real-Time Architectures for IoT with Apache Kudu
Ā 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Ā 
Unconference Round Table Notes
Unconference Round Table NotesUnconference Round Table Notes
Unconference Round Table Notes
Ā 
How to Build Continuous Ingestion for the Internet of Things
How to Build Continuous Ingestion for the Internet of ThingsHow to Build Continuous Ingestion for the Internet of Things
How to Build Continuous Ingestion for the Internet of Things
Ā 
Blockchain and Apache NiFi
Blockchain and Apache NiFiBlockchain and Apache NiFi
Blockchain and Apache NiFi
Ā 
Introduction to DDS: Context, Information Model, Security, and Applications.
Introduction to DDS: Context, Information Model, Security, and Applications.Introduction to DDS: Context, Information Model, Security, and Applications.
Introduction to DDS: Context, Information Model, Security, and Applications.
Ā 
Big Data LDN 2016: When Big Data Meets Fast Data
Big Data LDN 2016: When Big Data Meets Fast DataBig Data LDN 2016: When Big Data Meets Fast Data
Big Data LDN 2016: When Big Data Meets Fast Data
Ā 
Turning Petabytes of Data into Profit with Hadoop for the Worldā€™s Biggest Ret...
Turning Petabytes of Data into Profit with Hadoop for the Worldā€™s Biggest Ret...Turning Petabytes of Data into Profit with Hadoop for the Worldā€™s Biggest Ret...
Turning Petabytes of Data into Profit with Hadoop for the Worldā€™s Biggest Ret...
Ā 
IoT Connected Brewery
IoT Connected BreweryIoT Connected Brewery
IoT Connected Brewery
Ā 
Generative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and MilvusGenerative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and Milvus
Ā 
Global Data Stream Network for Internet of Things
Global Data Stream Network for Internet of ThingsGlobal Data Stream Network for Internet of Things
Global Data Stream Network for Internet of Things
Ā 
Horses for Courses: Database Roundtable
Horses for Courses: Database RoundtableHorses for Courses: Database Roundtable
Horses for Courses: Database Roundtable
Ā 
Solving Big Data Problems using Hortonworks
Solving Big Data Problems using Hortonworks Solving Big Data Problems using Hortonworks
Solving Big Data Problems using Hortonworks
Ā 
IoT Crash Course Hadoop Summit SJ
IoT Crash Course Hadoop Summit SJIoT Crash Course Hadoop Summit SJ
IoT Crash Course Hadoop Summit SJ
Ā 
Fighting cyber fraud with hadoop v2
Fighting cyber fraud with hadoop v2Fighting cyber fraud with hadoop v2
Fighting cyber fraud with hadoop v2
Ā 
Let's build a simple ingest to cloud datawarehouse with low code
Let's build a simple ingest to cloud datawarehouse with low codeLet's build a simple ingest to cloud datawarehouse with low code
Let's build a simple ingest to cloud datawarehouse with low code
Ā 
Gimel and PayPal Notebooks @ TDWI Leadership Summit Orlando
Gimel and PayPal Notebooks @ TDWI Leadership Summit OrlandoGimel and PayPal Notebooks @ TDWI Leadership Summit Orlando
Gimel and PayPal Notebooks @ TDWI Leadership Summit Orlando
Ā 

More from Timothy Spann

28March2024-Codeless-Generative-AI-Pipelines
28March2024-Codeless-Generative-AI-Pipelines28March2024-Codeless-Generative-AI-Pipelines
28March2024-Codeless-Generative-AI-Pipelines
Timothy Spann
Ā 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
Timothy Spann
Ā 
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Timothy Spann
Ā 
2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines
2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines
2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines
Timothy Spann
Ā 
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and FlinkDBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
Timothy Spann
Ā 
OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines
OSACon 2023_ Unlocking Financial Data with Real-Time PipelinesOSACon 2023_ Unlocking Financial Data with Real-Time Pipelines
OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines
Timothy Spann
Ā 
JConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and FlinkJConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and Flink
Timothy Spann
Ā 
AIDevWorldApacheNiFi101
AIDevWorldApacheNiFi101AIDevWorldApacheNiFi101
AIDevWorldApacheNiFi101
Timothy Spann
Ā 
26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup
26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup
26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup
Timothy Spann
Ā 

More from Timothy Spann (20)

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24  Building Real-Time Pipelines With FLaNKDATA SUMMIT 24  Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
Ā 
April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024
April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024
April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024
Ā 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max Princeton
Ā 
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
Ā 
28March2024-Codeless-Generative-AI-Pipelines
28March2024-Codeless-Generative-AI-Pipelines28March2024-Codeless-Generative-AI-Pipelines
28March2024-Codeless-Generative-AI-Pipelines
Ā 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
Ā 
2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits
Ā 
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Ā 
Conf42-Python-Building Apache NiFi 2.0 Python Processors
Conf42-Python-Building Apache NiFi 2.0 Python ProcessorsConf42-Python-Building Apache NiFi 2.0 Python Processors
Conf42-Python-Building Apache NiFi 2.0 Python Processors
Ā 
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Ā 
2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines
2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines
2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines
Ā 
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and FlinkDBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
Ā 
NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...
NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...
NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...
Ā 
OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines
OSACon 2023_ Unlocking Financial Data with Real-Time PipelinesOSACon 2023_ Unlocking Financial Data with Real-Time Pipelines
OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines
Ā 
Building Real-Time Travel Alerts
Building Real-Time Travel AlertsBuilding Real-Time Travel Alerts
Building Real-Time Travel Alerts
Ā 
JConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and FlinkJConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and Flink
Ā 
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
Ā 
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines DemoEvolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Ā 
AIDevWorldApacheNiFi101
AIDevWorldApacheNiFi101AIDevWorldApacheNiFi101
AIDevWorldApacheNiFi101
Ā 
26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup
26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup
26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup
Ā 

Recently uploaded

Abortion Pill Prices Tembisa [(+27832195400*)] šŸ„ Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] šŸ„ Women's Abortion Clinic in T...Abortion Pill Prices Tembisa [(+27832195400*)] šŸ„ Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] šŸ„ Women's Abortion Clinic in T...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
Ā 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
masabamasaba
Ā 
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
chiefasafspells
Ā 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
masabamasaba
Ā 
The title is not connected to what is inside
The title is not connected to what is insideThe title is not connected to what is inside
The title is not connected to what is inside
shinachiaurasa2
Ā 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Health
Ā 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
VictoriaMetrics
Ā 
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
masabamasaba
Ā 

Recently uploaded (20)

Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Ā 
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
Ā 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
Ā 
Abortion Pill Prices Tembisa [(+27832195400*)] šŸ„ Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] šŸ„ Women's Abortion Clinic in T...Abortion Pill Prices Tembisa [(+27832195400*)] šŸ„ Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] šŸ„ Women's Abortion Clinic in T...
Ā 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
Ā 
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Ā 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
Ā 
Shapes for Sharing between Graph Data SpacesĀ - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data SpacesĀ - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data SpacesĀ - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data SpacesĀ - and Epistemic Querying of RDF-...
Ā 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
Ā 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learn
Ā 
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
Ā 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand
Ā 
%in Soweto+277-882-255-28 abortion pills for sale in soweto
%in Soweto+277-882-255-28 abortion pills for sale in soweto%in Soweto+277-882-255-28 abortion pills for sale in soweto
%in Soweto+277-882-255-28 abortion pills for sale in soweto
Ā 
The title is not connected to what is inside
The title is not connected to what is insideThe title is not connected to what is inside
The title is not connected to what is inside
Ā 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Ā 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Ā 
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
Ā 
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
Ā 
tonesoftg
tonesoftgtonesoftg
tonesoftg
Ā 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
Ā 

Fast data for fitness 10 nov 2020

  • 1. Fast Data for Fitness Real-time Health Sensor Ingest Timothy Spann - Principal DataFlow Field Engineer
  • 2.
  • 3. Ā© 2020 Cloudera, Inc. All rights reserved. 3 Timothy Spann Data in Motion Principal Field Engineer @PaasDev DZone Zone Leader and Big Data MVB Princeton NJ Future of Data Meetup https://github.com/tspannhw https://www.datainmotion.dev/ https://meetup.com/futureofdata-princeton/
  • 4. Ā© 2020 Cloudera, Inc. All rights reserved. 4 Data Sources - From Images to Sub-Second Data Events Heart Rate Temperature Blood Sugar Altimeter Blood Pressure Weight GPS Live Video/Picture AI for Pose Estimation Improvement to Technique
  • 5. Ā© 2020 Cloudera, Inc. All rights reserved. 5 Data for Fitness+ Pipeline Weather Weather Climate Aggregates Sensors SQL Analytics Sources Pollution Wearables REST
  • 6. Ā© 2020 Cloudera, Inc. All rights reserved. 6 Fitness+ Data Streaming Pipeline Device Data SensorsLogs Weather Sensors Aggregates Energy SQL Analytics MiNiFi Agent Deep Learning Classiļ¬cation Edge Private Cloud Multi-Public Cloud
  • 7. Ā© 2020 Cloudera, Inc. All rights reserved. 7 Fast Fitness+ Data
  • 8. Ā© 2020 Cloudera, Inc. All rights reserved. 8 CLOUDERA FLOW AND EDGE MANAGEMENT Enable easy ingestion, routing, management and delivery of any data anywhere (Edge, cloud, data center) to any downstream system with built in end-to-end security and provenance Advanced tooling to industrialize ļ¬‚ow development (Flow Development Life Cycle) ACQUIRE ā€¢ Over 300 Prebuilt Processors ā€¢ Easy to build your own ā€¢ Parse, Enrich & Apply Schema ā€¢ Filter, Split, Merger & Route ā€¢ Throttle & Backpressure FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLOG PROCESS HASH MERGE EXTRACT DUPLICATE SPLIT ENCRYPT TALL EVALUATE EXECUTE GEOENRICH SCAN REPLACE TRANSLATE CONVERT ROUTE TEXT ROUTE CONTENT ROUTE CONTEXT ROUTE RATE DISTRIBUTE LOAD DELIVER ā€¢ Guaranteed Delivery ā€¢ Full data provenance from acquisition to delivery ā€¢ Diverse, Non-Traditional Sources ā€¢ Eco-system integration FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLOG
  • 9. Ā© 2019 Cloudera, Inc. All rights reserved. 9 Apache Kafka ā€¢ Highly reliable distributed messaging system ā€¢ Decouple applications, enables many-to-many patterns ā€¢ Publish-Subscribe semantics ā€¢ Horizontal scalability ā€¢ Eļ¬ƒcient implementation to operate at speed with big data volumes ā€¢ Organized by topic to support several use cases Source System Source System Source System Kafka Fraud Detection Security Systems Real-Time Monitoring Source System Source System Source System Fraud Detection Security Systems Real-Time Monitoring Many-To-Many Publish-Subscribe Point-To-Point Request-Response
  • 10. Ā© 2020 Cloudera, Inc. All rights reserved. 10 New features delivered with Cloudera Streaming Analytics (CSA) 1.2 Next Generation Streaming Analytics Flink SQL Support Agile Streaming App Development using SQL New Flink Atlas Hook Capture operational Flink app metadata and lineage Single View of Flink Yarn Jobs Improve Developer Experience & operational visibility
  • 11. Ā© 2020 Cloudera, Inc. All rights reserved. 11 ā— https://www.cloudera.com/tutorials/building-a-sentiment-analysis-application/3.html ā— https://blog.cloudera.com/benchmarking-niļ¬-performance-and-scalability/ ā— https://www.datainmotion.dev/2020/06/no-more-spaghetti-ļ¬‚ows.html ā— https://www.datainmotion.dev/2020/06/the-rise-of-mega-edge-ļ¬‚ank.html ā— https://www.datainmotion.dev/2020/01/cloudera-edge2ai-miniļ¬-java-agent-with.html ā— https://www.datainmotion.dev/2019/08/rapid-iot-development-with-cloudera.html ā— https://www.datainmotion.dev/2019/09/powering-edge-ai-for-sensor-reading.html ā— https://www.datainmotion.dev/2019/05/dataworks-summit-dc-2019-report.html ā— https://www.datainmotion.dev/2019/03/using-raspberry-pi-3b-with-apache-niļ¬.html ā— https://www.datainmotion.dev/2020/06/unboxing-most-amazing-edge-ai-device.html ā— https://www.datainmotion.dev/2019/08/edge-processing-with-jetson-nano-part-3.html ā— https://www.datainmotion.dev/2020/04/predicting-sensor-readings-with-time.html REFERENCES
  • 12. Ā© 2020 Cloudera, Inc. All rights reserved. 12 TH N Y U