Fast data for fitness 10 nov 2020

•

1 like•396 views

This document discusses fast data for fitness applications using real-time health sensor data. It describes various data sources like heart rate, temperature, and GPS that can be ingested from sensors. It then outlines a data pipeline for fitness applications using technologies like Apache Kafka, Cloudera Flow, streaming analytics, and edge computing. This pipeline collects data from devices and sensors, processes it using tools like SQL and analytics, and delivers it to cloud and edge systems for applications.

Software

Fast Data for Fitness
Real-time Health Sensor Ingest
Timothy Spann - Principal DataFlow Field Engineer

© 2020 Cloudera, Inc. All rights reserved. 3
Timothy Spann
Data in Motion Principal Field Engineer
@PaasDev
DZone Zone Leader and Big Data MVB
Princeton NJ Future of Data Meetup
https://github.com/tspannhw https://www.datainmotion.dev/
https://meetup.com/futureofdata-princeton/

© 2020 Cloudera, Inc. All rights reserved. 4
Data Sources - From Images to Sub-Second Data Events
Heart Rate
Temperature
Blood Sugar
Altimeter
Blood Pressure
Weight
GPS
Live Video/Picture
AI for Pose Estimation
Improvement to Technique

© 2020 Cloudera, Inc. All rights reserved. 5
Data for Fitness+ Pipeline
Weather
Weather
Climate
Aggregates
Sensors
SQL
Analytics
Sources
Pollution
Wearables
REST

© 2020 Cloudera, Inc. All rights reserved. 6
Fitness+ Data Streaming Pipeline
Device Data
SensorsLogs
Weather
Sensors
Aggregates
Energy
SQL
Analytics
MiNiFi
Agent
Deep Learning
Classiﬁcation
Edge Private
Cloud
Multi-Public
Cloud

© 2020 Cloudera, Inc. All rights reserved. 7
Fast Fitness+ Data

© 2020 Cloudera, Inc. All rights reserved. 8
CLOUDERA FLOW AND EDGE MANAGEMENT
Enable easy ingestion, routing, management and delivery of any data anywhere (Edge, cloud, data
center) to any downstream system with built in end-to-end security and provenance
Advanced tooling to industrialize
ﬂow development (Flow Development
Life Cycle)
ACQUIRE
• Over 300 Prebuilt Processors
• Easy to build your own
• Parse, Enrich & Apply Schema
• Filter, Split, Merger & Route
• Throttle & Backpressure
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLOG
PROCESS
HASH
MERGE
EXTRACT
DUPLICATE
SPLIT
ENCRYPT
TALL
EVALUATE
EXECUTE
GEOENRICH
SCAN
REPLACE
TRANSLATE
CONVERT
ROUTE TEXT
ROUTE CONTENT
ROUTE CONTEXT
ROUTE RATE
DISTRIBUTE LOAD
DELIVER
• Guaranteed Delivery
• Full data provenance from
acquisition to delivery
• Diverse, Non-Traditional Sources
• Eco-system integration
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLOG

© 2019 Cloudera, Inc. All rights reserved. 9
Apache Kafka
• Highly reliable distributed
messaging system
• Decouple applications, enables
many-to-many patterns
• Publish-Subscribe semantics
• Horizontal scalability
• Eﬃcient implementation to
operate at speed with big data
volumes
• Organized by topic to support
several use cases
Source
System
Source
System
Source
System
Kafka
Fraud
Detection
Security
Systems
Real-Time
Monitoring
Source
System
Source
System
Source
System
Fraud
Detection
Security
Systems
Real-Time
Monitoring
Many-To-Many
Publish-Subscribe
Point-To-Point
Request-Response

© 2020 Cloudera, Inc. All rights reserved. 10
New features delivered with Cloudera Streaming Analytics (CSA) 1.2
Next Generation Streaming Analytics
Flink SQL Support
Agile Streaming App
Development using SQL
New Flink Atlas Hook
Capture operational Flink
app metadata and lineage
Single View of Flink Yarn Jobs
Improve Developer Experience
& operational visibility

© 2020 Cloudera, Inc. All rights reserved. 11
● https://www.cloudera.com/tutorials/building-a-sentiment-analysis-application/3.html
● https://blog.cloudera.com/benchmarking-niﬁ-performance-and-scalability/
● https://www.datainmotion.dev/2020/06/no-more-spaghetti-ﬂows.html
● https://www.datainmotion.dev/2020/06/the-rise-of-mega-edge-ﬂank.html
● https://www.datainmotion.dev/2020/01/cloudera-edge2ai-miniﬁ-java-agent-with.html
● https://www.datainmotion.dev/2019/08/rapid-iot-development-with-cloudera.html
● https://www.datainmotion.dev/2019/09/powering-edge-ai-for-sensor-reading.html
● https://www.datainmotion.dev/2019/05/dataworks-summit-dc-2019-report.html
● https://www.datainmotion.dev/2019/03/using-raspberry-pi-3b-with-apache-niﬁ.html
● https://www.datainmotion.dev/2020/06/unboxing-most-amazing-edge-ai-device.html
● https://www.datainmotion.dev/2019/08/edge-processing-with-jetson-nano-part-3.html
● https://www.datainmotion.dev/2020/04/predicting-sensor-readings-with-time.html
REFERENCES

© 2020 Cloudera, Inc. All rights reserved. 12
TH N Y U

What's hot

Cybersecurity is a broad topic and many commercial products are related to it. We demonstrate a fundamental concept in network analysis: re-construction and visualization of temporal networks. Furthermore, we apply the method to describe operational conditions of a Hadoop cluster. Our experiments provide first results and allow a classification of the cluster state related to current workloads. The temporal networks show significant differences for different operation modes. In reallity we would expect mixed workloads. If such workload parameters are known, we are able to handle a-typical events accordingly - which means, we are able to create alerts based on context information, rather than only the package content. We show an end-to-end example: (1) Data collection is done via python, using the sniffer script; (2) using Apache Hive and Apache Spark we analyze the network traffic data and create the temporary network. Finally, we are able to visualize the results using Gephi in step (3). In a next step, we plan to contribute to the Apache Spot project.

PCAP Graphs for Cybersecurity and System Tuning

Dr. Mirko Kämpf

Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance. This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

HostedbyConfluent

Enterprise Metadata Integration

Dr. Mirko Kämpf

Apache Kafka is a popular distributed streaming data platform and more and more is the architectural backbone for integrating streaming data with a Data Lake, Microservices and Stream Processing. A lot of data necessary in stream processing is stored in traditional systems backed by relational databases. This session will present different approaches for integrating relational databases with Kafka, such as Kafka Connect, Oracle GoldenGate, ORDS APIs and bridging Kafka with Oracle AQ.

Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka

Guido Schmutz

Most data visualisation solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualisation capabilities. One was is to first persist the data into a data store and then use a traditional data visualisation solution to present the data. If latency is not an issue, such a solution might be good enough. An other question is which data store solution is necessary to keep up with the high load on write and read. If it is not an RDBMS but an NoSQL database, then not all traditional visualisation tools might already integrate with the specific data store. An other option is to use a Streaming Visualisation solution. They are specially built for streaming data and often do not support batch data. A much better solution would be to have one tool capable of handling both, batch and streaming data. This talk presents different architecture blueprints for integrating data visualisation into a fast data solution and highlights some of the products available to implement these blueprints.

Streaming Visualization

Guido Schmutz

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)

Kai Wähner

Rediscovering the Value of Apache Kafka® in Modern Data Architecture

confluent

Most data visualisation solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualisation capabilities. One option is to first persist the data into a data store and then use a traditional data visualisation solution to present the data. If latency is not an issue, such a solution might be good enough. An other question is which data store solution is necessary to keep up with the high load on write and read. If it is not an RDBMS but an NoSQL database, then not all traditional visualisation tools might already integrate with the specific data store. An other option is to use a Streaming Visualisation solution. They are specially built for streaming data and often do not support batch data. A much better solution would be to have one tool capable of handling both, batch and streaming data. This talk presents different architecture blueprints for integrating data visualisation into a fast data solution and then we show how the different blueprints can be implemented by mapping products onto the blueprints.

Streaming Visualization

Guido Schmutz

Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

HostedbyConfluent

Apache Spark in Scientific Applciations

Dr. Mirko Kämpf

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

confluent

Internet of Things use cases are a perfect match for processing with a streaming platform such as Kafka and the Confluent Platform. Some of the questions to be answered are: How do we feed the data from our devices into Kafka? Do we directly send data to Kafka? Is Kafka accessible from outside the organization over the internet? What if we want to use a more specific IoT protocol such as MQTT or CoAP in between? How would we integrate it with Kafka? How can we enrich IoT streaming data with static data sitting in a traditional system? This session will provide answers to these and other questions using a fictitious use case of a trucking company. Trucks are constantly sending data about position and driving habits, which can be used to derive real-time information and actions. A large part of the presentation will be a live demo. The demo will show the implementation of the pipeline incrementally: starting with sending the truck movement events directly to Kafka, then adding MQTT to the sensor data ingestion, followed by using Kafka Streams and KSQL to apply stream processing on the information received. The final pipeline will demonstrate the application of Kafka Connect with MQTT and JDBC source connectors for data ingestion and event stream enrichment, and Kafka Streams and KSQL for stream processing. The key takeaway is the live demonstration of a working end-to-end IoT streaming data ingestion pipeline using Kafka technologies.

Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL

Guido Schmutz

What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

Building Event Driven (Micro)services with Apache Kafka

Guido Schmutz

Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will dive into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

Building event-driven (Micro)Services with Apache Kafka Ecosystem

Guido Schmutz

Advanced time series analysis (TSA) requires very special data preparation procedures to convert raw data into useful and compatible formats. In this presentation you will see some typical processing patterns for time series based research, from simple statistics to reconstruction of correlation networks. The first case is relevant for anomaly detection and to protect safety. Reconstruction of graphs from time series data is a very useful technique to better understand complex systems like supply chains, material flows in factories, information flows within organizations, and especially in medical research. With this motivation we will look at typical data aggregation patterns. We investigate how to apply analysis algorithms in the cloud. Finally we discuss a simple reference architecture for TSA on top of the Confluent Platform or Confluent cloud.

Time Series Analysis Using an Event Streaming Platform

Dr. Mirko Kämpf

Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Storing such huge event streams into HDFS or a NoSQL datastore is feasible and not such a challenge anymore. But if you want to be able to react fast, with minimal latency, you can not afford to first store the data and doing the analysis/analytics later. You have to be able to include part of your analytics right after you consume the data streams. Products for doing event processing, such as Oracle Event Processing or Esper, are avaialble for quite a long time and used to be called Complex Event Processing (CEP). In the past few years, another family of products appeared, mostly out of the Big Data Technology space, called Stream Processing or Streaming Analytics. These are mostly open source products/frameworks such as Apache Storm, Spark Streaming, Flink, Kafka Streams as well as supporting infrastructures such as Apache Kafka. In this talk I will present the theoretical foundations for Stream Processing, discuss the core properties a Stream Processing platform should provide and highlight what differences you might find between the more traditional CEP and the more modern Stream Processing solutions.

Introduction to Stream Processing

Guido Schmutz

Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target. This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.

Apache Kafka - Scalable Message-Processing and more !

Guido Schmutz

Streaming and Social Media

Joe Olson

Cloud Connect 2012, Big Data @ Netflix

Jerome Boulon

Building Event Driven (Micro)services with Apache Kafka

Guido Schmutz

What's hot (20)

PCAP Graphs for Cybersecurity and System Tuning

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Enterprise Metadata Integration

Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka

Streaming Visualization

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)

Rediscovering the Value of Apache Kafka® in Modern Data Architecture

Streaming Visualization

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

Apache Spark in Scientific Applciations

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL

Building Event Driven (Micro)services with Apache Kafka

Building event-driven (Micro)Services with Apache Kafka Ecosystem

Time Series Analysis Using an Event Streaming Platform

Introduction to Stream Processing

Apache Kafka - Scalable Message-Processing and more !

Streaming and Social Media

Cloud Connect 2012, Big Data @ Netflix

Building Event Driven (Micro)services with Apache Kafka

Similar to Fast data for fitness 10 nov 2020

Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase. Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs. Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables. Resources: https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

DataWorks Summit

Meetup: Streaming Data Pipeline Development In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns. He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg. If you wish to follow along, please download open source projects beforehand. You can also download this helpful streaming platform: https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html All source code and slides will be shared for those interested in building their own FLaNK Apps. https://www.flankstack.dev/ You can join the meeting virtually here: https://cloudera.zoom.us/j/91603330726 Speaker - Tim Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

Meetup: Streaming Data Pipeline Development

Timothy Spann

IBM Aspera overview

Carlos Martin Hernandez

Simplifying Real-Time Architectures for IoT with Apache Kudu

Cloudera, Inc.

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines https://www.youtube.com/watch?v=Yeua8NlzQ3Y https://www.conf42.com/Large_Language_Models_LLMs_2024_Tim_Spann_generative_ai_streaming Adding Generative AI to Real-Time Streaming Pipelines Abstract Let’s build streaming pipelines that convert streaming events into prompts, call LLMs, and process the results. Summary Tim Spann: My talk is adding generative AI to real time streaming pipelines. I'm going to discuss a couple of different open source technologies. We'll touch on Kafka, Nifi, Flink, Python, Iceberg. All the slides, all the code and GitHub are out there. Llm, if you didn't know, is rapidly evolving. There's a lot of different ways to interact with models. That enrichment, transformation, processing really needs tools. The amount of models and projects and software that are available is massive. Nifi supports hundreds of different inputs and can convert them on the fly. Great way to distribute your data quickly to whoever needs it without duplication, without tight coupling. Fun to find new things to integrate into. So what we can do is, well, I want to get a meetup chat going. I have a processor here that just listens for events as they come from slack. And then I'm going to clean it up, add a couple fields and push that out to slack. Every model is a little bit of different tweaking. Nifi acts as a whole website. And as you see here, it can be get, post, put, whatever you want. We send that response back to flink and it shows up here. Thank you for attending this talk. I'm going to be speaking at some other events very shortly. Transcript This transcript was autogenerated. To make changes, submit a PR. Hi, Tim Spann here. My talk is adding generative AI to real time streaming pipelines, and we're here for the large language model conference at Comp 42, which is always a nice one, great place to be. I'm going to discuss a couple of different open source technologies that work together to enable you to build real time pipelines using large language models. So we'll touch on Kafka, Nifi, Flink, Python, Iceberg, and I'll show you a little bit of each one in the demos. I've been working with data machine learning, streaming IoT, some other things for a number of years, and you could contact me at any of these places, whether Twitter or whatever it's called, some different blogs, or in person at my meetups and at different conferences around the world. I do a weekly newsletter, cover streaming ML, a lot of LLM, open source, Python, Java, all kinds of fun stuff, as I mentioned, do a bunch of different meetups. They are not just in the east coast of the US, they are available virtually live, and I also put them on YouTube, and if you need them somewhere else, let me know. We publish all the slides, all the code and GitHub. Everything you need is out there. Let's get into the talk. Llm, if you didn't know, is rapidly evolving. While you're typing down the things that you use, it

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines

Timothy Spann

Unconference Round Table Notes

Timothy Spann

The Internet of Things is moving into the mainstream and this new world of data-driven products is transforming a vast number of industry sectors and technologies. However, IoT creates a new challenge: how to build and operationalize continual data ingestion from such a wide and ever-changing array of endpoints so that the data arrives consumption-ready and can drive analysis and action within the business. In this webinar, Sean Anderson from Cloudera and Kirit Busu, Director of Product Management at StreamSets, will discuss Hadoop's ecosystem and IoT capabilities and provide advice about common patterns and best practices. Using specific examples, they will demonstrate how to build and run end-to-end IOT data flows using StreamSets and Cloudera infrastructure.

How to Build Continuous Ingestion for the Internet of Things

Cloudera, Inc.

Blockchain and Apache NiFi

Timothy Spann

Introduction to the Data-Distribution Service (DDS): Context and Applications. This 50 minute presentation summarizes the main features of DDS including the information model, the type system, and security as well as how typical applications use DDS. It was presented at the Canadian Government Information Day in Ottawa on September 2018. There is also a video of this presentation at https://www.youtube.com/watch?v=6iICap5G7rw.

Introduction to DDS: Context, Information Model, Security, and Applications.

Gerardo Pardo-Castellote

Big Data LDN 2016: When Big Data Meets Fast Data

Matt Stubbs

PRGX is the world's leading provider of accounts payable audit services and works with leading global retailers. As new forms of data started to flow into their organizations, standard RDBMS systems were not allowing them to scale. Now, by using Talend with Cloudera Enterprise, they are able to acheive a 9-10x performance benefit in processing data, reduce errors, and now provide more innovative products and services to end customers. Watch this webinar to learn how PRGX worked with Cloudera and Talend to create a high-performance computing platform for data analytics and discovery that rapidly allows them to process, model, and serve massive amount of structured and unstructured data.

Turning Petabytes of Data into Profit with Hadoop for the World’s Biggest Ret...

Cloudera, Inc.

IoT Connected Brewery

Jason Hubbard

Gen AI on Enterprise Cloud Apache NiFi Milvus Apache Kafka Apache Flink Cloudera Machine Learning Cloudera DataFlow https://medium.com/@tspann/building-a-milvus-connector-for-nifi-34372cb3c7fa https://www.meetup.com/futureofdata-princeton/events/300737266/ https://lu.ma/q7pcfyjn?source=post_page-----34372cb3c7fa--------------------------------&tk=TTyakY If you're interested in working with Generative AI on the cloud, this virtual workshop is for you. Tim Spann from Cloudera and Yujian Tang from Zilliz will cover how you can implement your own GenAI workflows on the cloud at enterprise scale. 9:00 - 9:05: Intro 9:05 - 9:15: What is Milvus 9:15 - 9:25: Cloudera Development Platform 9:25 - 10:00: Demo Location https://www.youtube.com/watch?v=IfWIzKsoHnA https://github.com/tspannhw/SpeakerProfile https://www.linkedin.com/in/yujiantang/

Generative AI on Enterprise Cloud with NiFi and Milvus

Timothy Spann

Global Data Stream Network for Internet of Things

Bhavana Srinivas

The blessing and curse of today's database market? So many choices! While relational databases still dominate the day-to-day business, a host of alternatives has evolved around very specific use cases: graph, document, NoSQL, hybrid (HTAP), column store, the list goes on. And the database tools market is teeming with activity as well. Register for this special Research Webcast to hear Dr. Robin Bloor share his early findings about the evolving database market. He'll be joined by Steve Sarsfield of HPE Vertica, and Robert Reeves of Datical in a roundtable discussion with Bloor Group CEO Eric Kavanagh. Send any questions to info@insideanalysis.com, or tweet with #DBSurvival.

Horses for Courses: Database Roundtable

Eric Kavanagh

IoT Crash Course Hadoop Summit SJ

Daniel Madrigal

Solving Big Data Problems using Hortonworks

DataWorks Summit/Hadoop Summit

Fighting cyber fraud with hadoop v2

Niel Dunnage

Let's build a simple ingest to cloud datawarehouse with low code

Timothy Spann

Gimel and PayPal Notebooks @ TDWI Leadership Summit Orlando

Romit Mehta

Similar to Fast data for fitness 10 nov 2020 (20)

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

Meetup: Streaming Data Pipeline Development

IBM Aspera overview

Simplifying Real-Time Architectures for IoT with Apache Kudu

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines

Unconference Round Table Notes

How to Build Continuous Ingestion for the Internet of Things

Blockchain and Apache NiFi

Introduction to DDS: Context, Information Model, Security, and Applications.

Big Data LDN 2016: When Big Data Meets Fast Data

Turning Petabytes of Data into Profit with Hadoop for the World’s Biggest Ret...

IoT Connected Brewery

Generative AI on Enterprise Cloud with NiFi and Milvus

Global Data Stream Network for Internet of Things

Horses for Courses: Database Roundtable

IoT Crash Course Hadoop Summit SJ

Solving Big Data Problems using Hortonworks

Fighting cyber fraud with hadoop v2

Let's build a simple ingest to cloud datawarehouse with low code

Gimel and PayPal Notebooks @ TDWI Leadership Summit Orlando

More from Timothy Spann

Building Real-Time Pipelines With FLaNK Timothy Spann, Principal Developer Advocate, Streaming - Cloudera Future of Data meetup, startup grind, AI Camp The combination of Apache Flink, Apache NiFi, and Apache Kafka for building real-time data processing pipelines is extremely powerful, as demonstrated by this case study using the FLaNK-MTA project. The project leverages these technologies to process and analyze real-time data from the New York City Metropolitan Transportation Authority (MTA). FLaNK-MTA demonstrates how to efficiently collect, transform, and analyze high-volume data streams, enabling timely insights and decision-making. Apache NiFi Apache Kafka Apache Flink Apache Iceberg LLM Generative AI Slack Postgresql

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

Timothy Spann

April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024

Timothy Spann

Real-Time AI Streaming - AI Max Princeton

Timothy Spann

2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...

Timothy Spann

28March2024-Codeless-Generative-AI-Pipelines https://www.meetup.com/futureofdata-princeton/events/299440871/ https://www.meetup.com/real-time-analytics-meetup-ny/events/299290822/ ******Note***** The event is seat-limited, therefore please complete your registration here. Only people completing the form will be able to attend. ----------------------- We're excited to invite you to join us in-person, for a Real-Time Analytics exploration! Join us for an evening of insights, networking as we delve into the OSS technologies shaping the field! Agenda: 05:30-06:00: Pizza and friends 06:00- 06:40: Codeless GenAI Pipelines with Flink, Kafka, NiFi 06:40- 07:20 Real-Time Analytics in the Corporate World: How Apache Pinot® Powers Industry Leaders 07:20-07:30 QNA Codeless GenAI Pipelines with Flink, Kafka, NiFi | Tim Spann, Cloudera Explore the power of real-time streaming with GenAI using Apache NiFi. Learn how NiFi simplifies data engineering workflows, allowing you to focus on creativity over technical complexities. I'll guide you through practical examples, showcasing NiFi's automation impact from ingestion to delivery. Whether you're a seasoned data engineer or new to GenAI, this talk offers valuable insights into optimizing workflows. Join us to unlock the potential of real-time streaming and witness how NiFi makes data engineering a breeze for GenAI applications! Real-Time Analytics in the Corporate World: How Apache Pinot® Powers Industry Leaders | Viktor Gamov, StarTree Explore how industry leaders like LinkedIn, Uber Eats, and Stripe are mastering real-time data with Viktor as your guide. Discover how Apache Pinot transforms data into actionable insights instantly. Viktor will showcase Pinot's features, including the Star-Tree Index, and explain why it's a game-changer in data strategy. This session is for everyone, from data geeks to business gurus, eager to uncover the future of tech. Join us and be wowed by the power of real-time analytics with Apache Pinot! ------- Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more.

28March2024-Codeless-Generative-AI-Pipelines

Timothy Spann

https://princetonacm.acm.org/tcfpro/ 18th Annual IEEE IT Professional Conference (ITPC) Armstrong Hall at The College of New Jersey Friday, March 15th, 2024 | 10:00 AM to 5:00 PM IT Professional Conference at Trenton Computer Festival IEEE Information Technology Professional Conference on Friday, March 15th, 2024 TCFPro24 Building Real-Time Generative AI Pipelines Building Real-Time Generative AI Pipelines In this talk, Tim will delve into the exciting realm of building real-time generative AI pipelines with streaming capabilities. The discussion will revolve around the integration of cutting-edge technologies to create dynamic and responsive systems that harness the power of generative algorithms. From leveraging streaming data sources to implementing advanced machine learning models, the presentation will explore the key components necessary for constructing a robust real-time generative AI pipeline. Practical insights, use cases, and best practices will be shared, offering a comprehensive guide for developers and data scientists aspiring to design and implement dynamic AI systems in a streaming environment. Tim will show a live demo showing we can use Apache NiFi to provide a live chat between a person in Slack and several LLM models all orchestrated with Apache NiFi, Apache Kafka and Python. We will use RAG against Chroma and Pinecone vector data stores, Hugging Face and WatsonX.AI LLM, and add additional context with NiFi lookups of stocks, weather and other data streams in real-time. Timothy Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

TCFPro24 Building Real-Time Generative AI Pipelines

Timothy Spann

2024 Build Generative AI for Non-Profits

Timothy Spann

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipelines https://www.meetup.com/futureofdata-newyork/events/298660453/ Unlocking Financial Data with Real-Time Pipelines (Flink Analytics on Stocks with SQL ) By Timothy Spann Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In this talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processing falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. I will be utilizing NiFi 2.0 with Python and Vector Databases. Timothy Spann Principal Developer Advocate, Cloudera Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. https://twitter.com/PaaSDev https://www.linkedin.com/in/timothyspann/ https://medium.com/@tspann https://github.com/tspannhw/FLiPStackWeekly/

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...

Timothy Spann

Conf42-Python-Building Apache NiFi 2.0 Python Processors https://www.conf42.com/Python_2024_Tim_Spann_apache_nifi_2_processors Building Apache NiFi 2.0 Python Processors Abstract Let’s enhance real-time streaming pipelines with smart Python code. Adding code for vector databases and LLM. Summary Tim Spann: I'm going to be talking today, be building Apache 9520 Python processors. One of the main purposes of supporting Python in the streaming tool Apache Nifi is to interface with new machine learning and AI and Gen AI. He says Python is a real game changer for Cloudera. You're just going to add some metadata around it. It's a great way to pass a file along without changing it too substantially. We really need you to have Python 310 and again JDK 21 on your machine. You got to be smart about how you use these models. There are a ton of python processors available. You can use them in multiple ways. We're still in the early world of Python processors, so now's the time to start putting yours out there. Love to see a lot of people write their own. When we are parsing documents here, again, this is the Python one I'm picking PDF. Lots of different things you could do. If you're interested on writing your own python code for Apache Nifi, definitely reach out and thank.

Conf42-Python-Building Apache NiFi 2.0 Python Processors

Timothy Spann

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg with Stock Data and LLM Abstract In this talk, we’ll discuss how to use Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg to process and analyze stock data. We demonstrated the ingestion, processing, and analysis of stock data. Additionally, we illustrated how to use an LLM to generate predictions from the analyzed data. Karin Wolok Developer Relations, Dev Marketing, and Community Programming @ Project Elevate Karin Wolok's LinkedIn account Karin Wolok's twitter account Tim Spann Principal Developer Advocate @ Cloudera Tim Spann's LinkedIn account Tim Spann's twitter account https://www.conf42.com/Python_2024_Karin_Wolok_Tim_Spann_nifi__kafka_risingwave_iceberg_llm

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...

Timothy Spann

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines https://www.aicamp.ai/event/eventdetails/W2024022214 apache nifi llm generative ai gen ai ml dl machine learning apache kafka apache flink postgresql python AI Meetup (NYC): GenAI, LLMs, ML and Data Feb 22, 05:30 PM EST Welcome to the monthly in-person AI meetup in New York City, in collaboration with Microsoft. Join us for deep dive tech talks on AI, GenAI, LLMs and machine learning, food/drink, networking with speakers and fellow developers Agenda: * 5:30pm~6:00pm: Checkin, Food/drink and networking * 6:00pm~6:10pm: Welcome/community update * 6:10pm~8:30pm: Tech talks * 8:30pm: Q&A, Open discussion Tech Talk: Searching and Reasoning Over Multimedia Data with Vector Databases and LMMs Speaker: Zain Hasan (Weaviate LinkedIn) Abstract: In this talk, Zain Hasan will discuss how we can use open-source multimodal embedding models in conjunction with large generative multimodal models that can that can see, hear, read, and feel data(!), to perform cross-modal search(searching audio with images, videos with text etc.) and multimodal retrieval augmented generation (MM-RAG) at the billion-object scale with the help of open source vector databases. I will also demonstrate, with live code demos, how being able to perform this cross-modal retrieval in real-time can enables users to use LLMs that can reason over their enterprise multimodal data. This talk will revolve around how we can scale the usage of multimodal embedding and generative models in production. Tech Talk: Codeless Generative AI Pipelines Speaker: Timothy Spann (Cloudera LinkedIn) Abstract: Join us for an insightful talk on leveraging the power of real-time streaming tools, specifically Apache NiFi, to revolutionize GenAI data engineering. In this session, we’ll explore how the integration of Apache NiFi can automate the entire process of prompt building, making it a seamless and efficient task. Speakers/Topics: Stay tuned as we are updating speakers and schedules. If you have a keen interest in speaking to our community, we invite you to submit topics for consideration: Submit Topics Sponsors: We are actively seeking sponsors to support our community. Whether it is by offering venue spaces, providing food/drink, or cash sponsorship. Sponsors will have the chance to speak at the meetups, receive prominent recognition, and gain exposure to our extensive membership base of 20,000+ local or 300K+ developers worldwide. Venue: Microsoft NYC - Times Square, 11 Times Square, New York, NY 10036 Room Name: Central Park West 6501 Community on Slack/Discord - Event chat: chat and connect with speakers and attendees - Sharing blogs, events, job openings, projects collaborations Join Slack (search and join the #newyork channel) | Join Discord

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines

Timothy Spann

DBA Fundamentals Group: Continuous SQL with Kafka and Flink 20-Feb-2024 In this talk, I will walk through how someone can set up and run continuous SQL queries against Kafka topics utilizing Apache Flink. We will walk through creating Kafka topics, schemas, and publishing data. We will then cover consuming Kafka data, joining Kafka topics, and inserting new events into Kafka topics as they arrive. This basic overview will show hands-on techniques, tips, and examples of how to do this. Tim Spann Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.

DBA Fundamentals Group: Continuous SQL with Kafka and Flink

Timothy Spann

NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...

Timothy Spann

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines Unlocking Financial Data with Real-Time Pipelines Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In this talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processing falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. Key Points to be Covered: Introduction to Real-Time Data Pipelines: a. The limitations of traditional batch processing in the financial domain. b. Understanding the need for real-time data processing. Apache Flink: Powering Real-Time Stream Processing: a. Overview of Apache Flink and its role in real-time stream processing. b. Use cases for Apache Flink in the financial industry. c. How Flink enables fast, scalable, and fault-tolerant processing of streaming financial data. Apache Kafka: Building Resilient Event Streaming Platforms: a. Introduction to Apache Kafka and its role as a distributed streaming platform. b. Kafka's capabilities in handling high-throughput, fault-tolerant, and real-time data streaming. c. Integration of Kafka with financial data sources and consumers. Apache NiFi: Data Ingestion and Flow Management: a. Overview of Apache NiFi and its role in data ingestion and flow management. b. Data integration and transformation capabilities of NiFi for financial data. c. Utilizing NiFi to collect and process financial data from diverse sources. Iceberg: Efficient Data Lake Management: a. Understanding Iceberg and its role in managing large-scale data lakes. b. Iceberg's schema evolution and table-level metadata capabilities. c. How Iceberg simplifies data lake management in financial institutions. Real-World Use Cases: a. Real-time fraud detection using Flink, Kafka, and NiFi. b. Portfolio risk analysis with Iceberg and Flink. c. Streamlined regulatory reporting leveraging all four technologies. Best Practices and Considerations: a. Architectural considerations when building real-time financial data pipelines. b. Ensuring data integrity, security, and compliance in real-time pipelines. c. Scalability an

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines

Timothy Spann

Building Real-time Travel Alerts In this session, we will walk through how to build a complete streaming application to send alerts based on travel advisories from public data. We will also join in other data sources of relevance and push out alerts. We will show you how to build this streaming application with Apache NiFi, Apache Kafka, and Apache Flink and show you when/why/how, and what to build to maximize performance, productivity, and ease of development. Let's get streaming. Apache Flink Apache Kafka Apache NiFi FLaNK Stack Tim Spann Big Data Conference Europe 2023

Building Real-Time Travel Alerts

Timothy Spann

JConWorld: Continuous SQL with Kafka and Flink In this talk, I will walk through how someone can setup and run continous SQL queries against Kafka topics utilizing Apache Flink. We will walk through creating Kafka topics, schemas and publishing data. We will then cover consuming Kafka data, joining Kafka topics and inserting new events into Kafka topics as they arrive. This basic over view will show hands-on techniques, tips and examples of how to do this. Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. https://www.datainmotion.dev/p/about-me.html https://dzone.com/users/297029/bunkertor.html https://www.youtube.com/channel/UCDIDMDfje6jAvNE8DGkJ3_w?view_as=subscriber

JConWorld_ Continuous SQL with Kafka and Flink

Timothy Spann

[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines https://dssconf.pl/en/#agenda-section Integrating LLM with Streaming Data Pipelines Timothy Spann, Principal Developer Advocate, Cloudera APACHE NIFI, APACHE FLINK, APACHE KAFKA, LLM, HUGGINGFACE, REST, STREAMING LESS In this talk and demo I will walk through how to add LLMs to your streaming pipelines by integration through Apache NiFi. https://github.com/tspannhw/FLaNK-watsonx.ai Cloudera streaming llm generative ai slack to slack

[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines

Timothy Spann

Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo

Timothy Spann

AIDevWorld 23 Apache NiFi 101 Introduction and Best Practices https://sched.co/1RoAO Timothy Spann, Cloudera, Principal Developer Advocate In this talk, we will walk step by step through Apache NiFi from the first load to first application. I will include slides, articles and examples to take away as a Quick Start to utilizing Apache NiFi in your real-time dataflows. I will help you get up and running locally on your laptop, Docker or in CDP Public Cloud. Wednesday November 1, 2023 12:00pm - 12:25pm PDT VIRTUAL AI DevWorld -- Main Stage https://app.hopin.com/events/api-world-2023-ai-devworld/stages Retail & E-Commerce AI (Industry AI Conference) Session Type OPEN TALK Track or Conference Retail & E-Commerce AI (Industry AI Conference), Industry AI Conference, VIRTUAL, Tensorflow & PyTorch & Open Source Frameworks (AI/ML Engineering Conference), AI/ML Engineering Conference, AI DevWorld In-Person/Virtual Virtual, Virtual Exclusive apache nifi Timothy Spann Cloudera Principal Developer Advocate for Data in Motion Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. cloudera dataflow

AIDevWorldApacheNiFi101

Timothy Spann

26Oct2023_ Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup.pdf ## Details **Important** Please complete your registration in this short form. For on-site we have limited room, so please confirm if you are attending in-person in Manhattan, NYC. -------------------------------------------------------------------------------------------- We're at StarTree, excited to join forces with our friends at Cloudera, for a meetup that is all about The Latest in Real-Time Analytics: Generative AI and LLM, featuring Apache Pinot and Apache NiFi. Join us for an insightful discussion about cutting-edge analytics, meet the community in person, and catch up over drinks and snacks. What's the plan ? 05:30-06:00 Pizza and Networking 06:00-06:35 Adding Generative AI to Real-Time Streaming Pipelines | Tim Spann, Principal Developer Advocate, Cloudera 06:35-07:10 Apache Pinot and Kafka an excellent pairing for refined palates | Tim Veil, VP of Solutions Engineering and Enablement, StarTree 07:10-07:20 QNA 07:20- 07:30 More Snacks and Networking ;) **Important** Seats are limited Please complete your registration in this short form. Adding Generative AI to Real-Time Streaming Pipelines | Timothy Spann In this talk, Tim will discuss the basics of real-time streaming, walk through the tools used including Apache NiFi, Apache Kafka and Apache Flink and show how to build a real-time streaming pipeline that sends prompts to LLMs hosted by the likes of Hugging Face, IBM and Cloudera. He will also discuss where real-time data stores like Apache Pinot come into play. He will show a detailed demonstration of a few use cases involving different sources of data including Kafka, Medium Articles and interactive Question and Response in Slack. He will then show you how you can build your own and where areas of growth exist. Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. https://github.com/tspannhw/SpeakerProfile Apache Pinot and Kafka an excellent pairing for refined palates | Tim Veil The other Tim, Tim Veil, will dive into the history and architect

26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup

Timothy Spann

More from Timothy Spann (20)

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024

Real-Time AI Streaming - AI Max Princeton

2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...

28March2024-Codeless-Generative-AI-Pipelines

TCFPro24 Building Real-Time Generative AI Pipelines

2024 Build Generative AI for Non-Profits

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...

Conf42-Python-Building Apache NiFi 2.0 Python Processors

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines

DBA Fundamentals Group: Continuous SQL with Kafka and Flink

NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines

Building Real-Time Travel Alerts

JConWorld_ Continuous SQL with Kafka and Flink

[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines

Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo

AIDevWorldApacheNiFi101

26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup

Recently uploaded

Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024

MulesoftMunichMeetup

Scenarios are the central artifact of the Behaviour Driven Development (BDD) process. Although many teams use scenarios and tools like Cucumber or SpecFlow to automate them, in many cases their scenarios contain a lot of details, particularly test data, and therefore they become too complicated to support collaboration with the business. The "essential" principle of scenario writing (scenario formulation) states that only those details should be included in the scenario that are relevant for the outcome. This talk provides help for those who struggle implementing this principle or would be interested to learn how you can create brief and maintainable scenarios.

Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)

Gáspár Nagy

In his book, The Nature of the Physical World, Sir Arthur Eddington commented that “We have to appeal to the one outstanding law — the second law of thermodynamics — to put some sense into the world.” This sense-making goes beyond the physical world, too. Entropy is also essential in the fields of information and communication theory. During this lecture for the Princeton Plasma Physics Laboratory, lecturer Andrea Goulet discussed the application of entropy-related concepts in two communication systems: software and collaborative teams. She examined how concepts that help us understand systemic statistical disorder, such as ergodic systems, Lyapunov exponents, Kolmogorov-Sinai entropy, and Shannon-entropy can help us optimize for both software quality and innovation. She also provided several domain-specific models: Lehman’s Laws and Conway’s Law for software, as well as new models from her own research that relate to entropy and innovation. Entropy helps us understand the world and achieve great things. There is an underlying beauty in its principles that we can use to advance scientific discovery. When we understand the subtleties related to balancing surprise and structure, we increase our chances for effective collaboration and finding novel solutions to complex problems.

Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...

Andrea Goulet

How to download files safely from the internet ? Source: <a href="https://www.downloadsafely.com/">Download Safely</a> -Welcome to Secure Downloading&Surfing. Learn how to safely download files and protect your data while browsing the web. Let's dive into the world of secure surfing! Understanding the Risks Discover the potential threats associated with downloading files from the internet and the impact of malicious software on your device.

how-to-download-files-safely-from-the-internet.pdf

metbey

Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Building a notification campaign might seem easy and it is easy to get started with a simple set up. But once the scale kicks in, it becomes every important to have a resilient architecture that can handle hundreds of thousands of recipients. This talk will focus on the Serverless services consumed in building the architecture and the various architectural decisions. The talk covers the various challenges in building an architecture of this sorts and how we overcame them using Serverless services.

Lessons Learned from Building a Serverless Notifications System.pdf

Srushith Repakula

Transformer Neural Network Use Cases with Links

JinanKordab

Spring into AI presented by Dan Vega 5/14

VMware Tanzu

Remaining informed about evolving trends is crucial for both businesses and developers in the dynamic field of web development. The year 2024 heralds groundbreaking advancements poised to revolutionize website construction and interaction. From progressive web apps and voice search optimization to prioritizing accessibility and cybersecurity, staying attuned to these trends is imperative. In this blog, we delve deeper into the pivotal trends shaping the future of web development in the upcoming year.

The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf

kalichargn70th171

How to install and activate eGrabber JobGrabber

eGrabber

Unlock the full potential of your software development lifecycle with SpiraPlan's powerful REST APIs. Led by experts from Inflectra and our partner, MVerve, this webinar offers a practical guide to leveraging the API's power for automation, integration, and data-driven insights. Whether you're an API expert or beginner, our recap blog is packed with valuable information to enhance your SpiraPlan experience. The webinar emphasized how SpiraPlan's REST API allows you to: • Automate Workflows: Streamline your processes by automating repetitive tasks. • Integrate with Existing Tools: Seamlessly connect SpiraPlan with your existing software ecosystem. • Build Custom Solutions: Tailor SpiraPlan to your unique needs by developing custom applications and integrations. • Gain Deeper Insights: Extract valuable data from SpiraPlan for advanced reporting and analysis.

From Theory to Practice: Utilizing SpiraPlan's REST API

Inflectra

This is a classic migration case study (the past, current and the future) at scale from a world-wide company transitioning from Confluent Platform and Confluent Cloud to self-managed Apache Kafka on Kubernetes using Strimzi. At Maersk, we have been architecting, designing and implementing our 3rd generation Event Streaming Platform. This platform is based on Kubernetes in Azure and using Strimzi to operate Apache Kafka at large scale, highly reliable, segregating data based on isolated use cases. Our 2nd generation was based on OnPrem Confluent Platform and Confluent Cloud and this presentation is the story of this migration and reasoning behind it. Furthermore, we would get into details on how we monitor (Grafana, Prometheus), alert (GoAlert and alert as code), operate and provide self-service solutions on top of Strimzi to enable business critical application in Maersk, implemented in GoLang using the GitOps deployment model with Flux and Kustomization among others. Finally, if time allows we will end with a demo of an open-source self service tool to monitor and explore the cluster with most wanted features such as topic message browsing and configuring and restarting connectors.

StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf

steffenkarlsson2

Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Salesforce unveiled the Salesforce Zero Copy Partner Network, a global alliance of tech and solution providers developing secure, bidirectional zero-copy interfaces with Salesforce Data Cloud. This innovative approach enables seamless data actioning across the Salesforce Einstein Platform. Brian Miliham, Salesforce’s President and COO, highlighted the challenge companies face with fragmented data. This ecosystem empowers businesses to harness all their data, irrespective of its location, maximizing the potential for personalized customer experiences and facilitating quicker, cost-effective AI integration within Salesforce.

Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...

CloudMetic

Auto-Affiliate AI is the only Automated Affiliate system that generates $5,000/week in affiliate commissions without sales funnels, email marketing, or advertising. I've take all the guesswork out of affiliate marketing by taking the fastest, most beginner-friendly commission strategy and trained a ChatGTP system to implement the whole thing automatically. Inside Auto-Affiliate AI you'll discover: How to run this business in 2 hours per week with $0 out-of-pocket expense. Step-by-step beginner-friendly instructions with no technical requirements. Our Auto-Affiliate AI campaigns average $5,000/week in profits. No ad buying required - all traffic comes from free sources. No sales funnels or landing pages - Clicks go directly to your affiliate links. No manual content creation - Everything is generated with pre-trained AI. Generates affiliate link clicks immediately, first sale hours after getting started. Guaranteed affiliate approval with commissions as high as 85%. Automation allows anyone to earn maximum money in minimum time. How to recession-proof this opportunity to work in any economy: your passive income only grows year after year. (Note: Download the paper about this software or make it your own. do After that, click on Click here for Instant access inside the paper, and it will take you to the sales page of the product.)

Auto Affiliate AI Earns First Commission in 3 Hours..pdf

SelfMade bd

Talk given at NewCrafts Paris Conference 2024 Abstract: Have you ever observed quality problems with what your team is delivering? Have you ever felt your team is learning about the actual status quo late, too late to include these insights in your next steps? Have you ever wondered how to influence change when you’re “just” a specialist on the team, maybe even felt powerless? I’ve mostly worked with people who never experienced effective collaboration across roles and specialties. How to get such a team to move toward testing throughout the delivery workflow from idea to production, on all kinds of levels, for all parts of the product, by the whole team? How to consider quality from the start and have it continuously inform our decision-making? Over the years, I’ve built a repertoire of tactics to help teams transform toward holistic testing and quality. They range from meeting people where they are to creating guidelines, pairing and many others. It’s a toolbox of approaches to try out and find the sweet spot where people start moving toward better directions together. These tactics are fallible, not all of them work in all contexts, and yet they have helped my teams get to a better place. Join this talk to get inspired to take things into your hands. Learn what you can try yourself in your context to help your team make the first steps on your journey towards holistic testing and quality. Key learnings: - Learn how you can influence team culture, no matter your role or experience level - Add team transformation tactics to your toolbox of approaches to try - Understand the many ways we can move towards holistic testing and quality

Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...

Lisi Hocke

^Clinic ^%[+27788225528*Abortion Pills For Sale In harare

kasambamuno

Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024

SimonedeGijt

Choosing the right Recruitment Management Software can be difficult considering how many are available. Let us begin by describing what a recruitment management system is. In simplified terms, Recruitment Management Software refers to a comprehensive suite of tools designed to effectively oversee and streamline an organisation's recruitment process. To know more information here: https://sites.google.com/view/nyggs-recruitment-management/home

What is a Recruitment Management Software?

NYGGS Automation Suite

Nearly a decade ago, a small group of menders (and one maker) found community with each other at a conference and Legacy Code Rocks was born. Since then, the Legacy Code Rocks community has grown to over 1000 people, 150 podcast episodes, a weekly meetup and of course, MenderCon. In this talk, Andrea Goulet, the maker who kindled the spark, will take us back to those early days and reflect on how the bedrock of the Legacy Code Rocks community has always been creating a sense of belonging. Compassion and empathy are at the core of being a mender. Mending is caring deeply, seeing potential, and being willing to work through the frustration of making things better one small step at a time. This commitment to helping others feel seen instead of shame is why the Legacy Code Rocks community is so special — and it’s exactly why having the heart of the mender is what our world needs now more than ever.

Community is Just as Important as Code by Andrea Goulet

Andrea Goulet

Recently uploaded (20)

Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024

Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)

Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...

how-to-download-files-safely-from-the-internet.pdf

Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...

Lessons Learned from Building a Serverless Notifications System.pdf

Transformer Neural Network Use Cases with Links

Spring into AI presented by Dan Vega 5/14

The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf

How to install and activate eGrabber JobGrabber

From Theory to Practice: Utilizing SpiraPlan's REST API

StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf

Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...

Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...

Auto Affiliate AI Earns First Commission in 3 Hours..pdf

Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...

^Clinic ^%[+27788225528*Abortion Pills For Sale In harare

Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024

What is a Recruitment Management Software?

Community is Just as Important as Code by Andrea Goulet

Fast data for fitness 10 nov 2020

1. Fast Data for Fitness Real-time Health Sensor Ingest Timothy Spann - Principal DataFlow Field Engineer

3. © 2020 Cloudera, Inc. All rights reserved. 3 Timothy Spann Data in Motion Principal Field Engineer @PaasDev DZone Zone Leader and Big Data MVB Princeton NJ Future of Data Meetup https://github.com/tspannhw https://www.datainmotion.dev/ https://meetup.com/futureofdata-princeton/

4. © 2020 Cloudera, Inc. All rights reserved. 4 Data Sources - From Images to Sub-Second Data Events Heart Rate Temperature Blood Sugar Altimeter Blood Pressure Weight GPS Live Video/Picture AI for Pose Estimation Improvement to Technique

6. © 2020 Cloudera, Inc. All rights reserved. 6 Fitness+ Data Streaming Pipeline Device Data SensorsLogs Weather Sensors Aggregates Energy SQL Analytics MiNiFi Agent Deep Learning Classiﬁcation Edge Private Cloud Multi-Public Cloud

8. © 2020 Cloudera, Inc. All rights reserved. 8 CLOUDERA FLOW AND EDGE MANAGEMENT Enable easy ingestion, routing, management and delivery of any data anywhere (Edge, cloud, data center) to any downstream system with built in end-to-end security and provenance Advanced tooling to industrialize ﬂow development (Flow Development Life Cycle) ACQUIRE • Over 300 Prebuilt Processors • Easy to build your own • Parse, Enrich & Apply Schema • Filter, Split, Merger & Route • Throttle & Backpressure FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLOG PROCESS HASH MERGE EXTRACT DUPLICATE SPLIT ENCRYPT TALL EVALUATE EXECUTE GEOENRICH SCAN REPLACE TRANSLATE CONVERT ROUTE TEXT ROUTE CONTENT ROUTE CONTEXT ROUTE RATE DISTRIBUTE LOAD DELIVER • Guaranteed Delivery • Full data provenance from acquisition to delivery • Diverse, Non-Traditional Sources • Eco-system integration FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLOG

9. © 2019 Cloudera, Inc. All rights reserved. 9 Apache Kafka • Highly reliable distributed messaging system • Decouple applications, enables many-to-many patterns • Publish-Subscribe semantics • Horizontal scalability • Eﬃcient implementation to operate at speed with big data volumes • Organized by topic to support several use cases Source System Source System Source System Kafka Fraud Detection Security Systems Real-Time Monitoring Source System Source System Source System Fraud Detection Security Systems Real-Time Monitoring Many-To-Many Publish-Subscribe Point-To-Point Request-Response

10. © 2020 Cloudera, Inc. All rights reserved. 10 New features delivered with Cloudera Streaming Analytics (CSA) 1.2 Next Generation Streaming Analytics Flink SQL Support Agile Streaming App Development using SQL New Flink Atlas Hook Capture operational Flink app metadata and lineage Single View of Flink Yarn Jobs Improve Developer Experience & operational visibility

11. © 2020 Cloudera, Inc. All rights reserved. 11 ● https://www.cloudera.com/tutorials/building-a-sentiment-analysis-application/3.html ● https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/ ● https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html ● https://www.datainmotion.dev/2020/06/the-rise-of-mega-edge-flank.html ● https://www.datainmotion.dev/2020/01/cloudera-edge2ai-minifi-java-agent-with.html ● https://www.datainmotion.dev/2019/08/rapid-iot-development-with-cloudera.html ● https://www.datainmotion.dev/2019/09/powering-edge-ai-for-sensor-reading.html ● https://www.datainmotion.dev/2019/05/dataworks-summit-dc-2019-report.html ● https://www.datainmotion.dev/2019/03/using-raspberry-pi-3b-with-apache-nifi.html ● https://www.datainmotion.dev/2020/06/unboxing-most-amazing-edge-ai-device.html ● https://www.datainmotion.dev/2019/08/edge-processing-with-jetson-nano-part-3.html ● https://www.datainmotion.dev/2020/04/predicting-sensor-readings-with-time.html REFERENCES

Fast data for fitness 10 nov 2020

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Fast data for fitness 10 nov 2020

Similar to Fast data for fitness 10 nov 2020 (20)

More from Timothy Spann

More from Timothy Spann (20)

Recently uploaded

Recently uploaded (20)

Fast data for fitness 10 nov 2020