MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu

•

3 likes•986 views

Build an end-to-end Kafka-based pipeline in minutes! Enhanced Data pipelines with no code. An Apache Kafka tutorial with Lenses by Stefan Bocutiu.

Technology

Stefan Bocutiu
@stepib
London Apache Kafka Meetup, Feb 2018
End to end topologies on Apache Kafka  
An IoT use case CONNECT ANALYSE REACT

About Today
Intros
IoT
High Level Architecture
Build the Pipeline
Q&A

Hi! It’s Stefan! 
@stepib 
https://github.com/stheppi
CONNECT ANALYSE REACT
Streaming Data Processing & Integration 
KAFKA - KUBERNETES

EVOLVESCALEBUILDIoT initiative challenges
★ 94% face challenges collecting and analysing IoT data
★ 26+ Billion devices by 2020, Gartner
More Data, More Complexity
★ 2.500.000 trillion bytes produced daily
REAL TIME DATA INGESTION + IMMEDIATE QUERIES =  
ACTIONABLE / TIMELY INSIGHTS
IoT Data Flows

IoT data flow
store datastream & process datasensor data

IoT data flow
store datastream & process datasensor data
E T L

InﬂuxDBsensor data
MQTT
InﬂuxDB  
Connector
LENSES SQL
PROCESSOR
KAFKA  
CONNECT
KAFKA  
CONNECT
MQTT  
Connector
KAFKA STREAMS

InﬂuxDBsensor data
MQTT
InﬂuxDB  
Connector
LENSES SQL
PROCESSOR
KAFKA  
CONNECT
KAFKA  
CONNECT
MQTT  
Connector
KAFKA STREAMS
Synthetic Data
With Moquette 
1 Broker,  
data generate
sensor data
InﬂuxDB Docker

®
IoT &
High Volumes, N devices 
& irregular intervals
Real Time Analytics  
& Microservices
Unlimited streams of data, async transfers 
Producers & Consumers decoupled operation
Kafka Streams API 
Processing and analysing data in motion
Multiple sources of data  
& long term storeage
Kafka Connect API,  
move data with pluggable reusable & scalable connectors
An open source streaming framework
with messaging semantics where records are key-value pairs.

LENSES BOX 
For Kafka Development
Single docker image with all services
Growing list of examples & synthetic data generators.
Stream Reactor 25+ connectors
Kafka CLI autocomplete.
Rest and WebSocket endpoints
Redux JS Client
Testing Libraries
FREE!
All you need to start working with Apache Kafka ecosystem

MQTT Source connector 
• Message Queueing Telemetry Transport messaging protocol
• Optimised for connected devices and M2M protocol
• Low bandwidth, high-latency, unreliable networks
• Plain or SSL connections
• Kafka Connect Query Language
INSERT INTO kafka_topic
SELECT * FROM /mqtt_source_topic
[WITHCONVERTER=`myclass`]
MQTT
Distributed, Fault tolerant, scalable
No code required
CLI / UI / Monitor
KCQL Support
Multiple Distributions

• Process data in motion with Lenses SQL
• Create & Deploy Lenses SQL Processors
• Native Kafka & Kubernetes Support Scalability
• Visual Topology
• Monitor and manage processors via LENSES
Process & Analyse
Data in motion

InﬂuxDB Sink connector 
• AVRO & JSON Support
• Error handling ( Noop, Retry, Throw)
• Field selection & ﬁeld alias
• Multiple inserts supported
• Kafka Connect Query Language
• MEASUREMENTS, KCQL: INSERTS
• TAGS, KCQL: WITHTAG
• TIMESTAMPS, KCQL: WITHTIMESTAMP
• DURATION
Distributed, Fault tolerant, scalable
No code required
CLI / UI / Monitor
KCQL Support
Multiple Distributions
INSERT INTO sensorMeasure  
SELECT * FROM sensor_data_avg  
WITHTAG (id, poweredBy=Lenses)

Distributed, Fault tolerant, scalable
No code required
CLI / UI / Monitor
KCQL Support
Multiple Distributions
InﬂuxDB Sink connector 
• Timestamp nanosecond
• KCQL
• No ﬁlter support
• No function support
• Throughput involves domain knowlege
• Kafka Connect rebalances

visual data flow
Visual and Interactive Topology 
for a 360 view of your data
End-To-End Topology Management 
Visualise your Kafka-based data flows 
in real time

Wrap it up!
CONNECT
ANALYSE
REACT
Real Time, IoT pipelines in minutes!
MQTT

Contact us
www.landoop.com
info@landoop.com
Twitter @LandoopLtd
Github github.com/landoop
Gitter https://gitter.im/Landoop
Slack https://launchpass.com/landoop-community
Thank you!

What's hot

Kafka summit apac session

Christina Lin

Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...

Kai Wähner

Introduction to networking options available in Confluent Cloud Self Serve provisioning of confluent Kafka clusters. VPC Peering, VNet Peering, Transit Gateway and Private Link Options for AWS, GCP, Azure networking offering. Caveats of confluent's cloud networking solutions customers should be aware of. Details of two major pieces of the architecture of Confluent Cloud - Data Plane Network and Control Plane.

Confluent Cloud Networking | Rajan Sundaram, Confluent

HostedbyConfluent

New Features in Confluent Platform 6.0 / Apache Kafka 2.6

Kai Wähner

Let's build a simple ingest to cloud datawarehouse with low code

Timothy Spann

With the rapid onset of the global Covid-19 Pandemic from the start of this year the USA Centers for Disease Control and Prevention (CDC) had to quickly implement a new Covid-19 specific pipeline to collect testing data from all of the USA’s states and territories, and carry out other critical steps including integration, cleaning, checking, enrichment, analysis, and enforcing data governance and privacy etc. The pipeline then produces multiple consumable results for federal and public agencies. They did this in under 30 days, using Apache Kafka. In this presentation we'll build a similar (but simpler) pipeline for ingesting, integrating, indexing, searching/analysing and visualising some publicly available tidal data. We'll briefly introduce each technology and component, and walk through the steps of using Apache Kafka, Kafka Connect, Elasticsearch and Kibana to build the pipeline and visualise the results.

Building a real-time data processing pipeline using Apache Kafka, Kafka Conne...

Paul Brebner

Apache Kafka is taking the world by storm and is rapidly becoming the de-facto event bus for event-driven and streaming applications that respond to events and data in real time. OpenShift Streams for Apache Kafka is Red Hat's fully hosted and managed Apache Kafka service targeting development teams that want to incorporate streaming data and scalable messaging in their applications, without the burden of setting up and maintaining a Kafka cluster infrastructure. In this session you will discover how Apache Kafka can be used in an IoT scenario to ingest data from devices and make them available in real-time to other applications. More specifically you will learn how to: Simulate devices that send MQTT messages to a MQTT broker Use Apache Camel and Camel-K to bridge MQTT with Apache Kafka Use Kafka Streams in a Quarkus application to process the device messages Query the state of the devices using GraphQ

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...

Red Hat Developers

Use cases and architectures for IoT projects leveraging Apache Kafka, ksqlDB, machine Learning / deep Learning frameworks like TensorFlow, and cloud infrastructure. Large numbers of IoT devices lead to big data and the need for further processing and analysis. Apache Kafka is a highly scalable and distributed open source streaming platform, which can connect to MQTT and other IoT standards. Kafka ingests, stores, processes and forwards high volumes of data from thousands of IoT devices. The rapidly expanding world of stream processing can be daunting, with new concepts such as various types of time semantics, windowed aggregates, changelogs, and programming frameworks to master. KSQL is the streaming SQL engine on top of Apache Kafka which simplifies all this and make stream processing available to everyone without the need to write source code. This talk shows how to leverage Kafka and KSQL in an IoT sensor analytics scenario for predictive maintenance and integration with real time monitoring systems. A live demo shows how to embed and deploy Machine Learning models - built with frameworks like TensorFlow, DeepLearning4J or H2O - into mission-critical and scalable real time applications.

IoT Sensor Analytics with Kafka, ksqlDB and TensorFlow

Kai Wähner

Many cities are investing in technologies to transform their cities into smart city- environments in which data collection and analysis is utilized to manage assets and resources efficiently. Modern technology can help connect the right data, at the right time, to the right people, processes and systems. Innovations around smart cities and the Internet of Things give cities the ability to improve motor safety, unify and manage transportation systems and traffic, save energy and provide a better experience for the residents. By utilizing an event streaming platform, like Confluent, cities are able to process data in real-time from thousands of sources, such as sensors. By aggregating that data and analyzing real-time data streams, more informed decisions can be made and fine-tuned operations developed for a positive impact on everyday challenges faced by cities. Learn how to: -Overcome challenges for building a smarter city -Build a real time infrastructure to correlate relevant events -Connect thousands of devices, machines, and people -Leverage open source and fully managed solutions from the Apache Kafka ecosystem

Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...

Kai Wähner

What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

Building Event Driven (Micro)services with Apache Kafka

Guido Schmutz

End to-end large messages processing with Kafka Streams & Kafka Connect

confluent

With Apache Kafka’s rise for event-driven architectures, developers require a specification to design effective event-driven APIs. AsyncAPI has been developed based on OpenAPI to define the endpoints and schemas of brokers and topics. For Kafka applications, the broker’s design to handle high throughput serialized payloads brings challenges for consumers and producers managing the structure of the message. For this reason, a registry becomes critical to achieve schema governance. Apicurio Registry is an end-to-end solution to store API definitions and schemas for Kafka applications. The project includes serializers, deserializers, and additional tooling. The registry supports several types of artifacts including OpenAPI, AsyncAPI, GraphQL, Apache Avro, Google protocol buffers, JSON Schema, Kafka Connect schema, WSDL, and XML Schema (XSD). It also checks them for validity and compatibility. In this session, we will be covering the following topics: ● The importance of having a contract-first approach to event-driven APIs ● What is AsyncAPI, and how it helps to define Kafka endpoints and schemas ● The Kafka challenges on message structure when serializing and deserializing ● Introduction to Apicurio Registry and schema management for Kafka ● Examples of how to use Apicurio Registry with popular Java frameworks like Spring and Quarkus

Apicurio Registry: Event-driven APIs & Schema governance for Apache Kafka | F...

HostedbyConfluent

At BAADER, we design and engineer innovative and holistic solutions that ensure intelligent, safe, efficient and sustainable food processing in all phases, from the handling of live and raw protein materials to the finished food products. As a key player in the food value chain, we aim to take further significant steps toward greater efficiency, traceability, transparency, profitability, and sustainability through new digital solutions. During our digital transformation we are working on two ends: At one hand there are many brownfield factories unprepared for the digital journey and the other hand we have powerful greenfield technologies like Apache Kafka. Now, we have to bring two mindsets together – robust food processing machinery and highly scalable software technologies. In this talk, we will present how we successfully started to ingest various kinds of IoT data into our Kafka cluster - spotlighted from both ends.

Ingesting IoT data in Food Processing

confluent

Watch this talk here: https://www.confluent.io/online-talks/using-apache-kafka-to-optimize-real-time-analytics-financial-services-iot-applications When it comes to the fast-paced nature of capital markets and IoT, the ability to analyze data in real time is critical to gaining an edge. It’s not just about the quantity of data you can analyze at once, it’s about the speed, scale, and quality of the data you have at your fingertips. Modern streaming data technologies like Apache Kafka and the broader Confluent platform can help detect opportunities and threats in real time. They can improve profitability, yield, and performance. Combining Kafka with Panopticon visual analytics provides a powerful foundation for optimizing your operations. Use cases in capital markets include transaction cost analysis (TCA), risk monitoring, surveillance of trading and trader activity, compliance, and optimizing profitability of electronic trading operations. Use cases in IoT include monitoring manufacturing processes, logistics, and connected vehicle telemetry and geospatial data. This online talk will include in depth practical demonstrations of how Confluent and Panopticon together support several key applications. You will learn: -Why Apache Kafka is widely used to improve performance of complex operational systems -How Confluent and Panopticon open new opportunities to analyze operational data in real time -How to quickly identify and react immediately to fast-emerging trends, clusters, and anomalies -How to scale data ingestion and data processing -Build new analytics dashboards in minutes

Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...

confluent

Business digitalization trends like microservices, the Internet of Things or Machine Learning are driving the need to process events at a whole new scale, speed and efficiency. Traditional solutions like ETL/data integration or messaging are not build to serve these needs. Today, the open source project Apache Kafka® is being used by thousands of companies including over 60% of the Fortune 100 to power and innovate their businesses by focusing their data strategies around event-driven architectures leveraging event streaming.We will discuss the market and technology changes that have given rise to Kafka and to Event Streaming, and we will introduce the audience to the key aspects of building an Event streaming platform with Kafka. Examples of productive use cases from the automotive, manufacturing and transportation sector will showcase the power of event streaming.

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

Kai Wähner

To stay informed about the latest features in Confluent Platform 5.4 join Martijn Kieboom Solutions Engineer at Confluent, for the ‘What’s New in Confluent 5.4?’ on February 12 at 11 am GMT/ 12 Noon CET. Martijn will talk through the new features including: Role-Based Access Control and how it enables highly granular control of permissions and platform access Structured Audit Logs and how they enable the capture of authorization logs How Multi-Region Clusters deliver asynchronous replication at the topic level, allowing companies to run a single Kafka Cluster across multiple data-centres Schema validations role in enabling businesses that run Kafka at scale to deliver data compatibility across platforms

What's new in confluent platform 5.4 online talk

confluent

Tesla ingests trillions of events every day from hundreds of unique data sources through our streaming data platform. Find out how we developed a set of high-throughput, non-blocking primitives that allow us to transform and ingest data into a variety of data stores with minimal development time. Additionally, we will discuss how these primitives allowed us to completely migrate the streaming platform in just a few months. Finally, we will talk about how we scale team size sub-linearly to data volumes, while continuing to onboard new use cases.

0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019

confluent

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Watch this talk here: https://www.confluent.io/online-talks/gcp-for-apache-kafka-users-stream-ingestion-processing In private and public clouds, stream analytics commonly means stateless processing systems organized around Apache Kafka® or a similar distributed log service. GCP took a somewhat different tack, with Cloud Pub/Sub, Dataflow, and BigQuery, distributing the responsibility for processing among ingestion, processing and database technologies. We compare the two approaches to data integration and show how Dataflow allows you to join and transform and deliver data streams among on-prem and cloud Apache Kafka clusters, Cloud Pub/Sub topics and a variety of databases. The session will have a mix of architectural discussions and practical code reviews of Dataflow-based pipelines.

GCP for Apache Kafka® Users: Stream Ingestion and Processing

confluent

This session discusses uses cases leveraging Apache Kafka open source ecosystem as streaming platform to process IoT data. See use cases, architectural alternatives and a live demo of how devices connect to Kafka via MQTT. Learn how to analyze the IoT data either natively on Kafka with Kafka Streams/KSQL, or on an external big data cluster like Spark, Flink or Elastic leveraging Kafka Connect, and how to leverage TensorFlow for Machine Learning. The focus is on connected cars / connected vehicles and V2X use cases respectively mobility services. A live demo shows how to build a cloud-native IoT infrastructure on Kubernetes to connect and process streaming data in real-time from 100.000 cars to do predictive maintenance at scale in real-time. Code for the live demo on Github: https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

Connected Vehicles and V2X with Apache Kafka

Kai Wähner

What's hot (20)

Kafka summit apac session

Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...

Confluent Cloud Networking | Rajan Sundaram, Confluent

New Features in Confluent Platform 6.0 / Apache Kafka 2.6

Let's build a simple ingest to cloud datawarehouse with low code

Building a real-time data processing pipeline using Apache Kafka, Kafka Conne...

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...

IoT Sensor Analytics with Kafka, ksqlDB and TensorFlow

Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...

Building Event Driven (Micro)services with Apache Kafka

End to-end large messages processing with Kafka Streams & Kafka Connect

Apicurio Registry: Event-driven APIs & Schema governance for Apache Kafka | F...

Ingesting IoT data in Food Processing

Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

What's new in confluent platform 5.4 online talk

0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

GCP for Apache Kafka® Users: Stream Ingestion and Processing

Connected Vehicles and V2X with Apache Kafka

Similar to MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu

IoT Sensor Analytics with Apache Kafka, KSQL, TensorFlow and MQTT => Kafka-Native End-to-End IoT Data Integration and Processing. Large numbers of IoT devices lead to big data and the need for further processing and analysis. Apache Kafka is a highly scalable and distributed open source streaming platform, which can connect to MQTT and other IoT standards. Kafka ingests, stores, processes and forwards high volumes of data from thousands of IoT devices. The rapidly expanding world of stream processing can be daunting, with new concepts such as various types of time semantics, windowed aggregates, changelogs, and programming frameworks to master. KSQL is the streaming SQL engine on top of Apache Kafka which simplifies all this and make stream processing available to everyone without the need to write source code. This talk shows how to leverage Kafka and KSQL in an IoT sensor analytics scenario for predictive maintenance and integration with real time monitoring systems. A live demo shows how to embed and deploy Machine Learning models - built with frameworks like TensorFlow, DeepLearning4J or H2O - into mission-critical and scalable real time applications.

IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, K...

Kai Wähner

Beyond the Brokers: A Tour of the Kafka Ecosystem

confluent

Beyond the brokers - A tour of the Kafka ecosystem

Damien Gasparina

Beyond the brokers - Un tour de l'écosystème Kafka

Florent Ramiere

Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière

confluent

Not all workloads allow cloud computing. Low latency, cybersecurity, and cost-efficiency require a suitable combination of edge computing and cloud integration. This session explores architectures and design patterns for software and hardware considerations to deploy hybrid data streaming with Apache Kafka anywhere. A live demo shows data synchronization from the edge to the public cloud across continents with Kafka on Hivecell and Confluent Cloud.

Kafka for Real-Time Replication between Edge and Hybrid Cloud

Kai Wähner

Dataops on streaming data: Kafka to InfluxDb via Kubernetes native flows

Lenses.io

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and Kafka Apache NiFi, Apache Flink, Apache Kafka Timothy Spann Principal Developer Advocate Cloudera Data in Motion https://budapestdata.hu/2023/en/speakers/timothy-spann/ Timothy Spann Principal Developer Advocate Cloudera (US) LinkedIn · GitHub · datainmotion.dev June 8 · Online · English talk Building Modern Data Streaming Apps with NiFi, Flink and Kafka In my session, I will show you some best practices I have discovered over the last 7 years in building data streaming applications including IoT, CDC, Logs, and more. In my modern approach, we utilize several open-source frameworks to maximize the best features of all. We often start with Apache NiFi as the orchestrator of streams flowing into Apache Kafka. From there we build streaming ETL with Apache Flink SQL. We will stream data into Apache Iceberg. We use the best streaming tools for the current applications with FLaNK. flankstack.dev BIO Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...

Timothy Spann

Devoxx university - Kafka de haut en bas

Florent Ramiere

The Ohio Department of Transportation has adopted Confluent as the event driven enabler of DriveOhio, a modern Intelligent Transportation System. DriveOhio digitally links sensors, cameras, speed monitoring equipment, and smart highway assets in real time, to dynamically adjust the surface road network to maximize the safety and efficiency for travelers. Over the past 24 months the team has increased the number and types of devices within the DriveOhio environment, while also working to see their vendors adopt Kafka to better participate in data sharing.

Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transpo...

HostedbyConfluent

Bank of China Tech Talk 2: Introduction to Streaming Data and Stream Processi...

confluent

DataOps on Streaming Data: From Kafka to InfluxDB via Kubernetes Native Flows...

InfluxData

Spark Streaming + Kafka 0.10: an integration story by Joan Viladrosa Riera at...

Big Data Spain

If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.

Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...

HostedbyConfluent

JHipster conf 2019 - Kafka Ecosystem

Florent Ramiere

IoT and Event Streaming at Scale with Apache Kafka

confluent

Apache™ Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system. It offers higher throughput, reliability and replication. To manage growing data volumes, many companies are leveraging Kafka for streaming data ingest and processing. Join experts from Confluent, the creators of Apache™ Kafka, and the experts at Attunity, a leader in data integration software, for a live webinar where you will learn how to: -Realize the value of streaming data ingest with Kafka -Turn databases into live feeds for streaming ingest and processing -Accelerate data delivery to enable real-time analytics -Reduce skill and training requirements for data ingest The recorded webinar on slide 32 includes a demo using automation software (Attunity Replicate) to stream live changes from a database into Kafka and also includes a Q&A with our experts. For more information, please go to www.attunity.com/kafka.

Streaming Data Ingest and Processing with Apache Kafka

Attunity

(Bob Lehmann, Bayer) Kafka Summit SF 2018 You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why? In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform. In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

confluent

The Internet of Things (IoT) is getting more and more traction as valuable use cases come to light. Whether you are in Healthcare, Telecommunications, Manufacturing, Banking or Retail to name a few industries, there is one key challenge and that's the integration of backend IoT data logs and applications, business services and cloud services to process the data in real time and at scale. In this talk, we will be sharing how Kafka has become the leading technology used throughout the business to provide Real Time Event Streaming. Explore real life use cases of Kafka Connect, Kafka Streams and KSQL independent of the data deployment be it on a private or public Cloud, On Premise or at the Edge. Audi - Connected car infrastructure Robert Bosch Power Tools - Track and Trace of devices and people at construction areas Deutsche Bahn - Customer 360 for train timetable updates E.ON - IoT Streaming Platform to integrate and build smart home, smart building and smart grid infrastructures

IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...

Kai Wähner

Chti jug - 2018-06-26

Florent Ramiere

Similar to MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu (20)

IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, K...

Beyond the Brokers: A Tour of the Kafka Ecosystem

Beyond the brokers - A tour of the Kafka ecosystem

Beyond the brokers - Un tour de l'écosystème Kafka

Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière

Kafka for Real-Time Replication between Edge and Hybrid Cloud

Dataops on streaming data: Kafka to InfluxDb via Kubernetes native flows

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...

Devoxx university - Kafka de haut en bas

Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transpo...

Bank of China Tech Talk 2: Introduction to Streaming Data and Stream Processi...

DataOps on Streaming Data: From Kafka to InfluxDB via Kubernetes Native Flows...

Spark Streaming + Kafka 0.10: an integration story by Joan Viladrosa Riera at...

Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...

JHipster conf 2019 - Kafka Ecosystem

IoT and Event Streaming at Scale with Apache Kafka

Streaming Data Ingest and Processing with Apache Kafka

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...

Chti jug - 2018-06-26

Recently uploaded

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...

FIDO Alliance

What's New in Teams Calling, Meetings and Devices April 2024

Stephanie Beckett

I'm excited to share my latest predictions on how AI, robotics, and other technological advancements will reshape industries in the coming years. The slides explore the exponential growth of computational power, the future of AI and robotics, and their profound impact on various sectors. Why this matters: The success of new products and investments hinges on precise timing and foresight into emerging categories. This deck equips founders, VCs, and industry leaders with insights to align future products with upcoming tech developments. These insights enhance the ability to forecast industry trends, improve market timing, and predict competitor actions. Highlights: ▪ Exponential Growth in Compute: How $1000 will soon buy the computational power of a human brain ▪ Scaling of AI Models: The journey towards beyond human-scale models and intelligent edge computing ▪ Transformative Technologies: From advanced robotics and brain interfaces to automated healthcare and beyond ▪ Future of Work: How automation will redefine jobs and economic structures by 2040 With so many predictions presented here, some will inevitably be wrong or mistimed, especially with potential external disruptions. For instance, a conflict in Taiwan could severely impact global semiconductor production, affecting compute costs and related advancements. Nonetheless, these slides are intended to guide intuition on future technological trends.

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl

Peter Udo Diehl

PLAI - Acceleration Program for Generative A.I. Startups

Stefano

Discover the top Symfony development companies that excel in creating robust and scalable web applications. Our latest blog highlights the best firms specializing in Symfony, known for their expertise in delivering high-performance solutions. Whether you’re looking to start a new project or enhance an existing one, these companies offer the skills and experience needed to bring your vision to life. Read more to find your perfect Symfony development partner.

Top 10 Symfony Development Companies 2024

TopCSSGallery

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

FIDO Alliance

This presentation dives into the practical applications of machine learning within Google's operations, providing a comprehensive overview of how to leverage AI technologies to solve real-world business challenges. Key Points Covered: - Introduction to Machine Learning at Google: Discussion on the role of ML and its evolution in enhancing Google's operational efficiency. - Experience Sharing: Insights into the team's long-term engagement with machine learning projects and the impacts on Google’s operational strategies. - Practical Applications: Real-world examples of ML applications within Google’s daily operations, providing a blueprint to adapt similar strategies. - Challenges and Solutions: Discussion on the challenges faced during the implementation of ML projects and the strategic solutions employed to overcome them. - Future of ML at Google: Insights into future trends in machine learning at Google and how they plan to continue integrating AI into their ecosystem.

Strategic AI Integration in Engineering Teams

UXDXConf

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Julian Hyde

Unlock the mysteries of successful Salesforce interviews in this insightful session hosted by Hugo Rosario (Salesforce Customer), a seasoned hiring manager that leads the Salesforce Department of multinational company with over 100 interviews under their belt. Step into the manager's chair and gain exclusive behind-the-scenes insights into what makes a Salesforce consultant stand out during the interview process. From deciphering the unspoken cues to mastering key strategies, we'll explore the intricacies of the interview process and provide practical tips for consultants looking to not only pass interviews but also thrive in their roles. Whether you're a seasoned professional or just starting your Salesforce journey, this session is your backstage pass to the secrets that hiring managers wish you knew.

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

CzechDreamin

Already know how to write a basic SOQL query? Great! But what about an *aggregate* SOQL query? You know, the kind that uses aggregate functions like COUNT & MAX along with GROUP BY and HAVING clauses? No? Well, get ready to learn how to slice & dice your org’s data right inside your own dev console. From finding duplicate records to prototyping summary & matrix reports, learn the ins and outs of aggregate queries during this fast-paced but admin-friendly session on advanced SOQL concepts.

SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...

CzechDreamin

How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf

FIDO Alliance

WSO2CONMay2024OpenSourceConferenceDebrief.pptx

Jennifer Lim

The Metaverse: Are We There Yet?

Mark Billinghurst

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

FIDO Alliance

WebAssembly is Key to Better LLM Performance

Samy Fodil

Extensible Python: Robustness through Addition - PyCon 2024

Patrick Viafore

Ever caught yourself nodding along when someone mentions "delivering value" in Agile, but secretly wondering what the heck they actually mean? You're not alone! Join us for an eye-opening session where we'll strip away the buzzwords and dive into the heart of Agile—value delivery. But what is "value"? Is it a mythical unicorn in the world of software development, or is there more to this overused term? This isn't going to be a sit-and-get lecture. We're talking about a face-to-face, interactive meetup where YOU play a crucial role. Come along to: Define It: What does "value" really mean? We’ll build a definition that’s not just words, but a compass for your Agile journey. Contextualise It: Discover what value means specifically to you, your team, your company, and your industry. Because one size does not fit all. Deliver It: Share strategies and gather new ones for uncovering and delivering true value—no more shooting in the dark!

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx

David Michel

Intrigued by why some of the world's largest companies (Netflix, Google, Cisco, Twitter, Uber etc) are using gRPC? In this demo based talk we delve into the world of gRPC in .Net, what it does and why we should use it. We compare the interface with both Rest and graphQL. We will show you how to implement grpc server-side in .net and in the web. Finally, I will show you how the tooling helps you deliver powerful interfaces and interact with them quickly and simply.

Demystifying gRPC in .Net by John Staveley

John Staveley

The standard Salesforce Approval process can be limiting in many ways, especially in complex scenarios. What if there was a way to implement very flexible approvals where one can use Apex code to make data updates in unrelated records, dynamically generate next steps details, and compute assignees on the fly? And still use UI-based configurations to implement concrete approval processes. In this session, we will share ideas behind such a solution and show a few lines of code to get you started.

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

CzechDreamin

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf

FIDO Alliance

Recently uploaded (20)

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...

What's New in Teams Calling, Meetings and Devices April 2024

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl

PLAI - Acceleration Program for Generative A.I. Startups

Top 10 Symfony Development Companies 2024

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

Strategic AI Integration in Engineering Teams

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...

How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf

WSO2CONMay2024OpenSourceConferenceDebrief.pptx

The Metaverse: Are We There Yet?

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

WebAssembly is Key to Better LLM Performance

Extensible Python: Robustness through Addition - PyCon 2024

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx

Demystifying gRPC in .Net by John Staveley

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf

MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu

1. Stefan Bocutiu @stepib London Apache Kafka Meetup, Feb 2018 End to end topologies on Apache Kafka   An IoT use case CONNECT ANALYSE REACT

2. About Today Intros IoT High Level Architecture Build the Pipeline Q&A

3. Hi! It’s Stefan!  @stepib  https://github.com/stheppi CONNECT ANALYSE REACT Streaming Data Processing & Integration  KAFKA - KUBERNETES

4. EVOLVESCALEBUILDIoT initiative challenges ★ 94% face challenges collecting and analysing IoT data ★ 26+ Billion devices by 2020, Gartner More Data, More Complexity ★ 2.500.000 trillion bytes produced daily REAL TIME DATA INGESTION + IMMEDIATE QUERIES =   ACTIONABLE / TIMELY INSIGHTS IoT Data Flows

5. IoT data flow store datastream & process datasensor data

6. IoT data flow store datastream & process datasensor data E T L

7. InﬂuxDBsensor data MQTT InﬂuxDB   Connector LENSES SQL PROCESSOR KAFKA   CONNECT KAFKA   CONNECT MQTT   Connector KAFKA STREAMS

8. InfluxDBsensor data MQTT InfluxDB   Connector LENSES SQL PROCESSOR KAFKA   CONNECT KAFKA   CONNECT MQTT   Connector KAFKA STREAMS Synthetic Data With Moquette  1 Broker,   data generate sensor data InfluxDB Docker

9. ® IoT & High Volumes, N devices  & irregular intervals Real Time Analytics   & Microservices Unlimited streams of data, async transfers  Producers & Consumers decoupled operation Kafka Streams API  Processing and analysing data in motion Multiple sources of data   & long term storeage Kafka Connect API,   move data with pluggable reusable & scalable connectors An open source streaming framework with messaging semantics where records are key-value pairs.

10. LENSES BOX  For Kafka Development Single docker image with all services Growing list of examples & synthetic data generators. Stream Reactor 25+ connectors Kafka CLI autocomplete. Rest and WebSocket endpoints Redux JS Client Testing Libraries FREE! All you need to start working with Apache Kafka ecosystem

11. MQTT Source connector  • Message Queueing Telemetry Transport messaging protocol • Optimised for connected devices and M2M protocol • Low bandwidth, high-latency, unreliable networks • Plain or SSL connections • Kafka Connect Query Language INSERT INTO kafka_topic SELECT * FROM /mqtt_source_topic [WITHCONVERTER=`myclass`] MQTT Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions

12. • Process data in motion with Lenses SQL • Create & Deploy Lenses SQL Processors • Native Kafka & Kubernetes Support Scalability • Visual Topology • Monitor and manage processors via LENSES Process & Analyse Data in motion

13. InﬂuxDB Sink connector  • AVRO & JSON Support • Error handling ( Noop, Retry, Throw) • Field selection & ﬁeld alias • Multiple inserts supported • Kafka Connect Query Language • MEASUREMENTS, KCQL: INSERTS • TAGS, KCQL: WITHTAG • TIMESTAMPS, KCQL: WITHTIMESTAMP • DURATION Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions INSERT INTO sensorMeasure   SELECT * FROM sensor_data_avg   WITHTAG (id, poweredBy=Lenses)

14. Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions InﬂuxDB Sink connector  • Timestamp nanosecond • KCQL • No ﬁlter support • No function support • Throughput involves domain knowlege • Kafka Connect rebalances

15. visual data flow Visual and Interactive Topology  for a 360 view of your data End-To-End Topology Management  Visualise your Kafka-based data flows  in real time

16. Wrap it up! CONNECT ANALYSE REACT Real Time, IoT pipelines in minutes! MQTT

17. Contact us www.landoop.com info@landoop.com Twitter @LandoopLtd Github github.com/landoop Gitter https://gitter.im/Landoop Slack https://launchpass.com/landoop-community Thank you!

MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu

Similar to MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu (20)

Recently uploaded

Recently uploaded (20)

MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu