Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu

221 views

Published on

Build an end-to-end Kafka-based pipeline in minutes!
Enhanced Data pipelines with no code.
An Apache Kafka tutorial with Lenses by Stefan Bocutiu.

Published in: Technology
  • Be the first to comment

MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu

  1. 1. Stefan Bocutiu @stepib London Apache Kafka Meetup, Feb 2018 End to end topologies on Apache Kafka 
 An IoT use case CONNECT ANALYSE REACT
  2. 2. About Today Intros IoT High Level Architecture Build the Pipeline Q&A
  3. 3. Hi! It’s Stefan!
 @stepib
 https://github.com/stheppi CONNECT ANALYSE REACT Streaming Data Processing & Integration
 KAFKA - KUBERNETES
  4. 4. EVOLVESCALEBUILDIoT initiative challenges ★ 94% face challenges collecting and analysing IoT data ★ 26+ Billion devices by 2020, Gartner More Data, More Complexity ★ 2.500.000 trillion bytes produced daily REAL TIME DATA INGESTION + IMMEDIATE QUERIES = 
 ACTIONABLE / TIMELY INSIGHTS IoT Data Flows
  5. 5. IoT data flow store datastream & process datasensor data
  6. 6. IoT data flow store datastream & process datasensor data E T L
  7. 7. InfluxDBsensor data MQTT InfluxDB 
 Connector LENSES SQL PROCESSOR KAFKA 
 CONNECT KAFKA 
 CONNECT MQTT 
 Connector KAFKA STREAMS
  8. 8. InfluxDBsensor data MQTT InfluxDB 
 Connector LENSES SQL PROCESSOR KAFKA 
 CONNECT KAFKA 
 CONNECT MQTT 
 Connector KAFKA STREAMS Synthetic Data With Moquette
 1 Broker, 
 data generate sensor data InfluxDB Docker
  9. 9. ® IoT & High Volumes, N devices
 & irregular intervals Real Time Analytics 
 & Microservices Unlimited streams of data, async transfers
 Producers & Consumers decoupled operation Kafka Streams API
 Processing and analysing data in motion Multiple sources of data 
 & long term storeage Kafka Connect API, 
 move data with pluggable reusable & scalable connectors An open source streaming framework with messaging semantics where records are key-value pairs.
  10. 10. LENSES BOX
 For Kafka Development Single docker image with all services Growing list of examples & synthetic data generators. Stream Reactor 25+ connectors Kafka CLI autocomplete. Rest and WebSocket endpoints Redux JS Client Testing Libraries FREE! All you need to start working with Apache Kafka ecosystem
  11. 11. MQTT Source connector
 • Message Queueing Telemetry Transport messaging protocol • Optimised for connected devices and M2M protocol • Low bandwidth, high-latency, unreliable networks • Plain or SSL connections • Kafka Connect Query Language INSERT INTO kafka_topic SELECT * FROM /mqtt_source_topic [WITHCONVERTER=`myclass`] MQTT Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions
  12. 12. • Process data in motion with Lenses SQL • Create & Deploy Lenses SQL Processors • Native Kafka & Kubernetes Support Scalability • Visual Topology • Monitor and manage processors via LENSES Process & Analyse Data in motion
  13. 13. InfluxDB Sink connector
 • AVRO & JSON Support • Error handling ( Noop, Retry, Throw) • Field selection & field alias • Multiple inserts supported • Kafka Connect Query Language • MEASUREMENTS, KCQL: INSERTS • TAGS, KCQL: WITHTAG • TIMESTAMPS, KCQL: WITHTIMESTAMP • DURATION Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions INSERT INTO sensorMeasure 
 SELECT * FROM sensor_data_avg 
 WITHTAG (id, poweredBy=Lenses)
  14. 14. Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions InfluxDB Sink connector
 • Timestamp nanosecond • KCQL • No filter support • No function support • Throughput involves domain knowlege • Kafka Connect rebalances
  15. 15. visual data flow Visual and Interactive Topology
 for a 360 view of your data End-To-End Topology Management
 Visualise your Kafka-based data flows
 in real time
  16. 16. Wrap it up! CONNECT ANALYSE REACT Real Time, IoT pipelines in minutes! MQTT
  17. 17. Contact us www.landoop.com info@landoop.com Twitter @LandoopLtd Github github.com/landoop Gitter https://gitter.im/Landoop Slack https://launchpass.com/landoop-community Thank you!

×