This document discusses using Kafka to stream IoT time-series data from devices to InfluxDB for real-time analytics and insights. It introduces Lenses.io as an open-source streaming data platform that enables connecting Kafka to InfluxDB via reusable connectors without code. The document demonstrates how Lenses provides a unified view and SQL support for processing, filtering, and manipulating streaming data across infrastructure and application layers in a distributed, scalable manner on Kubernetes.

1. DataOps
on streaming data
Kafka to InfluxDb via Kubernetes
Native Flows - IoT demo
Chris Kotsis
lenses.io

2. #About lenses.io
Enterprise DataOps over Streaming Data

3. #About me

4. #DataOps
Embrace Collaboration
Eliminate Friction,
turn data to Value Faster

5. #DataOps Personas
people who
● Collect and Prepare the data
● Analyse the data
● Use the findings for business value

6. #DataOps IoT Challenges
Enormous Volumes of Data
“500 billion connected devices by 2025”

7. #DataOps IoT Challenges
Enormous Volumes of Data
“500 billion connected devices by 2025”
Its Management and Analysis will become
harder and continue to break traditional tools

8. The IoT Data Flow

9. How to implement a Streaming
Architecture for IoT Time-Series
data and Real-Time insights?

10. Infrastructure Layer
Where the Data lives
Self-service Data Access,
Multi-tenancy,
Security, Governance to
Accessibility & Visibility
for ALL

12. IoT & kafka
High Volumes, N devices & irregular intervals
Real Time Analytics & Microservices
Multiple sources of data & long term storage
An open source streaming framework with messaging
semantics where records are key-value pairs
Unlimited streams of data, async transfers
Producers & Consumers decoupled operation
Kafka Streams API
Processing and analysing data in motion
Kafka Connect API,
Move data with pluggable reusable & scalable connectors

13. Application Layer
Multiple Distributed
Microservices for
complex Data Flows

14. Application Layer
Provisioning,
Monitoring, Alerting,
Security,
Governance,
Accessibility,
Self-service
deployments,....

16. How Lenses integrates
with the Application Layer?

17. Lenses SQL Processors
● Simply Filter, Enrich, Split & Bind your data
● Manipulate Live Streams of Data
● Scalable (Kubernetes Native)

18. Connect Kafka to InfluxDB
Real Time Ingestion
Distributed
Fault tolerance
Scalability
Error Handling
Monitoring & Alerting
Governance & Security
Easy Data Manipulation
...

19. Lenses Connectors: Binding external
sources is a few clicks away

20. InfluxDB sink connector by Lenses.io
Kafka to InfluxDB
No code required
CLI/UI/API/Monitor
SQL Support
Multiple inserts supported
AVRO & JSON Support
Supported Features:
MEASUREMENTS (SQL: INSERT TO)
TIMESTAMPS (SQL: WITHTIMESTAMP)
TAGS (SQL: WITHTAG)
DURATION

21. The uncontrolled “Chaos”
of modern data streams

22. Lenses Topology, one view to bind them all

25. What it takes to demo this architecture

26. Lenses Box
● Always Free for Developers
● Single Broker setup with the
ecosystem services
● 25+ Open source Kafka
connectors
● Synthetic Data generators and
running examples
● Lenses intuitive UI
● The powerful Lenses SQL
● Works on your Laptop, works
on Cloud

27. Demo

28. Thank you and <3
Q&A