1.
DataOps
on streaming data
Kafka to InfluxDb via Kubernetes
Native Flows - IoT demo
Chris Kotsis
lenses.io

2.
#About me
Reliability Engineer at Lenses.io
Data Flows on Kubernetes
Kafka
Cloud Native techs
DataOps Expert

3.
#About lenses.io
Enterprise DataOps over Streaming Data

4.
#About lenses.io
Enterprise DataOps over Streaming Data

5.
#About lenses.io
Enterprise DataOps over Streaming Data
all Distributions - on Cloud - on Premises

6.
#DataOps
Embrace Collaboration
Eliminate Friction,
turn data to Value Faster

7.
#DataOps Personas
people who
● Collect and Prepare the data
● Analyse the data
● Use the findings for business value

8.
#DataOps IoT Challenges
Enormous Volumes of Data
“500 billion connected devices by 2025”

9.
#DataOps IoT Challenges
Enormous Volumes of Data
“500 billion connected devices by 2025”
Its Management and Analysis will become
harder and continue to break traditional tools

10.
The IoT Data Flow

11.
How to implement a Streaming
Architecture for IoT Time-Series
data and Real-Time insights?

12.
Infrastructure Layer
Where the Data lives
Self-service Data Access,
Multi-tenancy,
Security, Governance to
Accessibility & Visibility
for ALL

13.
IoT & kafka
High Volumes, N devices &
irregular intervals
Real Time Analytics &
Microservices
Multiple sources of data &
long term storage
An open source streaming framework
with messaging semantics where
records are key-value pairs
➔ Unlimited streams of data,
➔ Producers & Consumers
(pub/sub)
➔ Processing and analysing data
in motion
➔ Connect API to move data with
pluggable reusable components

14.
The flow implementation
➔ Sensor produces data to Kafka
➔ Create my Data Flows:
◆ Kafka - to - Kafka
processing
◆ Route to InfluxDB

15.
Application Layer
Multiple Distributed
Microservices for
complex Data Flows

16.
Application Layer
Provisioning,
Monitoring, Alerting,
Security,
Governance,
Accessibility,
Self-service
deployments,....

17.
How Lenses integrates
with the Application Layer?

18.
Lenses SQL Processors
● Simply Filter, Enrich, Split & Bind your data
● Manipulate Live Streams of Data
● Scalable: Cloud / Kubernetes Native
PROCESS

19.
Connect Kafka to InfluxDB
Real Time Ingestion
Distributed
Fault tolerance
Scalability
Error Handling
Monitoring & Alerting
Governance & Security
Easy Data Manipulation
...
INGEST

20.
Lenses Connectors: Binding external
sources and sinks is a few clicks away

21.
InfluxDB sink connector by Lenses.io
Kafka to InfluxDB
No code required
CLI/UI/API/Monitor
SQL Support
Multiple inserts supported
AVRO & JSON Support
Supported Features:
MEASUREMENTS (SQL: INSERT TO)
TIMESTAMPS (SQL: WITHTIMESTAMP)
TAGS (SQL: WITHTAG)
DURATION

22.
The uncontrolled “Chaos”
of modern data streams

23.
Lenses Topology, one view to bind them all

24.
Demo

25.
What it takes to demo this architecture

26.
Lenses Box
● FREE for Developers
● Single Broker setup
● All ecosystem services
● 25+ Connectors
● Synthetic Data generators
● Live examples
● Lenses intuitive UI
● The powerful Lenses SQL
● Works on your Laptop
● Works on Cloudhttps://lenses.io/lenses-box

27.
Thank you and <3

28.
https://lenses.io
Follow us @lensesio