Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DataOps on Streaming Data: From Kafka to InfluxDB via Kubernetes Native Flows by Christos Kotsis, Reliability Engineer | Lenses.io

306 views

Published on

In this session, we are going to create a Lenses DataOps hub for IoT data with Apache Kafka and InfluxDB flows over Kubernetes. We will demonstrate how to create streaming flows and securely explore and monitor real-time data. We will use Kubernetes to spin up scalable flows and go through how we can simply provision such flows with secret management and monitoring end to end out capabilities.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

DataOps on Streaming Data: From Kafka to InfluxDB via Kubernetes Native Flows by Christos Kotsis, Reliability Engineer | Lenses.io

  1. 1. DataOps on streaming data Kafka to InfluxDb via Kubernetes Native Flows - IoT demo Chris Kotsis lenses.io
  2. 2. #About me Reliability Engineer at Lenses.io Data Flows on Kubernetes Kafka Cloud Native techs DataOps Expert
  3. 3. #About lenses.io Enterprise DataOps over Streaming Data
  4. 4. #About lenses.io Enterprise DataOps over Streaming Data
  5. 5. #About lenses.io Enterprise DataOps over Streaming Data all Distributions - on Cloud - on Premises
  6. 6. #DataOps Embrace Collaboration Eliminate Friction, turn data to Value Faster
  7. 7. #DataOps Personas people who ● Collect and Prepare the data ● Analyse the data ● Use the findings for business value
  8. 8. #DataOps IoT Challenges Enormous Volumes of Data “500 billion connected devices by 2025”
  9. 9. #DataOps IoT Challenges Enormous Volumes of Data “500 billion connected devices by 2025” Its Management and Analysis will become harder and continue to break traditional tools
  10. 10. The IoT Data Flow
  11. 11. How to implement a Streaming Architecture for IoT Time-Series data and Real-Time insights?
  12. 12. Infrastructure Layer Where the Data lives Self-service Data Access, Multi-tenancy, Security, Governance to Accessibility & Visibility for ALL
  13. 13. IoT & kafka High Volumes, N devices & irregular intervals Real Time Analytics & Microservices Multiple sources of data & long term storage An open source streaming framework with messaging semantics where records are key-value pairs ➔ Unlimited streams of data, ➔ Producers & Consumers (pub/sub) ➔ Processing and analysing data in motion ➔ Connect API to move data with pluggable reusable components
  14. 14. The flow implementation ➔ Sensor produces data to Kafka ➔ Create my Data Flows: ◆ Kafka - to - Kafka processing ◆ Route to InfluxDB
  15. 15. Application Layer Multiple Distributed Microservices for complex Data Flows
  16. 16. Application Layer Provisioning, Monitoring, Alerting, Security, Governance, Accessibility, Self-service deployments,....
  17. 17. How Lenses integrates with the Application Layer?
  18. 18. Lenses SQL Processors ● Simply Filter, Enrich, Split & Bind your data ● Manipulate Live Streams of Data ● Scalable: Cloud / Kubernetes Native PROCESS
  19. 19. Connect Kafka to InfluxDB Real Time Ingestion Distributed Fault tolerance Scalability Error Handling Monitoring & Alerting Governance & Security Easy Data Manipulation ... INGEST
  20. 20. Lenses Connectors: Binding external sources and sinks is a few clicks away
  21. 21. InfluxDB sink connector by Lenses.io Kafka to InfluxDB No code required CLI/UI/API/Monitor SQL Support Multiple inserts supported AVRO & JSON Support Supported Features: MEASUREMENTS (SQL: INSERT TO) TIMESTAMPS (SQL: WITHTIMESTAMP) TAGS (SQL: WITHTAG) DURATION
  22. 22. The uncontrolled “Chaos” of modern data streams
  23. 23. Lenses Topology, one view to bind them all
  24. 24. Demo
  25. 25. What it takes to demo this architecture
  26. 26. Lenses Box ● FREE for Developers ● Single Broker setup ● All ecosystem services ● 25+ Connectors ● Synthetic Data generators ● Live examples ● Lenses intuitive UI ● The powerful Lenses SQL ● Works on your Laptop ● Works on Cloudhttps://lenses.io/lenses-box
  27. 27. Thank you and <3
  28. 28. https://lenses.io Follow us @lensesio

×