Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dataops on streaming data: Kafka to InfluxDb via Kubernetes native flows


Published on presentation slides for InfluxDays 2019

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Dataops on streaming data: Kafka to InfluxDb via Kubernetes native flows

  1. 1. DataOps on streaming data Kafka to InfluxDb via Kubernetes Native Flows - IoT demo Chris Kotsis
  2. 2. #About Enterprise DataOps over Streaming Data
  3. 3. #About me
  4. 4. #DataOps Embrace Collaboration Eliminate Friction, turn data to Value Faster
  5. 5. #DataOps Personas people who ● Collect and Prepare the data ● Analyse the data ● Use the findings for business value
  6. 6. #DataOps IoT Challenges Enormous Volumes of Data “500 billion connected devices by 2025”
  7. 7. #DataOps IoT Challenges Enormous Volumes of Data “500 billion connected devices by 2025” Its Management and Analysis will become harder and continue to break traditional tools
  8. 8. The IoT Data Flow
  9. 9. How to implement a Streaming Architecture for IoT Time-Series data and Real-Time insights?
  10. 10. Infrastructure Layer Where the Data lives Self-service Data Access, Multi-tenancy, Security, Governance to Accessibility & Visibility for ALL
  11. 11. IoT & kafka High Volumes, N devices & irregular intervals Real Time Analytics & Microservices Multiple sources of data & long term storage An open source streaming framework with messaging semantics where records are key-value pairs Unlimited streams of data, async transfers Producers & Consumers decoupled operation Kafka Streams API Processing and analysing data in motion Kafka Connect API, Move data with pluggable reusable & scalable connectors
  12. 12. Application Layer Multiple Distributed Microservices for complex Data Flows
  13. 13. Application Layer Provisioning, Monitoring, Alerting, Security, Governance, Accessibility, Self-service deployments,....
  14. 14. How Lenses integrates with the Application Layer?
  15. 15. Lenses SQL Processors ● Simply Filter, Enrich, Split & Bind your data ● Manipulate Live Streams of Data ● Scalable (Kubernetes Native)
  16. 16. Connect Kafka to InfluxDB Real Time Ingestion Distributed Fault tolerance Scalability Error Handling Monitoring & Alerting Governance & Security Easy Data Manipulation ...
  17. 17. Lenses Connectors: Binding external sources is a few clicks away
  18. 18. InfluxDB sink connector by Kafka to InfluxDB No code required CLI/UI/API/Monitor SQL Support Multiple inserts supported AVRO & JSON Support Supported Features: MEASUREMENTS (SQL: INSERT TO) TIMESTAMPS (SQL: WITHTIMESTAMP) TAGS (SQL: WITHTAG) DURATION
  19. 19. The uncontrolled “Chaos” of modern data streams
  20. 20. Lenses Topology, one view to bind them all
  21. 21. What it takes to demo this architecture
  22. 22. Lenses Box ● Always Free for Developers ● Single Broker setup with the ecosystem services ● 25+ Open source Kafka connectors ● Synthetic Data generators and running examples ● Lenses intuitive UI ● The powerful Lenses SQL ● Works on your Laptop, works on Cloud
  23. 23. Demo
  24. 24. Thank you and <3 Q&A