SlideShare a Scribd company logo
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
Starting sooooon ..
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
IoT meets Confluent meets Data Platform
MQTT
Broker
OPC UA
gRPC
Proxy
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
IoT meets Confluent meets Data Platform
MQTT
Broker
OPC UA
gRPC
Proxy
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Goal
Partners Tech Talks are webinars where subject matter experts from a Partner talk about a
specific use case or project. The goal of Tech Talks is to provide best practices and
applications insights, along with inspiration, and help you stay up to date about innovations
in confluent ecosystem.
@yourtwitterhandle | developer.confluent.io
Starting soon…
STARTING SOOOOON..
Starting sooooon ..
@yourtwitterhandle | developer.confluent.io
Starting soon…
STARTING SOOOOON..
Starting sooooon ..
@yourtwitterhandle | developer.confluent.io
Starting soon…
STARTING SOOOOON..
Starting sooooon ..
Let’s win together!
Confluent Perspective Real Time Analytics
10
11
Business challenges Technical challenges
Increasingly complex application
environments and mounting pressures to
track and respond to every indicator and
issue.
Data latency and lack of end-to-end,
scalable observability for monitoring
behavior, performance, and health of
complex systems, applications, and
infrastructure.
Technology failures and security risks
that result in disruption to
customer-facing services and costly losses
for the business.
Higher operational costs due to more
troubleshooting time, bottlenecks, and
suboptimal performance requiring
additional resources/infrastructure.
INDUSTRY: ALL
12
Why Confluent
Stream
data everywhere, on premises and in every
major public cloud.
Connect
operational data like logs, metrics, and traces
from across your entire business including
on-prem, cloud, and hybrid environments.
Process
data streams to feed real-time analytics
applications that query and visualize critical
metrics at scale including latencies, error
rates, overall service health statuses, etc.
Govern
data to ensure quality, security, and
compliance while enabling teams to discover
and leverage existing data products.
Business impact
Enable early detection of system-wide issues
to prevent incidents and downtime.
Deliver proactive, faster responses to open
incidents for quicker resolution.
Gain the ability to deeply analyze all systems
and make more informed decisions.
INDUSTRY: ALL
Building a custom observability platform for your
business delivers long-term benefits
Scalability & Latency
Ingest, query, and visualize
mission-critical log data in real
time, at scale and under load.
Privacy & Cost Controls
Maintain more control over
data sensitive to your business
and optimize for cost savings.
Customization &
Flexibility
Tailor a solution and all
features to the specific needs
of your organization.
Confluent and Imply deliver a simple solution for
building a real-time observability platform
Confluent Cloud
Fully managed data streaming platform
Imply Polaris
Fully managed Apache Druid® service
Connect all of your data in real time with a
cloud-native and complete data streaming platform
available everywhere you need it.
Get all the speed and performance of Druid
without having to manage the database or
configure infrastructure.
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
©2023 Imply
©2023 imply
From Streaming Data to Real-Time
Insights: Building with Apache Druid
Hellmar Becker, Senior Sales Engineer
16
©2023 Imply
Agenda
17
● Introduction to Apache Druid
● K2D Architecture - Kafka to Druid
● K2D Use Cases and Case Studies
● About Imply / Demo
©2023 Imply
Apache Druid is a real-time analytics database
Sub-second queries at any scale
Interactive analytics on TB-PBs of data
High concurrency at the lowest cost
100s to 1000s QPS via a highly efficient engine
Real-time and historical insights
True stream ingestion for Kafka and Kinesis
Plus, non-stop reliability with automated fault
tolerance and continuous backup
1
2
3
For analytics applications that require:
How It Works: Streaming Analytics with Druid
©2023 Imply
Files
App data
Data sources
Microservices
Database
Events Streaming Infrastructure
Databases Data Lake
Stream
ETL
Stream
Processors
Messaging
Realtime
Analytics
Events Analytics Infrastructure
ML/AI
Dashboards
& reports
Interactive
Data Apps
BI tools
Machine
Driven
Queries
Sensor data
20
K2D Architecture - Kafka to Druid
©2023 Imply
Druid is built to analyze Kafka data
21
No connectors
needed.
Unlike other analytics
databases, Druid was
purpose-built to ingest data
from Kafka, so there’s no need
to “connect” the two.
Built-in
scalability.
Druid easily scales real-time
and batch ingestion up to
millions of events per second /
tens of TBs per minute.
Event-based
ingestion.
With Druid, events don’t have
to be persisted to storage
first—they are instantly
available to applications with
exactly-once semantics.
Fast analytical
queries.
Druid is built for fast
analytical queries on
real-time and historical data,
enabling contextual analysis
and real-time insights.
©2023 Imply 22
Real-time use cases for Kafka-to-Druid
ALERTING MONITORING DASHBOARDS EXPLORATION DECISIONING
State or stateless
triggered actions
Continuous
tracking of KPIs
User-facing
operational visibility
Ad-hoc rapid
data exploration
On high throughput Kafka streams
API-triggered
automated workflows
if X, then Y
©2022, Imply
Live Demo
23
Chargers
IoT Core
Kinesis
Stream
Metrics
Consumers
Kafka
Charger
sensor metric
produced
Medium charger
sensor metric
Transient charger
sensor metric
Telemetry
filtering Cosmos
Service
engineering
Imply - Druid
2 days
30
days
©2022, Imply
Confidential. Do not redistribute. © Imply
Previous state
25
● SQL Server was slow & had stability issues, as the
portal is forward facing to the end users, this
caused the UX to be poor when querying data
● Reporting (and selling) this data, sometimes
resulted in the query never completing
● Old UX was clunky and ‘slow’
● Costs of maintaining SQL Server DW was high
● SQL Server had scalability issues
● Hitting limits of 2M devices (controllers + sensors)
● Plan is to scale to 10M+ devices so SQL already
cannot deliver
Proprietary
Photovoltaic
systems
Controller
©2022, Imply
Confidential. Do not redistribute. © Imply
New state
26
● Real Time Visibility & Analysis — Solar PV & Battery
● Removed intermediate steps between source data and
the analytics interface by connecting Kafka to Imply
● Ability to scale virtually infinitely
● Visualize Yield & Consumption — live & historical data to
ensure yield
● Compare Yield — detect and resolve deviations to save
time and reduce loss
Photovoltaic
systems Controller
©2022, Imply
Swisscom, the biggest Telecom company in
Switzerland (Mobile, Internet, TV).
Using Imply since 2016
DEPLOYMENT TYPE & VOLUME:
Imply Hybrid & On prem | 235 TB data
INDUSTRY / COMPANY SIZE:
Telco company / 15,000+ employees
USE CASES:
Network Monitoring/Troubleshooting,
Self-Service analytics for Business teams
2 Use Cases
Network Monitoring/Troubleshooting,
Self-Service analytics for Business teams
20x
Number of users with access to KPIs
20min MTRS
Reduce the Mean Time to Restore
Service from hours to minutes
IMPLY: CUSTOMER STORIES (2022)
©2023 Imply
Committer Expertise 24/7 Support | 100% of the Original Creators
Imply Pivot Custom UI Tableau, etc
Effortless Operations Management Tools | Performance Monitoring
Cloud Deployment Fully-Managed (Polaris) | Hybrid-Managed
Druid Database Commercial Distribution | Enhanced Security
Committer-driven support.
✔
Flexible deployment options.
✔
Native Confluent Cloud
integration.
✔
Accelerated time-to-value.
✔
Get the scalability, elasticity,
and resilience of Druid, plus:
Imply completes the Druid experience
Imply Customers
Technology Platform Advertising
Visit imply.io/success-stories
Comms Gaming Financial
Confluent and Imply deliver a simple solution for
building a real-time observability platform
Confluent Cloud
Fully managed data streaming platform
Imply Polaris
Fully managed Apache Druid® service
Connect all of your data in real time with a
cloud-native and complete data streaming platform
available everywhere you need it.
Get all the speed and performance of Druid
without having to manage the database or
configure infrastructure.
Access Confluent data streams
from directly within Imply Polaris
Screenshot of Confluent data streams access
embedded within Imply Polaris
See Imply’s new data streaming integration during today’s demo
©2023 Imply
©2023, imply
Imply Polaris
Confluent Cloud
Start your free trials today
New signups receive $400 to use during
their first 30 days.
confluent.io/get-started imply.io/get-started
New signups receive $500 to use during
their first 30 days.
©2023 Imply
©2023, imply
Questions
33
hellmar.becker@imply.io
https://www.linkedin.com/in/hellmarbecker/
https://blog.hellmar-becker.de/
©2023 Imply
©2023, imply
Backup
©2022, Imply
Real-Time Analytics Applications
Real-time Analytics Database
+
New market requires a new kind of database
35
Analytics
Data Warehouses
Applications
Transactional Databases
Read-optimized
TB-PBs of Data
High Cardinality
Sub-Sec Response
High Concurrency
Real-time Data
BI Reporting
Monthly Reporting
Static Dashboards
ACID Compliance
Small Data
Write-optimized
BI Reporting
Monthly Reporting
Static Dashboards
ACID Compliance
Small Data
Write-optimized
✓
✓
✓
✓
✓
✓

More Related Content

Similar to Unlocking the Power of IoT: A comprehensive approach to real-time insights

Getting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of ConceptsGetting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of Concepts
ThousandEyes
 
Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...
Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...
Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...
HostedbyConfluent
 
Wavefront-by-VMware-April-2019
Wavefront-by-VMware-April-2019Wavefront-by-VMware-April-2019
Wavefront-by-VMware-April-2019
Anil Gupta (AJ) - vExpert
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
confluent
 
How to Build Streaming Apps with Confluent II
How to Build Streaming Apps with Confluent IIHow to Build Streaming Apps with Confluent II
How to Build Streaming Apps with Confluent II
confluent
 
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
NetworkCollaborators
 
Predix Builder Roadshow
Predix Builder RoadshowPredix Builder Roadshow
Predix Builder Roadshow
Predix
 
Confluent Messaging Modernization Forum
Confluent Messaging Modernization ForumConfluent Messaging Modernization Forum
Confluent Messaging Modernization Forum
confluent
 
ciscothousandeyesusecase
ciscothousandeyesusecaseciscothousandeyesusecase
ciscothousandeyesusecase
RENJITHKNAIR5
 
CloudOne Continuous Engineering for IoT
CloudOne Continuous Engineering for IoTCloudOne Continuous Engineering for IoT
CloudOne Continuous Engineering for IoT
Benjamin Chodroff
 
Azure Overview Csco
Azure Overview CscoAzure Overview Csco
Azure Overview Csco
rajramab
 
IoT Meetup September 2019
IoT Meetup September 2019IoT Meetup September 2019
IoT Meetup September 2019
IoT Academy
 
Contrail Launch: Capitalize on SDN and Cloud. Now.
Contrail Launch: Capitalize on SDN and Cloud. Now.Contrail Launch: Capitalize on SDN and Cloud. Now.
Contrail Launch: Capitalize on SDN and Cloud. Now.
Juniper Networks
 
abiquo
abiquoabiquo
abiquo
guestf5c2fa
 
VMworld 2014: Virtualization 101
VMworld 2014: Virtualization 101VMworld 2014: Virtualization 101
VMworld 2014: Virtualization 101
VMworld
 
Which Application Modernization Pattern Is Right For You?
Which Application Modernization Pattern Is Right For You?Which Application Modernization Pattern Is Right For You?
Which Application Modernization Pattern Is Right For You?
Apigee | Google Cloud
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
confluent
 
Accelerating Cloud Services - Intel
Accelerating Cloud Services - IntelAccelerating Cloud Services - Intel
Accelerating Cloud Services - Intel
Amazon Web Services
 
offerings managed services NOC : network operations center
offerings managed services NOC : network operations centerofferings managed services NOC : network operations center
offerings managed services NOC : network operations center
Arjun Kumar Marya
 
Confluent Partner Tech Talk with QLIK
Confluent Partner Tech Talk with QLIKConfluent Partner Tech Talk with QLIK
Confluent Partner Tech Talk with QLIK
confluent
 

Similar to Unlocking the Power of IoT: A comprehensive approach to real-time insights (20)

Getting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of ConceptsGetting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of Concepts
 
Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...
Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...
Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...
 
Wavefront-by-VMware-April-2019
Wavefront-by-VMware-April-2019Wavefront-by-VMware-April-2019
Wavefront-by-VMware-April-2019
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
How to Build Streaming Apps with Confluent II
How to Build Streaming Apps with Confluent IIHow to Build Streaming Apps with Confluent II
How to Build Streaming Apps with Confluent II
 
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
Cisco Connect 2018 Thailand - Enabling the next gen data center transformatio...
 
Predix Builder Roadshow
Predix Builder RoadshowPredix Builder Roadshow
Predix Builder Roadshow
 
Confluent Messaging Modernization Forum
Confluent Messaging Modernization ForumConfluent Messaging Modernization Forum
Confluent Messaging Modernization Forum
 
ciscothousandeyesusecase
ciscothousandeyesusecaseciscothousandeyesusecase
ciscothousandeyesusecase
 
CloudOne Continuous Engineering for IoT
CloudOne Continuous Engineering for IoTCloudOne Continuous Engineering for IoT
CloudOne Continuous Engineering for IoT
 
Azure Overview Csco
Azure Overview CscoAzure Overview Csco
Azure Overview Csco
 
IoT Meetup September 2019
IoT Meetup September 2019IoT Meetup September 2019
IoT Meetup September 2019
 
Contrail Launch: Capitalize on SDN and Cloud. Now.
Contrail Launch: Capitalize on SDN and Cloud. Now.Contrail Launch: Capitalize on SDN and Cloud. Now.
Contrail Launch: Capitalize on SDN and Cloud. Now.
 
abiquo
abiquoabiquo
abiquo
 
VMworld 2014: Virtualization 101
VMworld 2014: Virtualization 101VMworld 2014: Virtualization 101
VMworld 2014: Virtualization 101
 
Which Application Modernization Pattern Is Right For You?
Which Application Modernization Pattern Is Right For You?Which Application Modernization Pattern Is Right For You?
Which Application Modernization Pattern Is Right For You?
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
Accelerating Cloud Services - Intel
Accelerating Cloud Services - IntelAccelerating Cloud Services - Intel
Accelerating Cloud Services - Intel
 
offerings managed services NOC : network operations center
offerings managed services NOC : network operations centerofferings managed services NOC : network operations center
offerings managed services NOC : network operations center
 
Confluent Partner Tech Talk with QLIK
Confluent Partner Tech Talk with QLIKConfluent Partner Tech Talk with QLIK
Confluent Partner Tech Talk with QLIK
 

More from confluent

Building API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructureBuilding API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructure
confluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
confluent
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
confluent
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
confluent
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
confluent
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
confluent
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
confluent
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
confluent
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
confluent
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
confluent
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
confluent
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
confluent
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
confluent
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
confluent
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
confluent
 
The Journey to Data Mesh with Confluent
The Journey to Data Mesh with ConfluentThe Journey to Data Mesh with Confluent
The Journey to Data Mesh with Confluent
confluent
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
confluent
 

More from confluent (20)

Building API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructureBuilding API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructure
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
 
The Journey to Data Mesh with Confluent
The Journey to Data Mesh with ConfluentThe Journey to Data Mesh with Confluent
The Journey to Data Mesh with Confluent
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
 

Recently uploaded

Flutter vs. React Native: A Detailed Comparison for App Development in 2024
Flutter vs. React Native: A Detailed Comparison for App Development in 2024Flutter vs. React Native: A Detailed Comparison for App Development in 2024
Flutter vs. React Native: A Detailed Comparison for App Development in 2024
dhavalvaghelanectarb
 
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...
Ortus Solutions, Corp
 
Secure-by-Design Using Hardware and Software Protection for FDA Compliance
Secure-by-Design Using Hardware and Software Protection for FDA ComplianceSecure-by-Design Using Hardware and Software Protection for FDA Compliance
Secure-by-Design Using Hardware and Software Protection for FDA Compliance
ICS
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
kgyxske
 
The Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdfThe Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdf
mohitd6
 
Call Girls Bangalore🔥7023059433🔥Best Profile Escorts in Bangalore Available 24/7
Call Girls Bangalore🔥7023059433🔥Best Profile Escorts in Bangalore Available 24/7Call Girls Bangalore🔥7023059433🔥Best Profile Escorts in Bangalore Available 24/7
Call Girls Bangalore🔥7023059433🔥Best Profile Escorts in Bangalore Available 24/7
manji sharman06
 
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid
 
How GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdfHow GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdf
Zycus
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
campbellclarkson
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
Yara Milbes
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
sandeepmenon62
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
KrishnaveniMohan1
 
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
widenerjobeyrl638
 
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data PlatformAlluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio, Inc.
 
Going AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applicationsGoing AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applications
Alina Yurenko
 
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
kalichargn70th171
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
kalichargn70th171
 
Computer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdfComputer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdf
chandangoswami40933
 
Software Test Automation - A Comprehensive Guide on Automated Testing.pdf
Software Test Automation - A Comprehensive Guide on Automated Testing.pdfSoftware Test Automation - A Comprehensive Guide on Automated Testing.pdf
Software Test Automation - A Comprehensive Guide on Automated Testing.pdf
kalichargn70th171
 

Recently uploaded (20)

Flutter vs. React Native: A Detailed Comparison for App Development in 2024
Flutter vs. React Native: A Detailed Comparison for App Development in 2024Flutter vs. React Native: A Detailed Comparison for App Development in 2024
Flutter vs. React Native: A Detailed Comparison for App Development in 2024
 
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...
 
Secure-by-Design Using Hardware and Software Protection for FDA Compliance
Secure-by-Design Using Hardware and Software Protection for FDA ComplianceSecure-by-Design Using Hardware and Software Protection for FDA Compliance
Secure-by-Design Using Hardware and Software Protection for FDA Compliance
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
 
The Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdfThe Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdf
 
Call Girls Bangalore🔥7023059433🔥Best Profile Escorts in Bangalore Available 24/7
Call Girls Bangalore🔥7023059433🔥Best Profile Escorts in Bangalore Available 24/7Call Girls Bangalore🔥7023059433🔥Best Profile Escorts in Bangalore Available 24/7
Call Girls Bangalore🔥7023059433🔥Best Profile Escorts in Bangalore Available 24/7
 
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
 
How GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdfHow GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdf
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
 
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
 
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data PlatformAlluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
 
Going AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applicationsGoing AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applications
 
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
 
Computer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdfComputer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdf
 
Software Test Automation - A Comprehensive Guide on Automated Testing.pdf
Software Test Automation - A Comprehensive Guide on Automated Testing.pdfSoftware Test Automation - A Comprehensive Guide on Automated Testing.pdf
Software Test Automation - A Comprehensive Guide on Automated Testing.pdf
 

Unlocking the Power of IoT: A comprehensive approach to real-time insights

  • 1. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)? Starting soon… STARTING SOOOOON.. Starting sooooon ..
  • 2. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)? Starting soon… STARTING SOOOOON..
  • 3. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. IoT meets Confluent meets Data Platform MQTT Broker OPC UA gRPC Proxy
  • 4. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. IoT meets Confluent meets Data Platform MQTT Broker OPC UA gRPC Proxy
  • 5. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)?
  • 6. Goal Partners Tech Talks are webinars where subject matter experts from a Partner talk about a specific use case or project. The goal of Tech Talks is to provide best practices and applications insights, along with inspiration, and help you stay up to date about innovations in confluent ecosystem.
  • 7. @yourtwitterhandle | developer.confluent.io Starting soon… STARTING SOOOOON.. Starting sooooon ..
  • 8. @yourtwitterhandle | developer.confluent.io Starting soon… STARTING SOOOOON.. Starting sooooon ..
  • 9. @yourtwitterhandle | developer.confluent.io Starting soon… STARTING SOOOOON.. Starting sooooon .. Let’s win together!
  • 10. Confluent Perspective Real Time Analytics 10
  • 11. 11 Business challenges Technical challenges Increasingly complex application environments and mounting pressures to track and respond to every indicator and issue. Data latency and lack of end-to-end, scalable observability for monitoring behavior, performance, and health of complex systems, applications, and infrastructure. Technology failures and security risks that result in disruption to customer-facing services and costly losses for the business. Higher operational costs due to more troubleshooting time, bottlenecks, and suboptimal performance requiring additional resources/infrastructure. INDUSTRY: ALL
  • 12. 12 Why Confluent Stream data everywhere, on premises and in every major public cloud. Connect operational data like logs, metrics, and traces from across your entire business including on-prem, cloud, and hybrid environments. Process data streams to feed real-time analytics applications that query and visualize critical metrics at scale including latencies, error rates, overall service health statuses, etc. Govern data to ensure quality, security, and compliance while enabling teams to discover and leverage existing data products. Business impact Enable early detection of system-wide issues to prevent incidents and downtime. Deliver proactive, faster responses to open incidents for quicker resolution. Gain the ability to deeply analyze all systems and make more informed decisions. INDUSTRY: ALL
  • 13. Building a custom observability platform for your business delivers long-term benefits Scalability & Latency Ingest, query, and visualize mission-critical log data in real time, at scale and under load. Privacy & Cost Controls Maintain more control over data sensitive to your business and optimize for cost savings. Customization & Flexibility Tailor a solution and all features to the specific needs of your organization.
  • 14. Confluent and Imply deliver a simple solution for building a real-time observability platform Confluent Cloud Fully managed data streaming platform Imply Polaris Fully managed Apache Druid® service Connect all of your data in real time with a cloud-native and complete data streaming platform available everywhere you need it. Get all the speed and performance of Druid without having to manage the database or configure infrastructure.
  • 15. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)? Starting soon… STARTING SOOOOON..
  • 16. ©2023 Imply ©2023 imply From Streaming Data to Real-Time Insights: Building with Apache Druid Hellmar Becker, Senior Sales Engineer 16
  • 17. ©2023 Imply Agenda 17 ● Introduction to Apache Druid ● K2D Architecture - Kafka to Druid ● K2D Use Cases and Case Studies ● About Imply / Demo
  • 18. ©2023 Imply Apache Druid is a real-time analytics database Sub-second queries at any scale Interactive analytics on TB-PBs of data High concurrency at the lowest cost 100s to 1000s QPS via a highly efficient engine Real-time and historical insights True stream ingestion for Kafka and Kinesis Plus, non-stop reliability with automated fault tolerance and continuous backup 1 2 3 For analytics applications that require:
  • 19. How It Works: Streaming Analytics with Druid
  • 20. ©2023 Imply Files App data Data sources Microservices Database Events Streaming Infrastructure Databases Data Lake Stream ETL Stream Processors Messaging Realtime Analytics Events Analytics Infrastructure ML/AI Dashboards & reports Interactive Data Apps BI tools Machine Driven Queries Sensor data 20 K2D Architecture - Kafka to Druid
  • 21. ©2023 Imply Druid is built to analyze Kafka data 21 No connectors needed. Unlike other analytics databases, Druid was purpose-built to ingest data from Kafka, so there’s no need to “connect” the two. Built-in scalability. Druid easily scales real-time and batch ingestion up to millions of events per second / tens of TBs per minute. Event-based ingestion. With Druid, events don’t have to be persisted to storage first—they are instantly available to applications with exactly-once semantics. Fast analytical queries. Druid is built for fast analytical queries on real-time and historical data, enabling contextual analysis and real-time insights.
  • 22. ©2023 Imply 22 Real-time use cases for Kafka-to-Druid ALERTING MONITORING DASHBOARDS EXPLORATION DECISIONING State or stateless triggered actions Continuous tracking of KPIs User-facing operational visibility Ad-hoc rapid data exploration On high throughput Kafka streams API-triggered automated workflows if X, then Y
  • 24. Chargers IoT Core Kinesis Stream Metrics Consumers Kafka Charger sensor metric produced Medium charger sensor metric Transient charger sensor metric Telemetry filtering Cosmos Service engineering Imply - Druid 2 days 30 days
  • 25. ©2022, Imply Confidential. Do not redistribute. © Imply Previous state 25 ● SQL Server was slow & had stability issues, as the portal is forward facing to the end users, this caused the UX to be poor when querying data ● Reporting (and selling) this data, sometimes resulted in the query never completing ● Old UX was clunky and ‘slow’ ● Costs of maintaining SQL Server DW was high ● SQL Server had scalability issues ● Hitting limits of 2M devices (controllers + sensors) ● Plan is to scale to 10M+ devices so SQL already cannot deliver Proprietary Photovoltaic systems Controller
  • 26. ©2022, Imply Confidential. Do not redistribute. © Imply New state 26 ● Real Time Visibility & Analysis — Solar PV & Battery ● Removed intermediate steps between source data and the analytics interface by connecting Kafka to Imply ● Ability to scale virtually infinitely ● Visualize Yield & Consumption — live & historical data to ensure yield ● Compare Yield — detect and resolve deviations to save time and reduce loss Photovoltaic systems Controller
  • 27. ©2022, Imply Swisscom, the biggest Telecom company in Switzerland (Mobile, Internet, TV). Using Imply since 2016 DEPLOYMENT TYPE & VOLUME: Imply Hybrid & On prem | 235 TB data INDUSTRY / COMPANY SIZE: Telco company / 15,000+ employees USE CASES: Network Monitoring/Troubleshooting, Self-Service analytics for Business teams 2 Use Cases Network Monitoring/Troubleshooting, Self-Service analytics for Business teams 20x Number of users with access to KPIs 20min MTRS Reduce the Mean Time to Restore Service from hours to minutes IMPLY: CUSTOMER STORIES (2022)
  • 28. ©2023 Imply Committer Expertise 24/7 Support | 100% of the Original Creators Imply Pivot Custom UI Tableau, etc Effortless Operations Management Tools | Performance Monitoring Cloud Deployment Fully-Managed (Polaris) | Hybrid-Managed Druid Database Commercial Distribution | Enhanced Security Committer-driven support. ✔ Flexible deployment options. ✔ Native Confluent Cloud integration. ✔ Accelerated time-to-value. ✔ Get the scalability, elasticity, and resilience of Druid, plus: Imply completes the Druid experience
  • 29. Imply Customers Technology Platform Advertising Visit imply.io/success-stories Comms Gaming Financial
  • 30. Confluent and Imply deliver a simple solution for building a real-time observability platform Confluent Cloud Fully managed data streaming platform Imply Polaris Fully managed Apache Druid® service Connect all of your data in real time with a cloud-native and complete data streaming platform available everywhere you need it. Get all the speed and performance of Druid without having to manage the database or configure infrastructure.
  • 31. Access Confluent data streams from directly within Imply Polaris Screenshot of Confluent data streams access embedded within Imply Polaris See Imply’s new data streaming integration during today’s demo
  • 32. ©2023 Imply ©2023, imply Imply Polaris Confluent Cloud Start your free trials today New signups receive $400 to use during their first 30 days. confluent.io/get-started imply.io/get-started New signups receive $500 to use during their first 30 days.
  • 35. ©2022, Imply Real-Time Analytics Applications Real-time Analytics Database + New market requires a new kind of database 35 Analytics Data Warehouses Applications Transactional Databases Read-optimized TB-PBs of Data High Cardinality Sub-Sec Response High Concurrency Real-time Data BI Reporting Monthly Reporting Static Dashboards ACID Compliance Small Data Write-optimized BI Reporting Monthly Reporting Static Dashboards ACID Compliance Small Data Write-optimized ✓ ✓ ✓ ✓ ✓ ✓