The document outlines best practices for debugging client applications, including producers and consumers in Kafka environments. It emphasizes the challenges businesses face with complex application environments, data latency, and operational costs, advocating for a real-time observability platform integrating Apache Druid and Confluent Cloud for enhanced data processing and monitoring. Key benefits include early issue detection, faster incident response, and the ability to analyze systems for informed decision-making.

1. @yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
Starting sooooon ..

2. @yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..

3. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
IoT meets Confluent meets Data Platform
MQTT
Broker
OPC UA
gRPC
Proxy

4. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
IoT meets Confluent meets Data Platform
MQTT
Broker
OPC UA
gRPC
Proxy

5. @yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?

6. Goal
Partners Tech Talks are webinars where subject matter experts from a Partner talk about a
specific use case or project. The goal of Tech Talks is to provide best practices and
applications insights, along with inspiration, and help you stay up to date about innovations
in confluent ecosystem.

7. @yourtwitterhandle | developer.confluent.io
Starting soon…
STARTING SOOOOON..
Starting sooooon ..

8. @yourtwitterhandle | developer.confluent.io
Starting soon…
STARTING SOOOOON..
Starting sooooon ..

9. @yourtwitterhandle | developer.confluent.io
Starting soon…
STARTING SOOOOON..
Starting sooooon ..
Let’s win together!

10. Confluent Perspective Real Time Analytics
10

11. 11
Business challenges Technical challenges
Increasingly complex application
environments and mounting pressures to
track and respond to every indicator and
issue.
Data latency and lack of end-to-end,
scalable observability for monitoring
behavior, performance, and health of
complex systems, applications, and
infrastructure.
Technology failures and security risks
that result in disruption to
customer-facing services and costly losses
for the business.
Higher operational costs due to more
troubleshooting time, bottlenecks, and
suboptimal performance requiring
additional resources/infrastructure.
INDUSTRY: ALL

12. 12
Why Confluent
Stream
data everywhere, on premises and in every
major public cloud.
Connect
operational data like logs, metrics, and traces
from across your entire business including
on-prem, cloud, and hybrid environments.
Process
data streams to feed real-time analytics
applications that query and visualize critical
metrics at scale including latencies, error
rates, overall service health statuses, etc.
Govern
data to ensure quality, security, and
compliance while enabling teams to discover
and leverage existing data products.
Business impact
Enable early detection of system-wide issues
to prevent incidents and downtime.
Deliver proactive, faster responses to open
incidents for quicker resolution.
Gain the ability to deeply analyze all systems
and make more informed decisions.
INDUSTRY: ALL

13. Building a custom observability platform for your
business delivers long-term benefits
Scalability & Latency
Ingest, query, and visualize
mission-critical log data in real
time, at scale and under load.
Privacy & Cost Controls
Maintain more control over
data sensitive to your business
and optimize for cost savings.
Customization &
Flexibility
Tailor a solution and all
features to the specific needs
of your organization.

14. Confluent and Imply deliver a simple solution for
building a real-time observability platform
Confluent Cloud
Fully managed data streaming platform
Imply Polaris
Fully managed Apache Druid® service
Connect all of your data in real time with a
cloud-native and complete data streaming platform
available everywhere you need it.
Get all the speed and performance of Druid
without having to manage the database or
configure infrastructure.

15. @yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..

16. ©2023 Imply
©2023 imply
From Streaming Data to Real-Time
Insights: Building with Apache Druid
Hellmar Becker, Senior Sales Engineer
16

17. ©2023 Imply
Agenda
17
● Introduction to Apache Druid
● K2D Architecture - Kafka to Druid
● K2D Use Cases and Case Studies
● About Imply / Demo

18. ©2023 Imply
Apache Druid is a real-time analytics database
Sub-second queries at any scale
Interactive analytics on TB-PBs of data
High concurrency at the lowest cost
100s to 1000s QPS via a highly efficient engine
Real-time and historical insights
True stream ingestion for Kafka and Kinesis
Plus, non-stop reliability with automated fault
tolerance and continuous backup
1
2
3
For analytics applications that require:

19. How It Works: Streaming Analytics with Druid

20. ©2023 Imply
Files
App data
Data sources
Microservices
Database
Events Streaming Infrastructure
Databases Data Lake
Stream
ETL
Stream
Processors
Messaging
Realtime
Analytics
Events Analytics Infrastructure
ML/AI
Dashboards
& reports
Interactive
Data Apps
BI tools
Machine
Driven
Queries
Sensor data
20
K2D Architecture - Kafka to Druid

21. ©2023 Imply
Druid is built to analyze Kafka data
21
No connectors
needed.
Unlike other analytics
databases, Druid was
purpose-built to ingest data
from Kafka, so there’s no need
to “connect” the two.
Built-in
scalability.
Druid easily scales real-time
and batch ingestion up to
millions of events per second /
tens of TBs per minute.
Event-based
ingestion.
With Druid, events don’t have
to be persisted to storage
first—they are instantly
available to applications with
exactly-once semantics.
Fast analytical
queries.
Druid is built for fast
analytical queries on
real-time and historical data,
enabling contextual analysis
and real-time insights.

22. ©2023 Imply 22
Real-time use cases for Kafka-to-Druid
ALERTING MONITORING DASHBOARDS EXPLORATION DECISIONING
State or stateless
triggered actions
Continuous
tracking of KPIs
User-facing
operational visibility
Ad-hoc rapid
data exploration
On high throughput Kafka streams
API-triggered
automated workflows
if X, then Y

23. ©2022, Imply
Live Demo
23

24. Chargers
IoT Core
Kinesis
Stream
Metrics
Consumers
Kafka
Charger
sensor metric
produced
Medium charger
sensor metric
Transient charger
sensor metric
Telemetry
filtering Cosmos
Service
engineering
Imply - Druid
2 days
30
days

25. ©2022, Imply
Confidential. Do not redistribute. © Imply
Previous state
25
● SQL Server was slow & had stability issues, as the
portal is forward facing to the end users, this
caused the UX to be poor when querying data
● Reporting (and selling) this data, sometimes
resulted in the query never completing
● Old UX was clunky and ‘slow’
● Costs of maintaining SQL Server DW was high
● SQL Server had scalability issues
● Hitting limits of 2M devices (controllers + sensors)
● Plan is to scale to 10M+ devices so SQL already
cannot deliver
Proprietary
Photovoltaic
systems
Controller

26. ©2022, Imply
Confidential. Do not redistribute. © Imply
New state
26
● Real Time Visibility & Analysis — Solar PV & Battery
● Removed intermediate steps between source data and
the analytics interface by connecting Kafka to Imply
● Ability to scale virtually infinitely
● Visualize Yield & Consumption — live & historical data to
ensure yield
● Compare Yield — detect and resolve deviations to save
time and reduce loss
Photovoltaic
systems Controller

27. ©2022, Imply
Swisscom, the biggest Telecom company in
Switzerland (Mobile, Internet, TV).
Using Imply since 2016
DEPLOYMENT TYPE & VOLUME:
Imply Hybrid & On prem | 235 TB data
INDUSTRY / COMPANY SIZE:
Telco company / 15,000+ employees
USE CASES:
Network Monitoring/Troubleshooting,
Self-Service analytics for Business teams
2 Use Cases
Network Monitoring/Troubleshooting,
Self-Service analytics for Business teams
20x
Number of users with access to KPIs
20min MTRS
Reduce the Mean Time to Restore
Service from hours to minutes
IMPLY: CUSTOMER STORIES (2022)

28. ©2023 Imply
Committer Expertise 24/7 Support | 100% of the Original Creators
Imply Pivot Custom UI Tableau, etc
Effortless Operations Management Tools | Performance Monitoring
Cloud Deployment Fully-Managed (Polaris) | Hybrid-Managed
Druid Database Commercial Distribution | Enhanced Security
Committer-driven support.
✔
Flexible deployment options.
✔
Native Confluent Cloud
integration.
✔
Accelerated time-to-value.
✔
Get the scalability, elasticity,
and resilience of Druid, plus:
Imply completes the Druid experience

29. Imply Customers
Technology Platform Advertising
Visit imply.io/success-stories
Comms Gaming Financial

30. Confluent and Imply deliver a simple solution for
building a real-time observability platform
Confluent Cloud
Fully managed data streaming platform
Imply Polaris
Fully managed Apache Druid® service
Connect all of your data in real time with a
cloud-native and complete data streaming platform
available everywhere you need it.
Get all the speed and performance of Druid
without having to manage the database or
configure infrastructure.

31. Access Confluent data streams
from directly within Imply Polaris
Screenshot of Confluent data streams access
embedded within Imply Polaris
See Imply’s new data streaming integration during today’s demo

32. ©2023 Imply
©2023, imply
Imply Polaris
Confluent Cloud
Start your free trials today
New signups receive $400 to use during
their first 30 days.
confluent.io/get-started imply.io/get-started
New signups receive $500 to use during
their first 30 days.

33. ©2023 Imply
©2023, imply
Questions
33
hellmar.becker@imply.io
https://www.linkedin.com/in/hellmarbecker/
https://blog.hellmar-becker.de/

34. ©2023 Imply
©2023, imply
Backup

35. ©2022, Imply
Real-Time Analytics Applications
Real-time Analytics Database
+
New market requires a new kind of database
35
Analytics
Data Warehouses
Applications
Transactional Databases
Read-optimized
TB-PBs of Data
High Cardinality
Sub-Sec Response
High Concurrency
Real-time Data
BI Reporting
Monthly Reporting
Static Dashboards
ACID Compliance
Small Data
Write-optimized
BI Reporting
Monthly Reporting
Static Dashboards
ACID Compliance
Small Data
Write-optimized
✓
✓
✓
✓
✓
✓