Kafka communicates within a larger complex and evolving environment. The current modular approach to the integration means that the structure of the software stack is much more dynamic than in the past and operators no longer have the time to become intimate with how dependent components interact. The number of dependencies combined with lack of familiarity can create significant risks to the business including increased outages and longer time to resolve incidents. Both can result in loss of revenue and customers. These risks are significantly reduced by applying best-practice monitoring. Monitoring can provide a complete end-to-end view of the touch points within the application flow, so they are presented in comprehensive service-based views. This provides the user with a true single-pane of glass for monitoring and alerting for Kafka and its dependent technologies.

1. Reduce Risk with End to End Monitoring of Middleware-based Applications
Rob Bakker
Senior Engineer
Grahame Aldus
Senior Developer
End to End Monitoring for Complex Applications

2. Title:
Complex Open Source Applications like Apache Kafka Provide Great Benefits but Come with Greater Risk Learn
to-End Monitoring Tools
Abstract:
Kafka communicates within a larger complex and evolving environment. The current modular approach to the
software stack is much more dynamic than in the past and operators no longer have the time to become
interact. The number of dependencies combined with lack of familiarity can create significant risks to the
longer time to resolve incidents. Both can result in loss of revenue and customers.
These risks are significantly reduced by applying best-practice monitoring. Monitoring can provide a complete
the application flow, so they are presented in comprehensive service-based views. This provides the user with a
and alerting for Kafka and its dependent technologies.
Kafka Meetup

3. • Bay Area-based monitoring & visualization software company
• Founded in 1987
• Private-label versions of product resold by TIBCO, Solace, Software AG
SL

4. • Presentation 101 says
• “know your audience”
• “take the temperature of the room”
• Questions:
• Who Here Is / would consider themselves
• Developers, Ops, or Dev/Ops?
• Who here is using Kafka in production?
• Who here is using a monitoring system for Kafka?
Audience Participation!

5. Modern Apps Are Highly Heterogeneous
Modern apps are built using open source and commercial technologies from multiple sources
RTView Solution Packages
TIBCO BW 5/6
Red Hat JBoss
BPM
Hazelcast
TIBCO EMS TIBCO FTL
SOA & Workflow
Messaging Servers
In-Memory Data Grid
App Servers
Java Processes
Databases
Virtual Machines
IBM WebSphere
MQ
RabbitMQ
Oracle WebLogic IBM WebSphere Red Hat JBoss Apache Tomcat Pivotal tc Server
Oracle
Coherence
JVM
Oracle MySQL IBM DB2 MS SQL Server
Amazon AWS
EC2
VMware vSphere
TIBCO
BusinessEvents
Docker
MongoDB
Node.js
TIBCO
BusinessConnect
TIBCO API
Exchange
TIBCO BW CE
Solace Apache Kafka Mulesoft
TIBCO
ActiveSpaces Terracota

6. Modern Apps Are Highly Heterogeneous
RTView Solution Packages
TIBCO BW 5/6
Red Hat JBoss
BPM
TIBCO EMS TIBCO FTL
SOA & Workflow
Messaging Servers
In-Memory Data Grid
App Servers
Java Processes
Databases
Virtual Machines
IBM WebSphere
MQ
RabbitMQ
Oracle WebLogic IBM WebSphere Red Hat JBoss Apache Tomcat Pivotal tc Server
Oracle
Coherence
JVM
Oracle MySQL IBM DB2 MS SQL Server
Amazon AWS
EC2
VMware vSphere
TIBCO
BusinessEvents
Docker
MongoDB
Node.js
TIBCO
BusinessConnect
TIBCO API
Exchange
TIBCO BW CE
Solace
Apache
Kafka Mulesoft
And a typical custom application will use different technologies across multiple tiers
And may have workflows running on-premise, in containers, in public or private clouds.
Or all four!
Hazelcast
TIBCO
ActiveSpaces
Terracota

7. • As application stacks become more dynamic and deployment options increase,
ensuring application uptime becomes more challenging
• Component interdependencies within a business service can increase
exponentially
• Monitoring of application and service components often an afterthought with no
centralized monitoring & alerting
• Monitoring, when present, tends to be siloed around specific technologies,
often as part of the admin tool
• Support teams are often stuck in a reactive mode rather than proactively
responding to issues early
Risks

8. End-to-End Middleware-centric Monitoring
Monitoring and alerting for complex, distributed applications and services built
on middleware
• Monitor dependencies and the middleware technologies and components
that support a critical business service
• Understand if a problem occurs in an application, how that will affect other
components used by the service
• Correlate information across multiple middleware platforms

9. Real-time Health Status of Critical Apps & Infrastructure

10. Key Metrics
• The most important performance metrics for a specific technology
• Key Metrics views show how close a metric is approaching its threshold
over a period of time – both before and after the alert threshold is
reached

11. • Track potential threats before they
show up as alerts
• Correlate real-time and historical
metrics of multiple resources that
support a critical application
• Understand how health problems
in one component may be caused
by performance problems in
another component – across tiers
and vendors
Need to Understand How Component Issues Affect Business Services

12. Context with Historical Metrics
• Enable more effective analysis by viewing information in a historical context
• Improve understanding of performance issues by persisting real-time
metrics to a database
• Automate data management tasks for compaction to reduce granularity and
size of dataset over time

13. Service-level alerts enable rapid identification of component issues

14. Service-level alerts enable rapid identification of component issues

15. Component Level View

16. Component ID
Service model
• Associates individual architecture
components hierarchically with business
services – both static and dynamic info
• Enables identification of service impact for
an individual component problem
• Easily created and maintained
Owner
Area
Group
Service
JPeters
Operations-
U.S.
Mortgage
Derivatives
Trading &
Operations
Middleware Analytics Inventory
Workflow Inventory MgmtOrder Mgmt
CI CICI CI CI CI CI
RMorrissey

17. Flow diagrams can provide context for understanding complex services and processes
Context Is King for Understanding Complex Services

18. Flow diagrams can provide context for understanding complex services and processes
Context Is King for Understanding Complex Services

19. • Don’t put critical technologies like Kafka into production without adequate
monitoring
• Augment technology-specific monitoring with end-to-end monitoring when the
technology is part of a critical business service or big money application
• Integrate alerting with third-party tools such as ServiceNow or enterprise
monitoring applications to fit into existing work flow for application and
middleware support teams
• Capture metric and alert history to baseline expected performance and manage
capacity
Best Practices for End-to-End Middleware Monitoring