Continuous Performance Monitoring of a Distributed Application [CON4730]

Insert Picture Here
Continuous Performance
Monitoring of a Distributed
Application
Ashish Srivastava
Principal Member of Technical Staff
Diana Yuryeva
Senior Member of Technical Staff

The following is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and
should not be relied upon in making purchasing decisions. The development,
release, and timing of any features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.

Session Goal
§ Components:
– Design patterns
– Tools
§ Qualities:
– Continuous
– Light-weight
– Recordable
Arrive at solution for extreme performance monitoring

Session Agenda
§ Use Case
§ Software Patterns
§ Tools
§ Pitfalls and Advice
§ Q&A

About Us
§ Oracle Billing and Revenue Management Elastic Charging Engine
– 100% real-time charging application
– Java
– Distributed grid
– Oracle Coherence
– Oracle NoSQL
– Focus on extreme performance

Operating Conditions
§ Low latency expectations
§ Heavy system load
§ Distributed environment
§ Multi-level software stack

Monitoring Requirements
§ Detailed insight about performance
– Latency
– Throughput
§ View over time
§ Reporting
§ Bottleneck detection
§ View of system as cohesive unit
Functional

Monitoring Requirements
§ Minimal impact on processing
§ Ease of use
§ Separation of concerns
Non-Functional

Session Agenda
§ Use Case
§ Tools
§ Q&A

Approach
§ Off-the-shelf software not sufficient
§ Custom development needed
– Incorporate monitoring into system
– Collect, analyze and present metrics
How do I address these requirements?

Collecting Metrics
§ Goal
– Incorporate metrics collection into general processing
§ Approach
– Enhance domain model with monitoring-related data structures
Problem overview

Collecting Metrics
Model of sample system
ECE Client
Network Mediation
A
B
B'
C
C'
A'
A
Node1
Node3
Node2
request
―Debatch the requests
―Data lookups
―Apply Tariff
―Save Session
―Prepare Response
―..
response

Collecting Metrics
Solution
ECE Client
Network Mediation
A
B
B'
C
C'
A'
A
Node1
Node3
Node2
request
―Debatch the requests
―Data lookups
―Apply Tariff
―Save Session
―Prepare Response
―..
response

Client Node
Processing
Node
Envelope
Routing Context
Payload
Tracking Context Chronicler
– TimePoints
Stat Reporterharvest
Envelope

Collecting Metrics
##### Elapsed time = 3600 seconds
##### Avg throughput = 20000 ops/sec
##### Avg latency = 50 ms
Result

Granular Reporting
§ Goal
– I need more granular reporting of performance over time
§ Approach
– Enhance reporting of collected metrics
Problem overview

Granular Reporting
Solution – data structure
Chronicler removed
―
A moving reporting window
―
100% reporting
―
Sampled reporting
―
Stats exposed over JMX
―
Fixed data set for a window
Chronicler added

Granular Reporting
Solution – class diagram

Granular Reporting
§ I can see min/max/avg latency and throughput over time
§ My throughput reporting is quite good: I can see whether I had
stable or erratic throughput
Result

Latency Percentile Report
§ Goal
– Latencies are still not detailed enough. I need to know more than the
average/min/max latencies
– Need to guarantee that 99.999% of the requests take less than 55ms
§ Approach
– Introduce range bucketing to count latencies
Problem overview

§ Pre-defined buckets of latency percentiles
§ Data set does not grow. Each bucket is updated
§ Multiple percentile breakdown
– End-to-end
– Server side processing
– Per batch reporting
Solution

2013-03-21 08:44:29.112 PDT INFO ##### Latency statistics based on percentiles:
Percentile: 0.1, Latency: 1ms, Total Count: 1173148
Result – printed report

Result – heat map

Method Breakdown
§ Goal
– I want to measure the impact of a new method under varying load
– End-to-end latency always ON
– Minimum performance impact
§ Approach
– Method annotations
– Aspect
Problem overview

Method Breakdown
Solution
public enum LabelEnum {
APPLY_TARIFF,
...
DEBATCH
}
public class ClassToBeTracked {
@Track(pointLabel = LabelEnum.APPLY_TARIFF)
private <ReturnObject> method(<Parameters>) {
...
}
}
<pointcut name="scope" expression="within(ClassName) "/>

Method Breakdown
Result – detailed breakdown report
2013-07-15 16:29:24.953 PDT
Chronicler Breakdown:
DEBATCH -> 64149 nanoseconds
LOOKUP_DATA -> 1056748 nanoseconds
APPLY_TARIFF -> 99994 nanoseconds
SAVE_SESSION -> 12989 nanoseconds
PREPARE_RESPONSE -> 15998 nanoseconds

Session Agenda
§ Use Case
§ Tools
§ Q&A

Storing and Presenting Metrics
§ Goal
– I collect detailed performance metrics, but I need to report them too
– I need a tool which stores these metrics and presents them in a unified
view
§ Approach
– Create monitoring dashboard
– Technologies: JRDS,RRD and in-house development
Problem overview

Result – monitoring dashboard
Configuration:
Topology 24 servers
Throughput 20000 ops/sec
Duration 10 hrs

§ Graphical
§ Supports various metrics
– Application-specific
– Machine-specific
– JVM-specific
§ Consolidated view
– All graphs on one page
Solution qualities

§ Easy to use
– Collects and saves data automatically
§ Easy to share
– Includes configuration for future references
– Send links to web pages
– Print page as PDF
Solution qualities

§ Stores data without losing precision
§ Supports drilling down
§ Light-weight
§ Customizable
Solution qualities

Session Agenda
§ Use Case
§ Tools
§ Q&A

Pitfalls and Advice
§ Distributed system monitoring != Single JVM monitoring
– Consolidated view is critical
§ Consistency of tools across team is important
– Same language across development, QE and Performance teams saves
hours
§ Solution should enable you to be agile
– Run monitoring on laptop AND realistic setup
Take into consideration

Pitfalls and Advice
§ These hide problems
– Averaging
– Sampling
§ GC has big impact, so include it in your metrics
§ Watch our for processes sharing the same host
§ Always run long-duration tests
Some things to pay attention to

Session Summary
 Detailed insight about performance
– Latency
– Throughput
 View over time
 Reporting
 Bottleneck detection
 View of system as cohesive unit
Let's see how we addressed original requirements

Session Summary
 Detailed insight about performance
– Latency
– Throughput
 View over time
 Reporting

Session Summary
– Latency
– Throughput
 View over time
 Reporting

Session Summary
– Latency
– Throughput
 View over time
 Reporting

Session Summary
– Latency
– Throughput
 View over time
 Reporting
 Bottleneck detection

Session Summary
– Latency
– Throughput
 View over time
 Reporting
 Bottleneck detection
 View of system as cohesive unit

Session Agenda
§ Use Case
§ Tools
§ Q&A

Links
§ ECE
– http://www.oracle.com/us/products/applications/communications/elastic-
charging-engine
§ JRDS
– http://www.jrds.fr

Continuous Performance Monitoring of a Distributed Application [CON4730]

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to Continuous Performance Monitoring of a Distributed Application [CON4730]

Similar to Continuous Performance Monitoring of a Distributed Application [CON4730] (20)

Recently uploaded

Recently uploaded (20)

Continuous Performance Monitoring of a Distributed Application [CON4730]