2. 2
Why Monitoring
• Applications often contain frustrating blind spots and mysterious,
recurring problems that are often difficult to identify, diagnose, and fix.
• With business-critical applications, it’s very important to have a simple
and fast way to monitor, diagnose, and resolve application problems
before they affect revenue.
4. 4
ADF Performance Monitor
• Advanced tool designed for measuring, analyzing, improving, and
checking the performance and health of Oracle ADF applications
• Tracks and collects crucial production performance information of the
application’s runtime ADF components that are not standard provided by
Oracle
• Helps development, QA, and operation teams to detect, analyze and
resolve common and less common issues in response times, health and
resource usage of ADF applications
• Can be used during development, test and production
• Is a library attached to your application
• Less than 1 day for implementation
• Needs to add a very small amount of code to application to make metrics
classes active
6. 6
Month, Week, Day, Hour, 5
Minute Overview
• Shows real-time critical information about the ADF application's performance
• Are response times within or outside SLA boundaries? Is immediate action required?
• What is the error rate? What is the health of the JVM?
7. 7
Monitor SLA
• from 15:00 to 16:00 the response times have increased significantly (more
red and yellow colored parts in the 15u bar)
• This should be a trigger to investigate
8. 8
Monitor Details of All HTTP
Requests
– Client Response Time (including network to client and Browser load time)
9. 9
Monitor HTTP Request Details
of Specific End-Users (1)
– Select an end-user name in list
– Performance and problems of specific end-users can now be analyzed
10. 10
Monitor Requests of Specific
End-Users (2)
– When and where did the end-user click ?
– What were the response times of this end-user ?
17. 17
Example Network/Browser
Load Time – Resolved
• Next day the network problem was resolved
- We can see this in the following hour overview graph were there is much less purple colour (right bottom):
18. 18
Zoom in Into WLS Managed
Server (or Monitor All Together)
• Select / Filter on Managed Server
• Compare Performance between Managed Servers
20. 20
Quickly Troubleshoot Customer
Complains
• All errors with all relevant detail information are recorded
• Error stack traces and call stacks are collected to allow the developer to
quickly troubleshoot these errors
• This information helps to discover, analyze and resolve errors that happen
frequently on test / production environments
24. 24
Very Easy to Find Root Cause
of Problems
• A call stack provides visibility into which ADF methods caused other
methods to be executed, organized by the sequence of their execution.
– A complete breakdown of the ADF request processing, including all ADF method
executions, along with elapsed times is printed, organized by lifecycle phase.
– Inefficiencies can be identified from this report, for example long running ViewObject
queries, slow ApplicationModule activations & passivations, e.g.
33. 33
ADF BC Memory Analyzer (2)
– Example: in this production ADF app, for a single ViewObject instance,
more than 900.000 rows were loaded in the ADF app (Blue) !
34. – In this production ADF app, for a single ViewObject instance,
more than 900.000 rows were loaded in the ADF app (!)
– Loading 900.000 rows takes more than 10 minutes
34
Loaded Rows in Memory (3)-
ADF CallStack
36. 36
ADF Performance Monitor
Warnings and Detailed Help
• Fetching thousands of rows from database
• Slow passivation and activation of transient attributes
• Redundant ViewObject query executions
• Inefficient ViewObject fetchsize
• Inefficient PageDef iterator rangesize
• Slow executions of
– ViewObject queries
– EntityObject DML operations, database locks
– ApplicationModule activations / passivations / transactions
– PageDefinition operations / ApplicationModule custom methods
– PageDefinition iterators
– Webservices
– Java methods
• More
37. 37
The tool can be dynamically turned
on and off (at runtime) at all times
• The ADF Performance Monitor can be dynamically turned on and off (at
runtime) at all times in the Enterprise Manager
• When the monitor is turned off there is no performance overhead because
the metrics classes are not active
39. 39
Configuration
• All configuration steps are well documented in the installation guide
– An ADF library needs to be added to your ADF application
– A very small amount of code needs to be added to instrument the ADF application.
– The metrics classes make use of extension points in the ADF framework in order to
measure the time every action/step takes in a http request.
• Implementation takes less than 1 day, usually no more than a few hours
40. 40
Overhead
• Load tests show that the overhead is less than 3 to 4 percent.
• This overhead is caused by the very detailed collection of performance
metrics.
• The metrics collection can be dynamically turned on and off (at runtime) at
all times.
• When the monitor is turned off there is no performance overhead because
the metrics classes are not active.
41. 41
Support
• Standard Oracle stack
– ADF Business Components
– ADF Model
– ADF Controller
– ADF Faces
• JPA/EJB
– EclipseLink/TopLink
• POJO JavaBean DataControl
• JAX-WS Webservices
• It is possible to instrument (calls to) custom Java methods and 3rd party
libraries
• Application servers:
– WebLogic
– Glassfish
– Tomcat
43. 43
Henri Peters - Project Manager
Dutch Ministry of Justice
“For us it is of great value that we can continuously identify the
weakest link, even if there is no actual performance problem (yet)."
“We can even drill down to the level of an individual user.
Sometimes we go so far that we proactively approach end-users:
before they complain we already have contacted them.”
* More quotes of ADF experts and managers at:
http://tinyurl.com/nxgkgr6 or http://www.amis.nl/ADFperformancemonitor
45. 45
Whitepaper
Whitepaper has detailed information about the architecture and implementation
http://tinyurl.com/qgfh22u or http://www.amis.nl/ADFperformancemonitor
Content:
Executive overview
Introduction
Oracle ADF applications and performance
ADF Performance Monitor overview
Use in JDeveloper
Use in test and production environment
Dashboard reporting application
Summary and details HTTP response times
ADF framework call stack
Warnings and suggested solutions
Worst performing executions in ADF BC and model layer
Error stacktraces
JVM performance
Product architecture
Configuration
Turn on/off at all times
Prerequisites
Monitored Events
Objectives:
-be able to instrument ADF to track performance
-use performance tracking data to improve performance
Applications often contain frustrating blind spots and mysterious, recurring problems that are often difficult to identify, diagnose, and fix.
With business-critical applications, it’s very important to have a simple and fast way to monitor, diagnose, and resolve application problems before they affect revenue.
Troubleshoot problems of specific end-user
Troubleshoot problems of specific end-user
Several overview graphs are added to the main dashboard. At the day overview, it shows for each hour of the day (right bottom) the layer where the time is spent. It shows the time spent in the database, webservice, application server, and network/browser load time:
At the top right graph (hourly overview of day performance) we can see a lot of red in the bars. Specifically from 11:00 to 12:00 – apparantly there were many very slow requests. In the graph at the right bottom we can now explain what happened; there were network problems. After talking to the infrastructure department – this was indeed the case. They were very busy with maintanance and were not aware (!) that end-users were suffering from there work.
In the top graph we see all individual http requests in managed server regi_adf_server1. Again, the stacked bars show the time spent in database, webservice, application server, and network/browser load time (purple). We can see that many end-users have to wait ± 5 seconds – on pure network time (!). Note that The JVM is not the problem and is working fine (bottom graph).
The next day the network problem was resolved. We can see this in the following hour overview graph were there is much less purple colour (right bottom):
The next day the network problem was resolved. We can see this in the following hour overview graph were there is much less purple colour (right bottom):
24/7 Overview
The dashboard (Figure 3) gives a 24/7 overview of the performance of the application in a selected time range.
It opens with a 24 hours overview for the current date. It shows real-time critical information about the ADF application's performance (are response times within or outside SLA boundaries? What is the error rate? What is the health of the JVM? Is immediate action required?).
In the ADF BC overview (filtered on ApplicationModule pooling) the ApplicationModule pooling performance can be analyzed. This overview gives more insight in the effect of these parameter settings (how often passivations and activations happen, how long their AVG and total execution time is). On each ApplicationModule activation/passivation can be drilled down, to see the ADF request call stacks. Dev and QA teams can research and experiment with these parameter settings on test environments during load tests and evaluate the performance results in the monitor. They can determine the most optimal ApplicationModule pooling parameter settings for their situation. Operation teams can monitor the ApplicationModule pooling behavior in production.
The performance impact of frequent invoked activations and passivations can be much higher than those that occur only a few times but are very slow on average. It is important to know which activations/passivations have the biggest impact on the resources of your ADF application. In this overview the x-as shows the AVG execution time (seconds) and the y-as shows the occurrences. The size of the bubble is total time (AVG execution time * occurrences).
We can see that PjtShowPaga001AM.passivateState, PjtShowPaga001AM.activateState and Pagd001AM.passivateState are executed very frequently. If we don’t want this behavior, we can adjust the pooling parameter settings.