Troubleshooting and Tuning WebLogic Jeffrey West Sr. Principal Product Manager
Program  Agenda  <ul><li>General Troubleshooting Tools & Tips </li></ul><ul><li>JRockit JVM </li></ul><ul><li>JRockit Miss...
Troubleshooting & Tuning Areas of Focus Where to start? <ul><li>Application </li></ul><ul><li>Application Server / Middlew...
The Near-Ideal ‘Performance Funnel’ <ul><li>In a well-designed distributed application, I/O becomes the bottleneck </li></...
The ‘Performance Funnel’: Where you start <ul><li>It’s OK - This is where you are supposed to start </li></ul><ul><li>Firs...
Steps along the way… <ul><li>As you open the flood gates, performance tuning is required down the stack </li></ul><ul><li>...
Oracle Overall Strategy <ul><li>Deliver a  complete ,  open ,  integrated  stack of hardware, infrastructure, database, mi...
<Insert Picture Here> Troubleshooting  Tools
Where there’s smoke… <ul><li>Application </li></ul><ul><ul><li>Java Profilers </li></ul></ul><ul><ul><li>Thread Dumps </li...
Our Focus Today <ul><li>Application </li></ul><ul><ul><li>Java Profilers </li></ul></ul><ul><ul><li>Thread Dumps </li></ul...
<Insert Picture Here> JRockit  JVM
The JRockit JVM <ul><li>Written for server-side applications </li></ul><ul><ul><li>Machines with a lot of physical memory ...
<Insert Picture Here> WebLogic  Monitoring & Troubleshooting
Overview of the WebLogic Diagnostics Framework Retrieve Expose Process Capture <ul><li>WLST API </li></ul><ul><li>JMX API ...
What is the JRockit Flight Recorder? <ul><li>New in JRockit R28 </li></ul><ul><li>“ Circular buffer” in JRockit JVM that s...
<ul><li>What it is designed for? </li></ul><ul><ul><li>Provide diagnostic information in running production systems </li><...
WebLogic/Flight Recorder Integration Goals and Points Integration Goals Integration Points 1 2 3 <ul><li>View and correlat...
WebLogic Diagnostic Events for Flight Recorder Integration <ul><li>Point-cuts woven into WebLogic to generate diagnostic e...
Easy Configuration with WLDF Volume <ul><li>Simple configuration: enabled and configured using a volume control per server...
WLDF Low Volume Events <ul><li>WLDF Logging Snapshot </li></ul><ul><li>WLDF LogRecord Snapshot </li></ul><ul><li>WLDF WLLo...
WLDF Medium Volume Events <ul><li>EJB Home Create </li></ul><ul><li>EJB Home Remove </li></ul><ul><li>EJB PoolManager Crea...
WLDF High Volume Events <ul><li>EJB Database Access </li></ul><ul><li>EJB Business Method Post Invoke Cleanup </li></ul><u...
Automated Recording Capture <ul><li>Integrated with the WebLogic Diagnostic Image and Watch and Notification system </li><...
WebLogic Experimental Plug-in for Mission Control 3
WebLogic Experimental Plugin
<Insert Picture Here> WebLogic  Tuning
Areas of Focus <ul><li>JVM’s and Machines </li></ul><ul><li>Threads / Work Managers </li></ul><ul><li>JMS </li></ul><ul><l...
The timeless battle with Garbage Collection <ul><li>How do you slice & dice your CPU and memory? </li></ul><ul><li>More JV...
JVM’s and Machines <ul><li>Machines: </li></ul><ul><ul><li>8GB RAM </li></ul></ul><ul><ul><li>Dual-CPU, Quad-Core </li></u...
Threads / Work Managers <ul><li>Define a work manager for each of your applications </li></ul><ul><li>WebLogic will self-t...
JMS <ul><li>Connection Factories </li></ul><ul><ul><li>Enable Load Balancing </li></ul></ul><ul><ul><li>Disable Server Aff...
JDBC <ul><li>Use JNDI Datasources! </li></ul><ul><ul><li>Don’t use application-level connection pools – you can’t monitor ...
Common Application ‘mistakes’ <ul><li>Read a properties from a file on each request </li></ul><ul><ul><li>Poor performance...
 
Base Content Slide <ul><li>First-level bullet </li></ul><ul><ul><li>Second-level bullet </li></ul></ul><ul><ul><ul><li>Thi...
Example of Title Extending to Two Lines <ul><li>First-level bullet </li></ul><ul><ul><li>Second-level bullet </li></ul></u...
Alternate Title with Subhead Arial Size 18 (Gray) <ul><li>First-level bullet </li></ul><ul><ul><li>Second-level bullet </l...
Upcoming SlideShare
Loading in …5
×

WebLogic Developer Webcast 5: Troubleshooting and Testing with WebLogic, SoapUI, JRockit Flight Recorder and JRockit Mission Control

4,803 views

Published on

1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total views
4,803
On SlideShare
0
From Embeds
0
Number of Embeds
518
Actions
Shares
0
Downloads
255
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide
  • Hi! This is Jeff West. Thanks for joining us for the fifth installment in our webcast series for developers. Today we will be covering Troubleshooting and Tuning with WebLogic.
  • Here is the agenda for the session. First, we will cover troubleshooting and tuning in a general sense and then we will focus on JRockit and WebLogic. After the presentation, we will have a demo of load testing with soapUI, monitoring with WebLogic Console and JRockit Mission Control and then diagnostics with JRockit Flight Recorder.
  • There are many layers of an application stack that can contribute to poor performance, scalability or throughput. You can take a top-down approach starting with your application and moving down the stack, but the best approach is to take a holistic approach and look at multiple levels of the stack at the same time when you are doing testing and troubleshooting. Many of our customers seem to start at the App Server layer because they claim that nothing is wrong with the applications they build. Most code written by all developers throughout the world has bugs and can be performance tuned, and even if you think you have the best developers in the world, chances are there are opportunities to increase performance and scalability in your application code
  • The Ideal ‘performance funnel’ is one that is not a funnel – it is a pipe. But this is not reality. At every layer of the stack there is a loss of performance and the task of performance tuning is to minimize the impact that each layer has. Applications that are well designed (and built) will typically bottleneck with I/O, and this is really where you want to be. This means that you have minimized the impact of the other layers and gotten to a physical limit, not a logical one.
  • In most cases, this is where you will start. You will start with an ‘average’ application running on an app server that has no tuning whatsoever. In this case, chances are the application will be the bottleneck, this is not an accusation, this is just how the process works. First you have to make it work and then you make it work faster. Only a very, very mature team of developers will produce an application that doesn’t require much performance tuning.
  • There are many steps along the way. Once you clear a bottleneck, you’ll find another one. Sometimes, a bottleneck that you thought you cleared will re-emerge or manifest itself in a different way. It is an iterative process, and while you may allocate 2 weeks for performance testing, there’s no guarantee that you’ll get it all done in that timeframe. As a developer participating in performance testing, this should be your expectation and you shouldn’t lose hope.
  • When you look at Oracle’s overall strategy, you see an attempt to get the best configuration, best performance, out of the box. We want to control the whole stack – not because we want to control the world – but because we want to be able to perform engineering at each level of the stack in order to provide the best performance and user experience possible. Since we own the whole stack, now with SUN hardware, we can engineer systems that would be very difficult, if not impossible to do if we only had influence over the database, application or middleware layer.
  • Enough marketing-speak. Lets talk about the tools that come with WebLogic for troubleshooting.
  • There are many tools available for troubleshooting. You can use Windows performance monitor, Unix TOP and other commands, or packaged software. And as you look at the different layers of the stack you will see that there are many levers to pull and variables to consider. When you are trying to find a problem, you start ruling out factors at each layer of the stack. Sometimes, you can narrow it down, sometimes you can’t. In many cases you find that you have to put in a lot of effort to correlate data from different sources in order to get to a root cause. And then you have to do it over again.
  • What we will be focusing on today is the set of tools that is available with WebLogic in order to troubleshoot and tune. This set includes the Jrockit Flight Recorder and Jrockit Mission Control.
  • For those of you who are not familiar with it, I want to give a brief introduction of the JRockit JVM.
  • JRockit is a JVM that was created by Appeal Virtual Machines which was acquired by BEA in 2002. In 2008 it became part of Oracle Fusion Middleware. It is a proprietary JVM that was written specifically for server-side applications. As such, it is written for machines that have a lot of physical memory and for applications that run for a very long time – days, weeks, even years – such as application servers. If you look at the startup time compared with HotSpot, you will find that in some cases JRockit starts up slower, but JRockit trades startup time and memory for improved performance. JRockit does not support JavaWebStart, but does support Swing applications. Since Desktop Swing applications are short-lived, you may find a better experience using HotSpot for these.
  • Lets get into the weeds a bit with WebLogic
  • The WebLogic Diagnostic Framework (WLDF) is a monitoring and diagnostic framework that defines and implements a set of services that run within WebLogic Server® processes and participate in the standard server life cycle. Using WLDF, you can create, collect, analyze, archive, and access diagnostic data generated by a running server and the applications deployed within its containers. This data provides insight into the run-time performance of servers and applications and enables you to isolate and diagnose faults when they occur. WLDF includes several components for collecting and analyzing data: The Diagnostic Image Capture creates a diagnostic snapshot from the server that can be used for post-failure analysis. Archive is used to persists data events, log records, and metrics from server instances and applications over time There are point-cut Instrumentations that add diagnostic code to WebLogic Server instances and the applications running on them to execute diagnostic actions at specified locations in the code. The Instrumentation component provides the means for associating a diagnostic  context  with requests so they can be tracked as they flow through the system. Mbean Harvester — Captures metrics from run-time MBeans, including WebLogic Server MBeans and custom MBeans, which can be archived and later accessed for viewing historical data. With watches and Notifications we provide the means for monitoring server and application states and sending notifications based on criteria set in the watches. And finally with Logging services you can manage logs for monitoring server, subsystem, and application events. The WebLogic Server logging services are documented separately from the rest of the WebLogic Diagnostic Framework.
  • I have mentioned the ‘Flight Recorder’ a couple times, and you may be wondering what that is. The JRockit flight recorder is much like the flight recorder in an airplane that records all of the events that happen in the system. The goal is to be able to use this data in the case of a problem or a crash in order to ‘go back in time’ and see what was happening leading up to the problem. It is always on, and is a circular buffer that only retains a certain amount of data. Even though it is always on, it does not have a significant amount of overhead. It also has built-in integration with JRockit Mission Control and replaces the JRA recording and LAT capabilities. It not only includes data from the JVM, but also from the WebLogic Diagnostic framework and the Fusion Middleware Dynamic Monitoring System which allows you to get events for ADF, Web Center and SOA Suite applications,.
  • Carges_TBAv1_04.01.05.ppt The flight recorder is intended to be non-intrusive when enabled and to provide diagnostic information in PRODUCTION systems. You can also adjust the level of data that is generated for capture higher, but it will have additional overhead.
  • We have engineered the integration of WebLogic and the Flight Recorder to be able to: View and correlate WebLogic data and events with JVM events and make problem identification easier. We have instrumented WebLogic to generate Flight Recorder events and have added a custom WebLogic event viewed in JRockit Mission Control We also wanted to make it easy to configure and use. It is available out of the box, and it is easy to adjust the volume of events that is generated. We also wanted to be able to leverage the diagnostics in WebLogic server in order to increase the usefulness of the flight recorder so we have added the ability to dump a Flight Recorder dataset as a result of a watch rule being met using the WLDF framework in WebLogic.
  • The WLDF FlightRecorder events are automatically throttled when the system is under stress. The algorithm that implements this throttling uses two different measurements, and there are &amp;quot;caps&amp;quot; for each of these criteria, currently at 128 req/sec and 800 events/sec, respectively. When these caps are exceeded, requests are sampled at a rate intended to keep things within those thresholds. So basically, when your system is under load, the FR adjusts in order to reduce the overhead.
  • It is very easy to configure the WLDF event volume, and I’ll show you how we can do that in our demo. There are 4 levels – Off, Low, Medium and High. We consulted with our developer user groups and support organization to come up with what we feel is a good segmentation of the events. In many cases, customers will enable the WLDF at HIGH in development and LOW or MEDIUM in production and they are comfortable with the small amount of overhead.
  • Here are some of the events that are generated at a ‘low’ volume. Here you will see that events for EJB invocation JDBC statements Servlet Invocation Web Application Load and Unload Web Services JAXRPC request/response JAXWS request/response In many cases this information is sufficient to identify the area(s) for further investigation.
  • As you go up to medium, you will see events for EJB home create/remove More detailed JDBC connection information And more detailed Servlet information
  • With HIGH, you will see even more events such as JDBC transaction information JTA transactions More detailed Servlet information And stack traces and user IDs
  • We have also integrated JFR with WLDF such that you can capture a FR recording as the result of a watch alert using the WLDF. We also provide integration with the Remote Diagnostics Agent such that our support organization can proactively capture a JFR image.
  • Currently unsupported, but gives you a preview of whats to come. Filtering of WebLogic specific events in the JRMC. We’ll cover this in the demo, so I won’t spend too much time on it now.
  • Basically, it gives you a WebLogic tab with WebLogic events organized in a user-friendly way. We’ll see this in the demo.
  • Lets cover
  • WebLogic Developer Webcast 5: Troubleshooting and Testing with WebLogic, SoapUI, JRockit Flight Recorder and JRockit Mission Control

    1. 2. Troubleshooting and Tuning WebLogic Jeffrey West Sr. Principal Product Manager
    2. 3. Program Agenda <ul><li>General Troubleshooting Tools & Tips </li></ul><ul><li>JRockit JVM </li></ul><ul><li>JRockit Mission Control </li></ul><ul><li>JRockit Flight Recorder </li></ul><ul><li>SOAP UI & JRMC/JRF Demo </li></ul><Insert Picture Here>
    3. 4. Troubleshooting & Tuning Areas of Focus Where to start? <ul><li>Application </li></ul><ul><li>Application Server / Middleware </li></ul><ul><li>Database </li></ul><ul><li>Operating System </li></ul><ul><li>Machine </li></ul><ul><li>Storage </li></ul><ul><li>Network </li></ul>My application is fine! (probably not) I have the best developers! (really? in the whole world?) Well, they’re really good! (how much are you paying them?) Some people start here
    4. 5. The Near-Ideal ‘Performance Funnel’ <ul><li>In a well-designed distributed application, I/O becomes the bottleneck </li></ul><ul><li>You really want each layer of the stack to have the smallest possible impact on the performance of your application </li></ul>Machine Database Middleware Application Network Storage Operating System
    5. 6. The ‘Performance Funnel’: Where you start <ul><li>It’s OK - This is where you are supposed to start </li></ul><ul><li>First, make your application work RIGHT </li></ul><ul><li>Then, make it work FASTER </li></ul><ul><li>Only a very mature team of developers will produce code that doesn’t need much performance tuning </li></ul>Machine Database Middleware Application Network Storage Operating System
    6. 7. Steps along the way… <ul><li>As you open the flood gates, performance tuning is required down the stack </li></ul><ul><li>Each iteration will likely yield a new bottleneck </li></ul>Machine Database Middleware Application Network Storage Operating System
    7. 8. Oracle Overall Strategy <ul><li>Deliver a complete , open , integrated stack of hardware, infrastructure, database, middleware, and business applications </li></ul><ul><li>Exploit processor, systems, storage, and networking trends to deliver breakthrough innovations by combining Oracle software with Sun hardware </li></ul><ul><li>Integrate components of Oracle’s software stack to provide unique value to customers </li></ul>©2011 Oracle Corporation
    8. 9. <Insert Picture Here> Troubleshooting Tools
    9. 10. Where there’s smoke… <ul><li>Application </li></ul><ul><ul><li>Java Profilers </li></ul></ul><ul><ul><li>Thread Dumps </li></ul></ul><ul><ul><li>Custom performance logging </li></ul></ul><ul><li>Application Server </li></ul><ul><ul><li># of JVM’s </li></ul></ul><ul><ul><li>Heap / GC </li></ul></ul><ul><ul><li>Threads </li></ul></ul><ul><ul><li>JMS </li></ul></ul><ul><ul><li>JDBC </li></ul></ul><ul><ul><li>EJB’s </li></ul></ul><ul><li>Operating System </li></ul><ul><ul><li>Swap Space </li></ul></ul><ul><ul><li>Open Files </li></ul></ul><ul><li>Database </li></ul><ul><ul><li>Sessions & Connections </li></ul></ul><ul><ul><li>Tablespace Separation (LOB, IDX, ROWDATA) </li></ul></ul><ul><li>Machine </li></ul><ul><ul><li>RAM </li></ul></ul><ul><ul><li>Cores </li></ul></ul><ul><ul><li>CPUs </li></ul></ul><ul><li>Storage </li></ul><ul><ul><li>RAID Levels & Performance </li></ul></ul><ul><ul><ul><li>RAID-0 – best all-around performance, no redundancy </li></ul></ul></ul><ul><ul><ul><li>RAID-1 – Full rendundancy, single-disk performance </li></ul></ul></ul><ul><ul><ul><li>RAID-3 – best for large datasets (sequential read/write) </li></ul></ul></ul><ul><ul><ul><li>RAID-5 – Most common, versatility and redundancy </li></ul></ul></ul><ul><ul><ul><li>RAID0+1 – RAID-0 performance with RAID-1 redundancy </li></ul></ul></ul><ul><li>Network </li></ul><ul><ul><li>Speed/Duplex of: </li></ul></ul><ul><ul><ul><li>Machine </li></ul></ul></ul><ul><ul><ul><li>Load Balancer </li></ul></ul></ul><ul><ul><ul><li>Switch </li></ul></ul></ul>
    10. 11. Our Focus Today <ul><li>Application </li></ul><ul><ul><li>Java Profilers </li></ul></ul><ul><ul><li>Thread Dumps </li></ul></ul><ul><ul><li>Custom performance logging </li></ul></ul><ul><li>Application Server </li></ul><ul><ul><li># of JVM’s </li></ul></ul><ul><ul><li>Heap / GC </li></ul></ul><ul><ul><li>Threads </li></ul></ul><ul><ul><li>JMS </li></ul></ul><ul><ul><li>JDBC </li></ul></ul><ul><ul><li>EJB’s </li></ul></ul><ul><li>Operating System </li></ul><ul><ul><li>Swap Space </li></ul></ul><ul><ul><li>Open Files </li></ul></ul><ul><li>Database </li></ul><ul><ul><li>Long Running SQL </li></ul></ul><ul><ul><li>Sessions & Connections </li></ul></ul><ul><ul><li>Tablespace Separation (LOB, IDX, ROWDATA) </li></ul></ul><ul><li>Machine </li></ul><ul><ul><li>RAM </li></ul></ul><ul><ul><li>SWAP </li></ul></ul><ul><ul><li>Cores </li></ul></ul><ul><ul><li>CPUs </li></ul></ul><ul><li>Storage </li></ul><ul><ul><li>RAID Levels & Performance </li></ul></ul><ul><ul><ul><li>RAID-0 – best all-around performance, no redundancy </li></ul></ul></ul><ul><ul><ul><li>RAID-1 – Full rendundancy, single-disk performance </li></ul></ul></ul><ul><ul><ul><li>RAID-3 – best for large datasets (sequential read/write) </li></ul></ul></ul><ul><ul><ul><li>RAID-5 – Most common, versatility and redundancy </li></ul></ul></ul><ul><ul><ul><li>RAID0+1 – RAID-0 performance with RAID-1 redundancy </li></ul></ul></ul><ul><li>Network </li></ul><ul><ul><li>Speed/Duplex of: </li></ul></ul><ul><ul><ul><li>Machine </li></ul></ul></ul><ul><ul><ul><li>Load Balancer </li></ul></ul></ul><ul><ul><ul><li>Switch </li></ul></ul></ul>} JRockit Flight Recorder JRockit Mission Control
    11. 12. <Insert Picture Here> JRockit JVM
    12. 13. The JRockit JVM <ul><li>Written for server-side applications </li></ul><ul><ul><li>Machines with a lot of physical memory </li></ul></ul><ul><ul><li>Applications run for a long time </li></ul></ul><ul><li>Trades Memory and start-up time for performance </li></ul><ul><ul><li>May load slower than HotSpot, but performs better in the long run </li></ul></ul><ul><li>No support for Java WebStart </li></ul><ul><li>JRockit supports (Java SE) UI applications </li></ul><ul><ul><li>Due to the short-lived nature of UI applications HotSpot might be a better choice </li></ul></ul>
    13. 14. <Insert Picture Here> WebLogic Monitoring & Troubleshooting
    14. 15. Overview of the WebLogic Diagnostics Framework Retrieve Expose Process Capture <ul><li>WLST API </li></ul><ul><li>JMX API </li></ul><ul><li>WLDF Dashboard </li></ul><ul><li>SNMP Trap </li></ul><ul><li>JMS Message </li></ul><ul><li>JMX Notification </li></ul><ul><li>SMTP Email </li></ul><ul><li>Diagnostic Image </li></ul>Pull Push Log File Event Collector MBean Property Harvester Code Instrumentor Archiver Watcher JRockit Flight Recorder Buffer Accessor Notifier JRockit Mission Control
    15. 16. What is the JRockit Flight Recorder? <ul><li>New in JRockit R28 </li></ul><ul><li>“ Circular buffer” in JRockit JVM that stores diagnostic data </li></ul><ul><ul><li>Always on </li></ul></ul><ul><ul><li>New data comes in and is stored, old data dropped off </li></ul></ul><ul><li>Built-in integration with JRMC </li></ul><ul><ul><li>Replaces JRMC Runtime Analyzer and Latency Analyzer </li></ul></ul><ul><li>Very low/near zero overhead </li></ul><ul><ul><li>Uses data already used by JVM </li></ul></ul><ul><li>Data can include events from the JVM and from any other event producer </li></ul><ul><ul><li>WebLogic Server (WLDF) </li></ul></ul><ul><ul><li>Fusion Middleware (DMS) </li></ul></ul>New Data Old Data Time
    16. 17. <ul><li>What it is designed for? </li></ul><ul><ul><li>Provide diagnostic information in running production systems </li></ul></ul><ul><ul><li>Look back in time to see what happened after a crash </li></ul></ul><ul><ul><li>Capture most recent activity to enable analysis leading up to an issue </li></ul></ul><ul><ul><li>Capture data from all levels JVM, WLS, DMS, etc… </li></ul></ul><ul><ul><li>Offline/offsite analysis can be done using the JRMC GUI </li></ul></ul><ul><ul><li>JRockit dumps capture information to assist in crash-analysis </li></ul></ul><ul><li>What it is not designed for? </li></ul><ul><ul><li>Large event payloads or very high volumes of events </li></ul></ul><ul><ul><li>Long history </li></ul></ul><ul><ul><li>Not a replacement for Debug logging or the server logging </li></ul></ul>JRockit Flight Recorder Use Cases
    17. 18. WebLogic/Flight Recorder Integration Goals and Points Integration Goals Integration Points 1 2 3 <ul><li>View and correlate WebLogic and JVM events </li></ul><ul><li>Make problem identification easy </li></ul><ul><li>Enable easy configuration </li></ul><ul><li>Leverage diagnostics in WebLogic Server </li></ul><ul><li>Enhance usefulness of Flight Recorder </li></ul><ul><li>WebLogic JFR events </li></ul><ul><li>Custom WebLogic event viewer in JRMC </li></ul><ul><li>Available OOTB </li></ul><ul><li>On/off/volume control </li></ul><ul><li>Automatic JFR recording control </li></ul><ul><li>WLDF Watch Rules and Notifications </li></ul><ul><li>Remote Diagnostics Agent </li></ul>
    18. 19. WebLogic Diagnostic Events for Flight Recorder Integration <ul><li>Point-cuts woven into WebLogic to generate diagnostic events </li></ul><ul><ul><li>Events are propagated to the Flight Recorder </li></ul></ul><ul><li>Logical event generators: low, medium, and high </li></ul><ul><li>Very low performance impact </li></ul><ul><ul><li>Low – 0.81 decrease in throughput </li></ul></ul><ul><ul><li>High – 3.83 decrease in throughput </li></ul></ul><ul><li>Events are throttled to avoid high overhead </li></ul><ul><ul><li>Request is tagged for event generation </li></ul></ul><ul><ul><li>All events for a specific context ID are captured regardless of throttling </li></ul></ul><ul><ul><li>Target number of reqs/events per second, checked every 2 seconds, throttling readjusted </li></ul></ul>1
    19. 20. Easy Configuration with WLDF Volume <ul><li>Simple configuration: enabled and configured using a volume control per server: Off (default), Low, Medium, High </li></ul><ul><li>Each level has a predetermined data set </li></ul><ul><ul><li>Determined through consultation with dev groups and Support </li></ul></ul><ul><ul><li>Data set was adjusted based on performance testing </li></ul></ul><ul><ul><li>High volume includes user info and stack traces, which are expensive data gathering operations </li></ul></ul><ul><li>Volume also controls “WLDF Recording” in JFR </li></ul><ul><ul><li>WLDF has it’s own recording independent of other “recordings” in JFR </li></ul></ul><ul><ul><li>At startup (WLDF volume = Off), WLDF recording created, but all events turned off </li></ul></ul><ul><ul><li>When WLDF volume set to Low or higher, WLDF events and default JVM events are generated </li></ul></ul><ul><ul><li>DMS leverages the same JFR recording </li></ul></ul>2
    20. 21. WLDF Low Volume Events <ul><li>WLDF Logging Snapshot </li></ul><ul><li>WLDF LogRecord Snapshot </li></ul><ul><li>WLDF WLLogRecord Snapshot </li></ul><ul><li>Connector Activate Endpoint </li></ul><ul><li>Connector Deactivate Endpoint </li></ul><ul><li>Connector Inbound Transaction Rollback </li></ul><ul><li>Connector Outbound Connection Error </li></ul><ul><li>Connector Outbound Destroy Connection </li></ul><ul><li>Connector Outbound Register Resource </li></ul><ul><li>Connector Outbound Release Connection </li></ul><ul><li>Connector Outbound Reserve Connection </li></ul><ul><li>Connector Outbound Transaction Rollback </li></ul><ul><li>Connector Outbound Unregister Resource </li></ul><ul><li>EJB Business Method Post Invoke </li></ul><ul><li>EJB Business Method Pre Invoke </li></ul><ul><li>EJB Pool Manager Post Invoke </li></ul><ul><li>EJB Pool Manager Pre Invoke </li></ul><ul><li>JDBC Connection Rollback </li></ul><ul><li>JDBC Statement Execute </li></ul><ul><li>Servlet Invocation </li></ul><ul><li>Web Application Load </li></ul><ul><li>Web Application Unload </li></ul><ul><li>Webservices JAXRPC Client Request </li></ul><ul><li>Webservices JAXRPC Client Response </li></ul><ul><li>Webservices JAXRPC Dispatch </li></ul><ul><li>Webservices JAXRPC Request </li></ul><ul><li>Webservices JAXRPC Response </li></ul><ul><li>Webservices JAXWS Endpoint </li></ul><ul><li>Webservices JAXWS Request </li></ul><ul><li>Webservices JAXWS Resource </li></ul>© 2010 Oracle Corporation Proprietary and Confidential
    21. 22. WLDF Medium Volume Events <ul><li>EJB Home Create </li></ul><ul><li>EJB Home Remove </li></ul><ul><li>EJB PoolManager Create </li></ul><ul><li>JDBC Connection Close </li></ul><ul><li>JDBC Connection Commit </li></ul><ul><li>JDBC Connection Create Statement </li></ul><ul><li>JDBC Connection Prepare </li></ul><ul><li>JDBC Connection Release </li></ul><ul><li>JDBC Connection Reserve </li></ul><ul><li>JDBC Data Source Get Connection </li></ul><ul><li>JDBC Driver Connect </li></ul><ul><li>JDBC Statement Creation </li></ul><ul><li>Servlet Execute </li></ul><ul><li>Servlet Request Run </li></ul><ul><li>Servlet Request Dispatch </li></ul><ul><li>Servlet Request </li></ul><ul><li>Servlet Filter </li></ul><ul><li>Servlet Async Action </li></ul><ul><li>Servlet Context Execute </li></ul><ul><li>Servlet Response Write Headers </li></ul><ul><li>Servlet Response Send </li></ul><ul><li>Servlet Stale Resource </li></ul><ul><li>Servlet Check Access </li></ul><ul><li>JMS BE Consumer Log </li></ul>© 2010 Oracle Corporation Proprietary and Confidential
    22. 23. WLDF High Volume Events <ul><li>EJB Database Access </li></ul><ul><li>EJB Business Method Post Invoke Cleanup </li></ul><ul><li>EJB Pool Manager Remove </li></ul><ul><li>EJB Replicated Session Manager </li></ul><ul><li>EJB Timer Manager </li></ul><ul><li>JDBC Transaction Commit </li></ul><ul><li>JDBC Transaction End </li></ul><ul><li>JDBC Transaction Get XA Resource </li></ul><ul><li>JDBC Transaction Is Same RM </li></ul><ul><li>JDBC Transaction Prepare </li></ul><ul><li>JDBC Transaction Rollback </li></ul><ul><li>JDBC Transaction Start </li></ul><ul><li>JTA Transaction Commit </li></ul><ul><li>JTA Transaction End </li></ul><ul><li>JTA Transaction Prepared </li></ul><ul><li>JTA Transaction Prepare </li></ul><ul><li>JTA Transaction Start </li></ul><ul><li>Servlet Request Overload </li></ul><ul><li>Servlet Request Cancel </li></ul><ul><li>Servlet Context Handle Throwable </li></ul><ul><li>Plus stack traces and User IDs </li></ul>© 2010 Oracle Corporation Proprietary and Confidential
    23. 24. Automated Recording Capture <ul><li>Integrated with the WebLogic Diagnostic Image and Watch and Notification system </li></ul><ul><ul><li>Enables capture based on system state, event – capture during event; no need to replicate </li></ul></ul><ul><ul><li>Watch for stuck thread count, heap size increase, available memory, etc. </li></ul></ul><ul><ul><li>Set up notifications: capture WLDF image </li></ul></ul><ul><ul><li>Diagnostic image capture spurs JFR file generation; JFR file included in diagnostic image </li></ul></ul><ul><ul><li>Includes full JFR data from all event generators </li></ul></ul><ul><li>Integration with Oracle Support’s Remote Diagnostics Agent </li></ul><ul><ul><li>Generates WL Diagnostic Image </li></ul></ul>3 WLDF Trigger JFR Buffer Write Capture JFR in Diag Image
    24. 25. WebLogic Experimental Plug-in for Mission Control 3
    25. 26. WebLogic Experimental Plugin
    26. 27. <Insert Picture Here> WebLogic Tuning
    27. 28. Areas of Focus <ul><li>JVM’s and Machines </li></ul><ul><li>Threads / Work Managers </li></ul><ul><li>JMS </li></ul><ul><li>JDBC </li></ul><ul><li>EJB </li></ul>
    28. 29. The timeless battle with Garbage Collection <ul><li>How do you slice & dice your CPU and memory? </li></ul><ul><li>More JVM’s with smaller heap… or fewer JVM’s with bigger heap? </li></ul><ul><li>Bigger Heap = longer pause times for GC </li></ul><ul><li>Ultimately depends on your application – can you tolerate longer GC times? </li></ul>
    29. 30. JVM’s and Machines <ul><li>Machines: </li></ul><ul><ul><li>8GB RAM </li></ul></ul><ul><ul><li>Dual-CPU, Quad-Core </li></ul></ul><ul><li>Why run only one JVM per machine? </li></ul><ul><li>Run multiple JVMs! </li></ul>512MB: Admin 2GB: OS, etc 2GB: MS-1 2GB: MS-3 2GB: MS-2 2GB: OS, etc 2GB: MS-4 2GB: MS-5
    30. 31. Threads / Work Managers <ul><li>Define a work manager for each of your applications </li></ul><ul><li>WebLogic will self-tune the work managers for the best performance </li></ul><ul><li>You can define: </li></ul><ul><ul><li>Context Classes for prioritizing requests </li></ul></ul><ul><ul><li>Minimum Thread Constraint </li></ul></ul><ul><ul><li>Maximum Threads Constraint </li></ul></ul><ul><li>If you have an application that makes a WS call back to an application on WLS, the applications should have different Work Managers (ESB) </li></ul>
    31. 32. JMS <ul><li>Connection Factories </li></ul><ul><ul><li>Enable Load Balancing </li></ul></ul><ul><ul><li>Disable Server Affinity </li></ul></ul><ul><li>Persistent Storage </li></ul><ul><ul><li>File Stores offer the best performance </li></ul></ul><ul><ul><li>JDBC stores offer ease of failover </li></ul></ul><ul><li>Distributed Destinations </li></ul><ul><ul><li>Spread the work across the cluster </li></ul></ul><ul><ul><li>Partitioned Distributed Topics eases event architectures </li></ul></ul>
    32. 33. JDBC <ul><li>Use JNDI Datasources! </li></ul><ul><ul><li>Don’t use application-level connection pools – you can’t monitor them holistically </li></ul></ul><ul><li>Number of Threads / Connection Pool Size </li></ul><ul><ul><li>Max Threads Constraint associated with connection pool </li></ul></ul><ul><ul><li>Monitor & Minimize waiters </li></ul></ul><ul><li>Max Size = Initial Size, disable Shrinkage </li></ul><ul><ul><li>Less latency for handling peak loads </li></ul></ul><ul><li>Prepared Statement Cache </li></ul><ul><ul><li>Tune it so you have a high (+90%) hit rate </li></ul></ul><ul><ul><li>Write your applications to use prepared statements </li></ul></ul>
    33. 34. Common Application ‘mistakes’ <ul><li>Read a properties from a file on each request </li></ul><ul><ul><li>Poor performance – File I/O, doesn’t scale </li></ul></ul><ul><ul><li>Try storing your properties in a database and/or caching them in the JVM </li></ul></ul><ul><li>New JAXB context for each parse event </li></ul><ul><ul><li>Create one singleton context and reuse </li></ul></ul><ul><li>Too many session variables </li></ul><ul><ul><li>Clear a variable when you don’t need it anymore </li></ul></ul>
    34. 36. Base Content Slide <ul><li>First-level bullet </li></ul><ul><ul><li>Second-level bullet </li></ul></ul><ul><ul><ul><li>Third-level bullet </li></ul></ul></ul><ul><ul><ul><ul><li>Fourth-level bullet </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Fifth-level bullet </li></ul></ul></ul></ul></ul>
    35. 37. Example of Title Extending to Two Lines <ul><li>First-level bullet </li></ul><ul><ul><li>Second-level bullet </li></ul></ul><ul><ul><ul><li>Third-level bullet </li></ul></ul></ul><ul><ul><ul><ul><li>Fourth-level bullet </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Fifth-level bullet </li></ul></ul></ul></ul></ul>
    36. 38. Alternate Title with Subhead Arial Size 18 (Gray) <ul><li>First-level bullet </li></ul><ul><ul><li>Second-level bullet </li></ul></ul><ul><ul><ul><li>Third-level bullet </li></ul></ul></ul><ul><ul><ul><ul><li>Fourth-level bullet </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Fifth-level bullet </li></ul></ul></ul></ul></ul>

    ×