Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

WebLogic Developer Webcast 5: Troubleshooting and Testing with WebLogic, SoapUI, JRockit Flight Recorder and JRockit Mission Control

4,999 views

Published on

WebLogic Developer Webcast 5: Troubleshooting and Testing with WebLogic, SoapUI, JRockit Flight Recorder and JRockit Mission Control

  1. 2. Troubleshooting and Tuning WebLogic Jeffrey West Sr. Principal Product Manager
  2. 3. Program Agenda <ul><li>General Troubleshooting Tools & Tips </li></ul><ul><li>JRockit JVM </li></ul><ul><li>JRockit Mission Control </li></ul><ul><li>JRockit Flight Recorder </li></ul><ul><li>SOAP UI & JRMC/JRF Demo </li></ul><Insert Picture Here>
  3. 4. Troubleshooting & Tuning Areas of Focus Where to start? <ul><li>Application </li></ul><ul><li>Application Server / Middleware </li></ul><ul><li>Database </li></ul><ul><li>Operating System </li></ul><ul><li>Machine </li></ul><ul><li>Storage </li></ul><ul><li>Network </li></ul>My application is fine! (probably not) I have the best developers! (really? in the whole world?) Well, they’re really good! (how much are you paying them?) Some people start here
  4. 5. The Near-Ideal ‘Performance Funnel’ <ul><li>In a well-designed distributed application, I/O becomes the bottleneck </li></ul><ul><li>You really want each layer of the stack to have the smallest possible impact on the performance of your application </li></ul>Machine Database Middleware Application Network Storage Operating System
  5. 6. The ‘Performance Funnel’: Where you start <ul><li>It’s OK - This is where you are supposed to start </li></ul><ul><li>First, make your application work RIGHT </li></ul><ul><li>Then, make it work FASTER </li></ul><ul><li>Only a very mature team of developers will produce code that doesn’t need much performance tuning </li></ul>Machine Database Middleware Application Network Storage Operating System
  6. 7. Steps along the way… <ul><li>As you open the flood gates, performance tuning is required down the stack </li></ul><ul><li>Each iteration will likely yield a new bottleneck </li></ul>Machine Database Middleware Application Network Storage Operating System
  7. 8. Oracle Overall Strategy <ul><li>Deliver a complete , open , integrated stack of hardware, infrastructure, database, middleware, and business applications </li></ul><ul><li>Exploit processor, systems, storage, and networking trends to deliver breakthrough innovations by combining Oracle software with Sun hardware </li></ul><ul><li>Integrate components of Oracle’s software stack to provide unique value to customers </li></ul>©2011 Oracle Corporation
  8. 9. <Insert Picture Here> Troubleshooting Tools
  9. 10. Where there’s smoke… <ul><li>Application </li></ul><ul><ul><li>Java Profilers </li></ul></ul><ul><ul><li>Thread Dumps </li></ul></ul><ul><ul><li>Custom performance logging </li></ul></ul><ul><li>Application Server </li></ul><ul><ul><li># of JVM’s </li></ul></ul><ul><ul><li>Heap / GC </li></ul></ul><ul><ul><li>Threads </li></ul></ul><ul><ul><li>JMS </li></ul></ul><ul><ul><li>JDBC </li></ul></ul><ul><ul><li>EJB’s </li></ul></ul><ul><li>Operating System </li></ul><ul><ul><li>Swap Space </li></ul></ul><ul><ul><li>Open Files </li></ul></ul><ul><li>Database </li></ul><ul><ul><li>Sessions & Connections </li></ul></ul><ul><ul><li>Tablespace Separation (LOB, IDX, ROWDATA) </li></ul></ul><ul><li>Machine </li></ul><ul><ul><li>RAM </li></ul></ul><ul><ul><li>Cores </li></ul></ul><ul><ul><li>CPUs </li></ul></ul><ul><li>Storage </li></ul><ul><ul><li>RAID Levels & Performance </li></ul></ul><ul><ul><ul><li>RAID-0 – best all-around performance, no redundancy </li></ul></ul></ul><ul><ul><ul><li>RAID-1 – Full rendundancy, single-disk performance </li></ul></ul></ul><ul><ul><ul><li>RAID-3 – best for large datasets (sequential read/write) </li></ul></ul></ul><ul><ul><ul><li>RAID-5 – Most common, versatility and redundancy </li></ul></ul></ul><ul><ul><ul><li>RAID0+1 – RAID-0 performance with RAID-1 redundancy </li></ul></ul></ul><ul><li>Network </li></ul><ul><ul><li>Speed/Duplex of: </li></ul></ul><ul><ul><ul><li>Machine </li></ul></ul></ul><ul><ul><ul><li>Load Balancer </li></ul></ul></ul><ul><ul><ul><li>Switch </li></ul></ul></ul>
  10. 11. Our Focus Today <ul><li>Application </li></ul><ul><ul><li>Java Profilers </li></ul></ul><ul><ul><li>Thread Dumps </li></ul></ul><ul><ul><li>Custom performance logging </li></ul></ul><ul><li>Application Server </li></ul><ul><ul><li># of JVM’s </li></ul></ul><ul><ul><li>Heap / GC </li></ul></ul><ul><ul><li>Threads </li></ul></ul><ul><ul><li>JMS </li></ul></ul><ul><ul><li>JDBC </li></ul></ul><ul><ul><li>EJB’s </li></ul></ul><ul><li>Operating System </li></ul><ul><ul><li>Swap Space </li></ul></ul><ul><ul><li>Open Files </li></ul></ul><ul><li>Database </li></ul><ul><ul><li>Long Running SQL </li></ul></ul><ul><ul><li>Sessions & Connections </li></ul></ul><ul><ul><li>Tablespace Separation (LOB, IDX, ROWDATA) </li></ul></ul><ul><li>Machine </li></ul><ul><ul><li>RAM </li></ul></ul><ul><ul><li>SWAP </li></ul></ul><ul><ul><li>Cores </li></ul></ul><ul><ul><li>CPUs </li></ul></ul><ul><li>Storage </li></ul><ul><ul><li>RAID Levels & Performance </li></ul></ul><ul><ul><ul><li>RAID-0 – best all-around performance, no redundancy </li></ul></ul></ul><ul><ul><ul><li>RAID-1 – Full rendundancy, single-disk performance </li></ul></ul></ul><ul><ul><ul><li>RAID-3 – best for large datasets (sequential read/write) </li></ul></ul></ul><ul><ul><ul><li>RAID-5 – Most common, versatility and redundancy </li></ul></ul></ul><ul><ul><ul><li>RAID0+1 – RAID-0 performance with RAID-1 redundancy </li></ul></ul></ul><ul><li>Network </li></ul><ul><ul><li>Speed/Duplex of: </li></ul></ul><ul><ul><ul><li>Machine </li></ul></ul></ul><ul><ul><ul><li>Load Balancer </li></ul></ul></ul><ul><ul><ul><li>Switch </li></ul></ul></ul>} JRockit Flight Recorder JRockit Mission Control
  11. 12. <Insert Picture Here> JRockit JVM
  12. 13. The JRockit JVM <ul><li>Written for server-side applications </li></ul><ul><ul><li>Machines with a lot of physical memory </li></ul></ul><ul><ul><li>Applications run for a long time </li></ul></ul><ul><li>Trades Memory and start-up time for performance </li></ul><ul><ul><li>May load slower than HotSpot, but performs better in the long run </li></ul></ul><ul><li>No support for Java WebStart </li></ul><ul><li>JRockit supports (Java SE) UI applications </li></ul><ul><ul><li>Due to the short-lived nature of UI applications HotSpot might be a better choice </li></ul></ul>
  13. 14. <Insert Picture Here> WebLogic Monitoring & Troubleshooting
  14. 15. Overview of the WebLogic Diagnostics Framework Retrieve Expose Process Capture <ul><li>WLST API </li></ul><ul><li>JMX API </li></ul><ul><li>WLDF Dashboard </li></ul><ul><li>SNMP Trap </li></ul><ul><li>JMS Message </li></ul><ul><li>JMX Notification </li></ul><ul><li>SMTP Email </li></ul><ul><li>Diagnostic Image </li></ul>Pull Push Log File Event Collector MBean Property Harvester Code Instrumentor Archiver Watcher JRockit Flight Recorder Buffer Accessor Notifier JRockit Mission Control
  15. 16. What is the JRockit Flight Recorder? <ul><li>New in JRockit R28 </li></ul><ul><li>“ Circular buffer” in JRockit JVM that stores diagnostic data </li></ul><ul><ul><li>Always on </li></ul></ul><ul><ul><li>New data comes in and is stored, old data dropped off </li></ul></ul><ul><li>Built-in integration with JRMC </li></ul><ul><ul><li>Replaces JRMC Runtime Analyzer and Latency Analyzer </li></ul></ul><ul><li>Very low/near zero overhead </li></ul><ul><ul><li>Uses data already used by JVM </li></ul></ul><ul><li>Data can include events from the JVM and from any other event producer </li></ul><ul><ul><li>WebLogic Server (WLDF) </li></ul></ul><ul><ul><li>Fusion Middleware (DMS) </li></ul></ul>New Data Old Data Time
  16. 17. <ul><li>What it is designed for? </li></ul><ul><ul><li>Provide diagnostic information in running production systems </li></ul></ul><ul><ul><li>Look back in time to see what happened after a crash </li></ul></ul><ul><ul><li>Capture most recent activity to enable analysis leading up to an issue </li></ul></ul><ul><ul><li>Capture data from all levels JVM, WLS, DMS, etc… </li></ul></ul><ul><ul><li>Offline/offsite analysis can be done using the JRMC GUI </li></ul></ul><ul><ul><li>JRockit dumps capture information to assist in crash-analysis </li></ul></ul><ul><li>What it is not designed for? </li></ul><ul><ul><li>Large event payloads or very high volumes of events </li></ul></ul><ul><ul><li>Long history </li></ul></ul><ul><ul><li>Not a replacement for Debug logging or the server logging </li></ul></ul>JRockit Flight Recorder Use Cases
  17. 18. WebLogic/Flight Recorder Integration Goals and Points Integration Goals Integration Points 1 2 3 <ul><li>View and correlate WebLogic and JVM events </li></ul><ul><li>Make problem identification easy </li></ul><ul><li>Enable easy configuration </li></ul><ul><li>Leverage diagnostics in WebLogic Server </li></ul><ul><li>Enhance usefulness of Flight Recorder </li></ul><ul><li>WebLogic JFR events </li></ul><ul><li>Custom WebLogic event viewer in JRMC </li></ul><ul><li>Available OOTB </li></ul><ul><li>On/off/volume control </li></ul><ul><li>Automatic JFR recording control </li></ul><ul><li>WLDF Watch Rules and Notifications </li></ul><ul><li>Remote Diagnostics Agent </li></ul>
  18. 19. WebLogic Diagnostic Events for Flight Recorder Integration <ul><li>Point-cuts woven into WebLogic to generate diagnostic events </li></ul><ul><ul><li>Events are propagated to the Flight Recorder </li></ul></ul><ul><li>Logical event generators: low, medium, and high </li></ul><ul><li>Very low performance impact </li></ul><ul><ul><li>Low – 0.81 decrease in throughput </li></ul></ul><ul><ul><li>High – 3.83 decrease in throughput </li></ul></ul><ul><li>Events are throttled to avoid high overhead </li></ul><ul><ul><li>Request is tagged for event generation </li></ul></ul><ul><ul><li>All events for a specific context ID are captured regardless of throttling </li></ul></ul><ul><ul><li>Target number of reqs/events per second, checked every 2 seconds, throttling readjusted </li></ul></ul>1
  19. 20. Easy Configuration with WLDF Volume <ul><li>Simple configuration: enabled and configured using a volume control per server: Off (default), Low, Medium, High </li></ul><ul><li>Each level has a predetermined data set </li></ul><ul><ul><li>Determined through consultation with dev groups and Support </li></ul></ul><ul><ul><li>Data set was adjusted based on performance testing </li></ul></ul><ul><ul><li>High volume includes user info and stack traces, which are expensive data gathering operations </li></ul></ul><ul><li>Volume also controls “WLDF Recording” in JFR </li></ul><ul><ul><li>WLDF has it’s own recording independent of other “recordings” in JFR </li></ul></ul><ul><ul><li>At startup (WLDF volume = Off), WLDF recording created, but all events turned off </li></ul></ul><ul><ul><li>When WLDF volume set to Low or higher, WLDF events and default JVM events are generated </li></ul></ul><ul><ul><li>DMS leverages the same JFR recording </li></ul></ul>2
  20. 21. WLDF Low Volume Events <ul><li>WLDF Logging Snapshot </li></ul><ul><li>WLDF LogRecord Snapshot </li></ul><ul><li>WLDF WLLogRecord Snapshot </li></ul><ul><li>Connector Activate Endpoint </li></ul><ul><li>Connector Deactivate Endpoint </li></ul><ul><li>Connector Inbound Transaction Rollback </li></ul><ul><li>Connector Outbound Connection Error </li></ul><ul><li>Connector Outbound Destroy Connection </li></ul><ul><li>Connector Outbound Register Resource </li></ul><ul><li>Connector Outbound Release Connection </li></ul><ul><li>Connector Outbound Reserve Connection </li></ul><ul><li>Connector Outbound Transaction Rollback </li></ul><ul><li>Connector Outbound Unregister Resource </li></ul><ul><li>EJB Business Method Post Invoke </li></ul><ul><li>EJB Business Method Pre Invoke </li></ul><ul><li>EJB Pool Manager Post Invoke </li></ul><ul><li>EJB Pool Manager Pre Invoke </li></ul><ul><li>JDBC Connection Rollback </li></ul><ul><li>JDBC Statement Execute </li></ul><ul><li>Servlet Invocation </li></ul><ul><li>Web Application Load </li></ul><ul><li>Web Application Unload </li></ul><ul><li>Webservices JAXRPC Client Request </li></ul><ul><li>Webservices JAXRPC Client Response </li></ul><ul><li>Webservices JAXRPC Dispatch </li></ul><ul><li>Webservices JAXRPC Request </li></ul><ul><li>Webservices JAXRPC Response </li></ul><ul><li>Webservices JAXWS Endpoint </li></ul><ul><li>Webservices JAXWS Request </li></ul><ul><li>Webservices JAXWS Resource </li></ul>© 2010 Oracle Corporation Proprietary and Confidential
  21. 22. WLDF Medium Volume Events <ul><li>EJB Home Create </li></ul><ul><li>EJB Home Remove </li></ul><ul><li>EJB PoolManager Create </li></ul><ul><li>JDBC Connection Close </li></ul><ul><li>JDBC Connection Commit </li></ul><ul><li>JDBC Connection Create Statement </li></ul><ul><li>JDBC Connection Prepare </li></ul><ul><li>JDBC Connection Release </li></ul><ul><li>JDBC Connection Reserve </li></ul><ul><li>JDBC Data Source Get Connection </li></ul><ul><li>JDBC Driver Connect </li></ul><ul><li>JDBC Statement Creation </li></ul><ul><li>Servlet Execute </li></ul><ul><li>Servlet Request Run </li></ul><ul><li>Servlet Request Dispatch </li></ul><ul><li>Servlet Request </li></ul><ul><li>Servlet Filter </li></ul><ul><li>Servlet Async Action </li></ul><ul><li>Servlet Context Execute </li></ul><ul><li>Servlet Response Write Headers </li></ul><ul><li>Servlet Response Send </li></ul><ul><li>Servlet Stale Resource </li></ul><ul><li>Servlet Check Access </li></ul><ul><li>JMS BE Consumer Log </li></ul>© 2010 Oracle Corporation Proprietary and Confidential
  22. 23. WLDF High Volume Events <ul><li>EJB Database Access </li></ul><ul><li>EJB Business Method Post Invoke Cleanup </li></ul><ul><li>EJB Pool Manager Remove </li></ul><ul><li>EJB Replicated Session Manager </li></ul><ul><li>EJB Timer Manager </li></ul><ul><li>JDBC Transaction Commit </li></ul><ul><li>JDBC Transaction End </li></ul><ul><li>JDBC Transaction Get XA Resource </li></ul><ul><li>JDBC Transaction Is Same RM </li></ul><ul><li>JDBC Transaction Prepare </li></ul><ul><li>JDBC Transaction Rollback </li></ul><ul><li>JDBC Transaction Start </li></ul><ul><li>JTA Transaction Commit </li></ul><ul><li>JTA Transaction End </li></ul><ul><li>JTA Transaction Prepared </li></ul><ul><li>JTA Transaction Prepare </li></ul><ul><li>JTA Transaction Start </li></ul><ul><li>Servlet Request Overload </li></ul><ul><li>Servlet Request Cancel </li></ul><ul><li>Servlet Context Handle Throwable </li></ul><ul><li>Plus stack traces and User IDs </li></ul>© 2010 Oracle Corporation Proprietary and Confidential
  23. 24. Automated Recording Capture <ul><li>Integrated with the WebLogic Diagnostic Image and Watch and Notification system </li></ul><ul><ul><li>Enables capture based on system state, event – capture during event; no need to replicate </li></ul></ul><ul><ul><li>Watch for stuck thread count, heap size increase, available memory, etc. </li></ul></ul><ul><ul><li>Set up notifications: capture WLDF image </li></ul></ul><ul><ul><li>Diagnostic image capture spurs JFR file generation; JFR file included in diagnostic image </li></ul></ul><ul><ul><li>Includes full JFR data from all event generators </li></ul></ul><ul><li>Integration with Oracle Support’s Remote Diagnostics Agent </li></ul><ul><ul><li>Generates WL Diagnostic Image </li></ul></ul>3 WLDF Trigger JFR Buffer Write Capture JFR in Diag Image
  24. 25. WebLogic Experimental Plug-in for Mission Control 3
  25. 26. WebLogic Experimental Plugin
  26. 27. <Insert Picture Here> WebLogic Tuning
  27. 28. Areas of Focus <ul><li>JVM’s and Machines </li></ul><ul><li>Threads / Work Managers </li></ul><ul><li>JMS </li></ul><ul><li>JDBC </li></ul><ul><li>EJB </li></ul>
  28. 29. The timeless battle with Garbage Collection <ul><li>How do you slice & dice your CPU and memory? </li></ul><ul><li>More JVM’s with smaller heap… or fewer JVM’s with bigger heap? </li></ul><ul><li>Bigger Heap = longer pause times for GC </li></ul><ul><li>Ultimately depends on your application – can you tolerate longer GC times? </li></ul>
  29. 30. JVM’s and Machines <ul><li>Machines: </li></ul><ul><ul><li>8GB RAM </li></ul></ul><ul><ul><li>Dual-CPU, Quad-Core </li></ul></ul><ul><li>Why run only one JVM per machine? </li></ul><ul><li>Run multiple JVMs! </li></ul>512MB: Admin 2GB: OS, etc 2GB: MS-1 2GB: MS-3 2GB: MS-2 2GB: OS, etc 2GB: MS-4 2GB: MS-5
  30. 31. Threads / Work Managers <ul><li>Define a work manager for each of your applications </li></ul><ul><li>WebLogic will self-tune the work managers for the best performance </li></ul><ul><li>You can define: </li></ul><ul><ul><li>Context Classes for prioritizing requests </li></ul></ul><ul><ul><li>Minimum Thread Constraint </li></ul></ul><ul><ul><li>Maximum Threads Constraint </li></ul></ul><ul><li>If you have an application that makes a WS call back to an application on WLS, the applications should have different Work Managers (ESB) </li></ul>
  31. 32. JMS <ul><li>Connection Factories </li></ul><ul><ul><li>Enable Load Balancing </li></ul></ul><ul><ul><li>Disable Server Affinity </li></ul></ul><ul><li>Persistent Storage </li></ul><ul><ul><li>File Stores offer the best performance </li></ul></ul><ul><ul><li>JDBC stores offer ease of failover </li></ul></ul><ul><li>Distributed Destinations </li></ul><ul><ul><li>Spread the work across the cluster </li></ul></ul><ul><ul><li>Partitioned Distributed Topics eases event architectures </li></ul></ul>
  32. 33. JDBC <ul><li>Use JNDI Datasources! </li></ul><ul><ul><li>Don’t use application-level connection pools – you can’t monitor them holistically </li></ul></ul><ul><li>Number of Threads / Connection Pool Size </li></ul><ul><ul><li>Max Threads Constraint associated with connection pool </li></ul></ul><ul><ul><li>Monitor & Minimize waiters </li></ul></ul><ul><li>Max Size = Initial Size, disable Shrinkage </li></ul><ul><ul><li>Less latency for handling peak loads </li></ul></ul><ul><li>Prepared Statement Cache </li></ul><ul><ul><li>Tune it so you have a high (+90%) hit rate </li></ul></ul><ul><ul><li>Write your applications to use prepared statements </li></ul></ul>
  33. 34. Common Application ‘mistakes’ <ul><li>Read a properties from a file on each request </li></ul><ul><ul><li>Poor performance – File I/O, doesn’t scale </li></ul></ul><ul><ul><li>Try storing your properties in a database and/or caching them in the JVM </li></ul></ul><ul><li>New JAXB context for each parse event </li></ul><ul><ul><li>Create one singleton context and reuse </li></ul></ul><ul><li>Too many session variables </li></ul><ul><ul><li>Clear a variable when you don’t need it anymore </li></ul></ul>
  34. 36. Base Content Slide <ul><li>First-level bullet </li></ul><ul><ul><li>Second-level bullet </li></ul></ul><ul><ul><ul><li>Third-level bullet </li></ul></ul></ul><ul><ul><ul><ul><li>Fourth-level bullet </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Fifth-level bullet </li></ul></ul></ul></ul></ul>
  35. 37. Example of Title Extending to Two Lines <ul><li>First-level bullet </li></ul><ul><ul><li>Second-level bullet </li></ul></ul><ul><ul><ul><li>Third-level bullet </li></ul></ul></ul><ul><ul><ul><ul><li>Fourth-level bullet </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Fifth-level bullet </li></ul></ul></ul></ul></ul>
  36. 38. Alternate Title with Subhead Arial Size 18 (Gray) <ul><li>First-level bullet </li></ul><ul><ul><li>Second-level bullet </li></ul></ul><ul><ul><ul><li>Third-level bullet </li></ul></ul></ul><ul><ul><ul><ul><li>Fourth-level bullet </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Fifth-level bullet </li></ul></ul></ul></ul></ul>

×