First Failure Data Capture for your enterprise application with WebSphere Application Server


Published on

How to add first failure data capture to your enterprise application

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • We’re here today to discuss First Failure Data Capture (FFDC) which is a serviceability component that complements log and trace in understanding root cause of problems while a system is running.
  • The agenda for today starts with a description of FFDC, an explanation of some of the key concepts necessary to fully exploit FFDC, then we start into samples that show FFDC usage.
  • FFDC is used only when problems occur in java code What differentiates it from logging is that: An exception has already occurred. This means that performance has been impacted and the code is in a failure path FFDC is in the category of dump or snapshot type tools. These tools aim to provide a broad view of the system at a particular point in time. Logging keeps a narrow view, over a period of time Each FFDC statement executes only once (some rare caveats we discuss later). If the statement is executed again, it will know that it has already been executed, and it will simply update summary information. This means that FFDC processing can focus more on capturing all needed information, and less on performance. The FFDC infrastructure provides many points where developers can plug in code that will be called when appropriate. The class or method experiencing a failure rarely knows all of the needed context to resolve the problem. FFDC provides extensions so that more focused serviceability code can take a more holistic view and gather a much broader context. The extension points can be used in OSGi or J2SE environments. FFDC is used in WebSphere, but it is not dependent on WebSphere. Its only dependency is JDK 1.5 or later. FFDC keeps a running tab on all FFDC incidents that occur. When a particular incident occurs multiple times, it is only processed the first time. Subsequent calls simply update the summary information. A reference in the back refers to a CAPS (Council for Advanced cross Product Serviceability) web site discussing FFDC best practices and concepts further
  • As you’ll see from the samples, FFDC is simple to use. While it provides tremendous functionality and extensibility, simple usage requires little more than including one jar in a class path and making calls that closely resemble logging calls When an FFDC log call executes, if it is the first time, it generates an incident and updates a summary. Depending on configuration, each incident can be a separate file in a directory, or the incidents and summary information can be appended into a single file or outputStream FFDC provides logging-like guards which can be used to avoid the cost of gathering information if that process is expensive One of the most powerful extension points in FFDC is the Data Collector. This is a class provided by the caller that will be called if the callStack in the exception includes certain classes or methods. An example of this is that a caller can provide a Data Collector that capture all key information about the WebContainer, and register this Data Collector to run if the callStack includes WebContainer classes. Basically, a Data Collector has relevant domain knowledge that the caller of FFDC need not have. Formatters are another important FFDC extension point. FFDC can use java reflection to render all of the context objects passed in as well as their child objects down to 3 levels (grand-children). In some cases, however, a custom formatting of an object greatly improves the usability of its rendering. This extension allows callers to create classes that do custom formatting. IncidentForwarders are 3 rd of 4 current extensions. This enables a caller to be informed any time an FFDC incident is created. This enables the caller to provide auxiliary function. An example is the FFDC Analyst project which will forward incidents to a Prosol data base where problem reDiscovery will occur based on advanced heuristics which compare callStack qualities. The final extension point is a provider. This is also for advanced users only. It enables custom handling of FFDC incidents for callers who have additional functionality requirements and/or legacy compatibility issues FFDC maintains a table in memory (also dumped periodically to a file or output stream) with all the incidents that have occurred along with additional properties about the incident (how many times has the log call been executed, when was the last time, where did the original incident get written). It’s important to note here that, even though FFDC provides highly extensible and powerful functionality, very little is required to get started and to greatly improve your software serviceability. The added functionality, if needed, is something your software can grow into.
  • This is an example of using FFDC. When an exception is caught, a call to Ffdc.log is made. The arguments are described there: The Exception. This is a java exception which provides much of the information that FFDC needs to function The reporting class. This is important to the FFDC processing and becomes the first object rendered The sourceId. This is part of the key that makes an incident unique. Most callers use the className and methodName concatenated together The ProbeId. This is a second part of the key. Most callers either use a line number, or some indication to uniquely identify this log call The rest is a list of context data elements. These are objects whose information will be valuable in understanding exactly what went wrong. It is generally best to err on the side of sending too much.
  • This slide shows how FFDC handles the log call from the last slide First it checks to see if this log statement has occurred before. It compares the “incident key” (which is made up of the “sourceId”, “probeId”, and exception name) against the summary table to see if this incident has already occurred. If it has already occurred, then FFDC updates the summary table and returns to the caller (no new incident is created) FFDC then goes through the call stack and compares each stack frame to see if any Data Collectors are listening for that package/class/method combination. Remember data collectors are a mechanism callers can use to gather additional context data. They register to be called if certain classes are in the call stack. So if there is a match between a class in the call stack and a class that a data collector has registered to listen to, then the data collector will be called, and it will return additional context data we call captured data elements (CDEs) FFDC creates an incidentStream and writes key information from the call and the exception into the incidentStream Now that all “captured data elements” (those from the log call and those from the Data Collectors) are there, FFDC begins to render all of the information into the incident stream. For each cde, it first checks to see if there is a registered formatter to do custom formatting. If not, it checks to see if the cde is formattable (if it implements the formattable interface). If not, the cde is rendered via java reflection. If there is sensitive data in the object that should not be rendered, it is not required to do custom formatting. Placing the annotation @FFDC_OMIT above an object will tell FFDC not to render the information even if using reflection Incident is finally rendered to the output location (separate file or append to running file or outputStream) and the summary table is updated The summary report is updated to reflect addition of this incident If there are any registered incidentForwarders, they will be notified of the incident
  • This is a graphical depiction of the flow from the previous slide. It focuses on what is built into the infrastructure and what is provided by the caller. Remember that everything but the Log call itself is a customization that is not needed to get started. These are there to help provide better context and customized behavior without having the primary code in your software focus on detailed context collection.
  • This is a typical example in code. This is an excerpt, the full compilable and runnable samples are referenced in the Resources section at the end of the presentation. Import of Manager.Ffdc is the only required FFDC import. Other imports are needed only if exploiting more advanced FFDC functions System Property determines which default FFDC Provider is used. Providers determine the behavior of FFDC and the handling of incidents. Several providers are provided with FFDC and developers can implement the provider interface and create their own back end behavior. The code using FFDC need not be concerned with which provider is in place Providers can be changed at a later time by the application using FFDC The values for the default (startup) provider are: <fileName> if a file name is specified, the logic is as follows: If it exists and it is a file, a file of that name is created and all incidents and summary reporting are appended into that file If it exists and is a directory, then all incidents are written as separate files into that directory. The summary report will also be a separate file in that directory If it does not exist and ends in File.separator (\\ or /) a directory is created and all incidents are written as separate files into that directory. The summary report will also be a separate file in that directory If it does not exist and does not end in File.separator (\\ or /) a file of that name is created and all incidents and summary reporting are appended into that file Output Stream options System.out or System.err append the incidents and summary report to the stdout or stderr output stream (System.err is the default) A final option of Suppress is available which will discard all ffdc information In the caller’s code, you see that the Ffdc.log appears in the catch block, when an exception has been generated You can see the option to call Ffdc.log directly as is done here, or to create the ffdc object and use the isLoggable trace guard. The trace guard option is for when it will be expensive to gather the needed cdes for the call Log statement can have an arbitraty number of objects at the end. If collections or arrays are passed explicitly, they will be rendered completely. Every element will show up in the incident If arrays or collections are discovered in rendering other objects, just the properties of the collection or array will be rendered (number of elements and type)
  • This is an example of using a Formatter. Note the extra import statements for the registration process and the trace guard. This shows J2SE programmatic registration, OSGi enables declarative and programmatic registration You can see the 2 lines that construct and register the formatter. This can be done anywhere and any time. It will take affect immediately. The code doing FFDC logging need not be aware of registered formatters When the FFDC infrastructure renders the customer object, it will find this registered formatter and drive it
  • This is the actual formatter referenced on the previous slide. Note that it is passed a reference to the object and the IncidentStream being used to render the object. Formatter must have access to the information in the object. Examples would be public/protected members, getr methods, or reflection The formatTo method uses write methods on the incident stream to pass the information from the object back to the FFDC infrastructure The getSupportedTypeNames method returns an array of package.class names that this formatter can format. Class can be a regular expression, package cannot The isSupported method takes a Class and determines if this formatter will work on it.
  • This is a sample using a Data Collector. Note that the registration of the data collector is similar to registering a formatter. Note here, that it would be easiest to pass exposedGlobals on the log call, but we are getting it via the Data Collector to demonstrate data collector functionality Remember, a DataCollector is a specialist in collecting data from a particular piece of the environment. The caller need not know about that part of the environment or that a Data Collector is even registered.
  • This is the data collector used on the prior slide Unlike a Formatter which uses IncidentStream write methods, the Data Collector returns its information as a Collection which FFDC sees as Captured Data Elements or CDEs. The getSupportedTypeNames provides a list of package qualified class names, and optionally a method with each. If any of these classes are seen in the callStack of the exception, then this data collector will be called. If this data collector matches multiple entries in the callStack, it will only be called once. Data Collector must have a mechanism for accessing the data it needs. Advanced exploiters of FFDC have used singleton global classes to give the DataCollectors starting points to gather the information needed. In the WebSphere space, a Data Collector can use MBeans, HealthCheckers, Diagnostic Providers, or any other mechanism that exposes data
  • When FFDC renders an object, it first determines if the caller has any custom formatting. Registered formatters are the first option, then Formattable, and finally reflection Note that each original context data element in the Ffdc.log call are rendered down to 3 levels of children. As each child, grandChild, or greatGrandchild object is rendered, it uses the same formatting hierarchy. An example: If a connection pool uses Ffdc.log and sends a collection of connections … each connection may be rendered by a registered Formatter This connection may include a connectionStatus child that the Formatter writes back to FFDC. This connectionStatus is Formattable. FFDC will find it implements Formattable and drive its formatTo object The connectionStatus child object may include a date in it that is rendered via reflection
  • FFDC exploits, but does not require OSGi. While our prior examples showed registration in a J2SE environment, the next 2 slides demonstrate registration in an OSGi environment. In an OSGi envioronment, declarative registration is a simple approach. In our sample, an entry is made in the MANIFEST.MF pointing to a separate XML file The contents of that XML file define the class that will get registered (as the formatter in this case)
  • Another option for registering extensions in OSGi is programmatic registration of a service. This is generally done in the Activator class of a bundle using the start method The process is to construct your class, then register it as a service
  • The registration process of extending FFDC is completely dynamic. At any time during the life of process; data collectors, formatters, providers, and incident forwarders can be registered or unregistered This is a nice feature but … if an incident has already occurred, then it will not normally occur again until the process stops and restarts To resolve this situation, FFDC enables unblocking of incidents, a specified incident, or all incidents. Unblocking an incident allows the associated FFDC call to render an incident on the next execution. That is, it allows the same incident to occur a second time. This is especially helpful for longRunning processes.
  • Hopefully slide says it all
  • First Failure Data Capture for your enterprise application with WebSphere Application Server

    1. 1. First Failure Data Capture Getting Started Guide Authors: Michael Casile Stefan Derdak
    2. 2. Agenda <ul><li>What is FFDC </li></ul><ul><li>Key Concepts </li></ul><ul><li>Usage Sample </li></ul><ul><li>Flow Example (resulting from the usage) </li></ul><ul><li>Advanced usage samples </li></ul><ul><li>Advanced Topics </li></ul><ul><li>Summary </li></ul>
    3. 3. What is FFDC <ul><li>First Failure Data Capture (FFDC) is used to capture diagnostic data when a problem occurs in code. </li></ul><ul><li>Different from logging </li></ul><ul><ul><li>Called only when exceptions have occurred </li></ul></ul><ul><ul><li>More snapshot/dump than a history </li></ul></ul><ul><ul><li>Executes only once (so performance less of an issue) </li></ul></ul><ul><ul><li>Includes functionality and extensibility to capture more data and renders more information </li></ul></ul><ul><ul><li>Goal is to capture enough context information when a problem occurs, that there is no need to reCreate the problem </li></ul></ul><ul><li>Highly extensible in OSGi and J2SE </li></ul><ul><li>Exists as a jar/bundle with no dependencies (JDK) </li></ul><ul><li>Tracks summary information on all incidents </li></ul>
    4. 4. Key Concepts <ul><li>Simple to use (Ffdc.log) </li></ul><ul><li>Unique incident “file” created for first execution of any Ffdc.log </li></ul><ul><li>isLoggable ffdc guard </li></ul><ul><li>Data Collectors – dynamic event listener based on stack frames </li></ul><ul><li>Formatters – Part of special rendering framework for objects </li></ul><ul><li>IncidentForwarder – Listener called at completion of any incident creation </li></ul><ul><li>Provider – Custom FFDC implementation (dynamically pluggable) </li></ul><ul><li>Summary Report/Table – View of info on incidents that have occurred </li></ul>
    5. 5. Usage Sample <ul><li>try { </li></ul><ul><li>// Application code here </li></ul><ul><li>} catch (Exception e) { </li></ul><ul><li>Ffdc .log (e, myClass, myClassNm+myMethodNm, “lineNumber”, cde1, cde2, …) ; </li></ul><ul><li>} </li></ul><ul><li>Args: Exception, reporting class, “sourceId”, “probeId”, context data elements </li></ul><ul><li>where sourceId and probeId are any strings, but this pattern is common </li></ul>
    6. 6. Flow Example (how is that call handled) <ul><li>Determines if this incident has already occurred (stops if it has) </li></ul><ul><li>Checks for registered Data Collectors (does any registered DC want to be called on anything in stack). DC’s capture additional captured data elements (CDEs) </li></ul><ul><li>Creates incident stream and writes header/exception </li></ul><ul><li>Render each CDE from call or from Data collectors </li></ul><ul><ul><li>Looks to format each cde with registered formatter, or formattable, or reflection. </li></ul></ul><ul><ul><li>@FFDC_OMIT to skip certain discovered cdes. </li></ul></ul><ul><li>Renders the incident to the output location (file/dir) </li></ul><ul><li>Updates the summary </li></ul><ul><li>Notifies registered incidentForwarders </li></ul>
    7. 7. Flow Example (Diagram) Client Code Log API call Registered DataCollectors Registered Formatters Registered Incident Forwarders FFDC Infrastructure Incident Incident Stream Summary Table Summary Report 1 7 4b 4a 2a 3 2b 5 6
    8. 8. Advanced Usage Topics (Simple, with isLoggable sample) <ul><li>package howto_ffdc._1_simple; </li></ul><ul><li>import static Ffdc ; </li></ul><ul><li>// import; // Used if alternate call is done below </li></ul><ul><li>public class SimpleTest extends TestCase { </li></ul><ul><li>protected void setUp() throws Exception { </li></ul><ul><li>System. setProperty ( &quot;; , “/opt/IBM/WebSphere/logs/ffdc/&quot; ); </li></ul><ul><li>} </li></ul><ul><li>public void testWithoutFormatter(){ </li></ul><ul><li>try { </li></ul><ul><li>// ... do work </li></ul><ul><li>} catch (Exception e) { </li></ul><ul><li>Ffdc .log(e, this , getClass().getName(), &quot;24&quot; , customer); </li></ul><ul><li>/*alternate if generating the parms for the call can be expensive </li></ul><ul><li>* Ffdc ffdc = Ffdc.getFfdc(e, this, getClass().getName(),&quot;24&quot;) ; </li></ul><ul><li>* if (ffdc.isLoggable()) </li></ul><ul><li>* MyData myData = expensiveCallToGetData() ; </li></ul><ul><li>* ffdc.log(customer, myData) ; </li></ul><ul><li>*/ </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    9. 9. Advanced Usage Topics (Formatter part 1) <ul><li>import; </li></ul><ul><li>import; </li></ul><ul><li>import static Ffdc ; </li></ul><ul><li>/** </li></ul><ul><li>* This example shows how to register a formatter at program startup, and illustrates the usage. </li></ul><ul><li>*/ </li></ul><ul><li>public class FormatterTest extends TestCase { </li></ul><ul><li>protected void setUp() throws Exception { </li></ul><ul><li>System. setProperty ( &quot;; , &quot;System.err&quot; ); </li></ul><ul><li>/* Construct and register the formatter. */ </li></ul><ul><li>CustomerFormatter customerFormatter = new CustomerFormatter(); </li></ul><ul><li>FfdcConfigurator. register (customerFormatter); </li></ul><ul><li>} </li></ul><ul><li>public void testFormatter(){ </li></ul><ul><li>Customer customer = null ; </li></ul><ul><li>try { </li></ul><ul><li>// ... do work </li></ul><ul><li>customer = new Customer(1001, &quot;Jane&quot; , &quot;Dow&quot; ); </li></ul><ul><li>} catch (Exception e) { </li></ul><ul><li>Ffdc ffdc = Ffdc .getFfdc(e, this , &quot;24&quot; ); </li></ul><ul><li>if (ffdc.isLoggable()) { </li></ul><ul><li>String ctx = &quot;expensive to retrieve context data&quot; ; </li></ul><ul><li>ffdc.log(customer, ctx); </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    10. 10. Advanced Usage Topics (Formatter part 2) <ul><li>import; </li></ul><ul><li>import; </li></ul><ul><li>public class CustomerFormatter implements Formatter { </li></ul><ul><li>public void formatTo(Object objectToFormat, IncidentStream is) throws IllegalArgumentException { </li></ul><ul><li>formatTo((Customer)objectToFormat, is); </li></ul><ul><li>} </li></ul><ul><li>public void formatTo(Customer customer, IncidentStream is) throws IllegalArgumentException { </li></ul><ul><li>is.write( &quot;id&quot; , customer. id ); </li></ul><ul><li>is.write( &quot;name&quot; , customer. name ); </li></ul><ul><li>is.write( &quot;surname&quot; , customer. surname ); </li></ul><ul><li>} </li></ul><ul><li>public String[] getSupportedTypeNames() { </li></ul><ul><li>return new String[] {Customer. class .getName()}; </li></ul><ul><li>} </li></ul><ul><li>public boolean isSupported(Class<?> clazz) { </li></ul><ul><li>return Customer. class .equals(clazz); </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    11. 11. Advanced Usage Topics (DataCollector part 1) <ul><li>import; </li></ul><ul><li>import static Ffdc ; </li></ul><ul><li>public class DataCollectorTest extends TestCase { </li></ul><ul><li>public void setUp() throws Exception { </li></ul><ul><li>System. setProperty ( &quot;; , &quot;System.err&quot; ); </li></ul><ul><li>FfdcConfigurator. register ( new DataCollectorSimple()); </li></ul><ul><li>} </li></ul><ul><li>public void testDataCollector() { </li></ul><ul><li>ExposedGlobals exposedGlobals = new ExposedGlobals() ; </li></ul><ul><li>try { </li></ul><ul><li>throw new Exception( &quot;Yes, had ExposedGlobals, but getting them thru DataCollector for this example”) ; </li></ul><ul><li>} catch (Exception e) { </li></ul><ul><li>Ffdc .log(e, this , DataCollectorTest. class .getName()+ &quot;testDC&quot; , &quot;01&quot; ) ; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    12. 12. Advanced Usage Topics (DataCollector part 2) <ul><li>import; </li></ul><ul><li>import java.util.Collection; </li></ul><ul><li>import java.util.Collections; </li></ul><ul><li>import java.util.Properties; </li></ul><ul><li>class DataCollectorSimple implements DataCollector { </li></ul><ul><li>// Return collection of CDEs </li></ul><ul><li>public Collection<? extends Object> collect(Throwable ex) { </li></ul><ul><li>Properties propsToGather = ExposedGlobals. getInstance ().getProps() ; </li></ul><ul><li>return Collections. singleton (propsToGather) ; </li></ul><ul><li>} </li></ul><ul><li>public String[] getSupportedTypeNames() { // What to look for in stackFrames </li></ul><ul><li>return new String[]{ </li></ul><ul><li>DataCollectorTest. class .getName() + &quot;#testDataCollector&quot; </li></ul><ul><li>}; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    13. 13. Advanced Topics: Formatter details <ul><li>Formatting techniques and priority </li></ul><ul><ul><li>Formatter – if a registered formatter is found for a class, it is used </li></ul></ul><ul><ul><li>Formattable – If a class is formattable </li></ul></ul><ul><ul><ul><li>If the formatTo method is in this class, it is used </li></ul></ul></ul><ul><ul><ul><li>If it is in a parent class, it is used, then the remainder of this class is rendered via reflection </li></ul></ul></ul><ul><ul><li>Reflection – anything w/out a formatter or formattable is rendered with reflection. In reflection, objects annotated with @FFDC_OMIT are not rendered </li></ul></ul><ul><li>Recursive dispatch </li></ul><ul><ul><li>As each object is rendered, it’s children are rendered (to 3 levels) </li></ul></ul><ul><ul><li>When the child is rendered, the formatting technique is applied to the child. Ie: a reflected object may include a Formattable object or an object for which a Formatter has been registered </li></ul></ul>
    14. 14. Advanced topics: OSGi registration Best practice for registering FFDC extensions (formatters, data collectors, providers, incident forwarders) in OSGi is to use declarative services. Here is an example: Add to the MANIFEST.MF the line: Service-Component: OSGI-INF/CustomerFormatter.xml and add the OSGI-INF/CustomerFormatter.xml file with the content: <? xml version = &quot;1.0&quot; ?> < scr:component xmlns:scr = &quot;; immediate = &quot;true&quot; name = &quot;CustomerFormatter&quot; > < implementation class = &quot;howto_ffdc.domain.ffdcsupport.CustomerFormatter&quot; /> < service > < provide interface = &quot;; /> </ service > </ scr:component >
    15. 15. Programmatic registration in OSGi <ul><li>A simple way is to register an OSGi service via your bundles Activator as this sample demonstrates: </li></ul><ul><li>public void start(BundleContext context) throws Exception { </li></ul><ul><li>Formatter formatter = new CustomerFormatter(); </li></ul><ul><li>context.registerService(Formatter. class .getName(), </li></ul><ul><li>formatter, new Hashtable ()); </li></ul><ul><li>System. out .println( &quot;Exported service:&quot; +formatter.getClass().getName()); </li></ul><ul><li>} </li></ul>
    16. 16. Advanced topics: Incident reset <ul><li>Dynamic extensibility is a key them of FFDC but … </li></ul><ul><ul><li>What good is dynamically adding a new dataCollector (formatter, forwarder, …) if the incident already occurred </li></ul></ul><ul><ul><li>FFDC also provides access to the Summary table </li></ul></ul><ul><ul><ul><li>List < Incident > incidentList = Ffdc .getIncidents(); </li></ul></ul></ul><ul><ul><ul><li>boolean unblocked = Ffdc .unblockLogging( myIncident ); </li></ul></ul></ul><ul><ul><ul><li>Ffdc .unblockLogging() ; </li></ul></ul></ul><ul><ul><li>With these method calls, one incident or all incidents can be modified so that the next time this Ffdc.log statement executes, it will create another incident </li></ul></ul>
    17. 17. Summary <ul><li>FFDC is a simple java facility to improve the serviceability of you java software </li></ul><ul><ul><li>Low cost to implement </li></ul></ul><ul><ul><li>Extensible, can grow w/you (Data Collectors, Formatters, Providers, and Incident Forwarders) </li></ul></ul><ul><ul><li>Extensions do not impact core code (no change needed to Ffdc.log statements to affect improved information) </li></ul></ul>