Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
IBM BPM Problem
Determination
http://www.coutureconsulting.com/
long.nguyen@coutureconsulting.com
Overview
Websphere application server architecture
Network deployment concepts
Runtime Flow
Administration flow
Clustered application servers
IBM BPM key databases
● CMNDB
o BPEL instances
● BPMDB
o BPD instances
● PDWDB
● The CMNDB and BPMDB are the runtime datab...
Tuning method
Tuning considerations
Problem determination
Problem analysis
1. review the log file (SystemOut.log, ffdc logs, jvm
logs)
2. check product versio...
Log files and locations
Log or trace file format
SystemOut.log
- Useful data such as software levels for the WebSphere
Application Server binary files and the IBM Business...
SystemOut.log examples
FFDC (first-failure data capture)
file naming convention:
<server name>_<thread ID>_yy.MM.dd_HH.mm.ss_<unique ID>.txt
and ...
ffdc in logs folder example
ffdc content example
High Performance Extensible Logging (HPEL)
• Performance
– Log and trace events are stored as binary data so performance i...
High Performance Extensible Logging
(HPEL)
HPEL on developerworks:
http://www.ibm.com/developerworks/webspher
e/techjourna...
Problem diagnostic tips
- understanding the system architecture
- timestamp in the log files (eg:
ttps://docs.google.com/s...
Kind of data should be collect for
performance-related problem
- Thread dumps
- Heap dumps
- Java gc logging
http://www-01...
Troubleshooting JVM problems
Problem Determination - collect log files
1. verbose gc log, heap memory dump
2. java core fi...
Troubleshooting JVM problems
Symptoms analysis
1. server crashes
- application server stop or exit unexpectedly
1. hung pr...
Troubleshooting JVM problems
garbage collection logs
● verbosegc is written to native_stderr.log (ibm jdk), native_stdout....
Troubleshooting JVM problems
heapdumps and javacores (javadumps, threaddumps)
heapdumps
● heapdumps is dump of all the liv...
Troubleshooting JVM problems
root profile location
Troubleshooting JVM problems
memory and heap analysis
● memory analyzer tool
● memory dump diagnostic for java
● ibm heapa...
Troubleshooting JVM problems
jvm documents
http://www.ibm.com/developerworks/java/jdk/do
cs.html
demo: java -jar
/home/lon...
Problem prevention and best practices
- set up monitoring and detect problems
- create and maintain system architecture
di...
Resources:
course materials
https://www.evernote.com/shard/s210/sh/8b02a558-639a-4e54-a610-
3f247d8849c4/3ca31d18ebfe78df8...
Upcoming SlideShare
Loading in …5
×

Ibm bpm problem determination

1,845 views

Published on

ibm bpm problem determination

Published in: Technology
  • Be the first to comment

Ibm bpm problem determination

  1. 1. IBM BPM Problem Determination http://www.coutureconsulting.com/ long.nguyen@coutureconsulting.com
  2. 2. Overview
  3. 3. Websphere application server architecture
  4. 4. Network deployment concepts
  5. 5. Runtime Flow
  6. 6. Administration flow
  7. 7. Clustered application servers
  8. 8. IBM BPM key databases ● CMNDB o BPEL instances ● BPMDB o BPD instances ● PDWDB ● The CMNDB and BPMDB are the runtime databases, which store process instances information for BPD and BPEL ● The default configuration when creating the BPE container through the administrative console is CMNDB, but can be changed More details: https://www.evernote.com/shard/s210/sh/70704404-0d24-42f0-9e1a- 6231855b02af/1a3fd21649db5656ed4163f45d0b5948
  9. 9. Tuning method
  10. 10. Tuning considerations
  11. 11. Problem determination Problem analysis 1. review the log file (SystemOut.log, ffdc logs, jvm logs) 2. check product version, search message code with the product version 3. ibm support assistant, forum, bpm community , tech notes, ... Reference link: http://www- 01.ibm.com/support/docview.wss?uid=swg21569731
  12. 12. Log files and locations
  13. 13. Log or trace file format
  14. 14. SystemOut.log - Useful data such as software levels for the WebSphere Application Server binary files and the IBM Business Process Manager binary files - A mix of informational, warning, and error messages
  15. 15. SystemOut.log examples
  16. 16. FFDC (first-failure data capture) file naming convention: <server name>_<thread ID>_yy.MM.dd_HH.mm.ss_<unique ID>.txt and contains information that is relative to the value of the ffdcRun.properties Level property value
  17. 17. ffdc in logs folder example
  18. 18. ffdc content example
  19. 19. High Performance Extensible Logging (HPEL) • Performance – Log and trace events are stored as binary data so performance is substantially faster than default log and trace framework (basic mode) – Less impact to systems when trace is enabled • Extensions – HPEL includes information about which application each log or trace record is from – When used with cross-component trace, HPEL includes information about which request each log or trace record is from – Developers can add their own log or trace record extensions – HPEL logViewer command enables filtering logs or trace data by extension name and value • Filtering – HPEL logViewer command helps you filter logs and trace data by date, time, level, and thread ID
  20. 20. High Performance Extensible Logging (HPEL) HPEL on developerworks: http://www.ibm.com/developerworks/webspher e/techjournal/1208_bourne/1208_bourne.html
  21. 21. Problem diagnostic tips - understanding the system architecture - timestamp in the log files (eg: ttps://docs.google.com/spreadsheets/d/1c65eHb8mZTBeva2HOvmyuHv5ksl7UiEW-MLq3LTgzXE/pubhtml ) - count occurrence times of error message - guest in which context may cause the errors
  22. 22. Kind of data should be collect for performance-related problem - Thread dumps - Heap dumps - Java gc logging http://www-01.ibm.com/support/docview.wss?uid=swg21114927 - Event manager and process monitor - Custom solution code logging - Instrumentation logging https://bp3.zendesk.com/hc/en-us/articles/202144583- Instrumentation-Monitor-page-in-IBM-BPM-the-missing-manual-part-I)
  23. 23. Troubleshooting JVM problems Problem Determination - collect log files 1. verbose gc log, heap memory dump 2. java core files
  24. 24. Troubleshooting JVM problems Symptoms analysis 1. server crashes - application server stop or exit unexpectedly 1. hung process - verify that app server process is running - threads might be deadlocked - looping in code 1. out of memory - errors and exceptions that are logged without process exit - at times, can process in unexpected process exit 4. performance degradation - app server might crash and the nodeagent restarts it - check to see whether the process ID is continually changing
  25. 25. Troubleshooting JVM problems garbage collection logs ● verbosegc is written to native_stderr.log (ibm jdk), native_stdout.log (sun solaris, hp-ux java package) ● to enable verbosegc log (http://www- 01.ibm.com/support/docview.wss?uid=swg21114927 ) ● tool to visualize http://www.ibm.com/developerworks/java/jdk/tools/gcmv/
  26. 26. Troubleshooting JVM problems heapdumps and javacores (javadumps, threaddumps) heapdumps ● heapdumps is dump of all the live objects that are on the java heap ● heapdumps file (phd portable heap dump) - OutOfMemory error located in <profile root> dir javacore ( javadump, threadump) ● all about java thread present, classes loaded, heap statistics ● javacore is generated automatically by event of java crash or OutOfMemory error ● manually: o - kill -3 on unix or wsadmin o cmd>wsadmin wasdmin>set jvm [$AdminControl completeObjectName type=JVM,process=%1,*] > %FILENAME% wsadmin>$AdminControl invoke $jvm dumpThreads >> %FILENAME% location ● <profile root> for ibm jdk ● native_sdout.log solaris and hp-ux native_stdout.log
  27. 27. Troubleshooting JVM problems root profile location
  28. 28. Troubleshooting JVM problems memory and heap analysis ● memory analyzer tool ● memory dump diagnostic for java ● ibm heapanalyzer verbosegc log analysis ● ibm garbage collection and memory visualizer ● ibm pattern modeling and analysis tool for java garbage collector javacore analysis ● ibm thread and monitor dump analyzer for java
  29. 29. Troubleshooting JVM problems jvm documents http://www.ibm.com/developerworks/java/jdk/do cs.html demo: java -jar /home/longnguyen/ccvietnam/workspace/costo/ utils/ga456.jar
  30. 30. Problem prevention and best practices - set up monitoring and detect problems - create and maintain system architecture diagram - establish baselines (system load time, response time) - define a diagnostic collection plan - create maintenance plan
  31. 31. Resources: course materials https://www.evernote.com/shard/s210/sh/8b02a558-639a-4e54-a610- 3f247d8849c4/3ca31d18ebfe78df8e1106db4f6f623f ibm knowledge center for trouble shooting http://www- 01.ibm.com/support/knowledgecenter/SS964W/com.ibm.wbpm.admin.doc/topics/welcome_wps_tro.html ibm wait https://wait.ibm.com/ ibm developer works http://www-01.ibm.com/support/knowledgecenter/SSFPJS_8.5.0/ditamaps/ic-homepage-bpm.html analyze java thread dumps http://architects.dzone.com/articles/how-analyze-java-thread-dumps ?google

×