1. Java: Finding Bugs, Fixing Bugs
in IBM Domino Designer and XPages
Julian Robichaux Senior Software Engineer, panagenda
Paul T. Calhoun Senior Software Engineer, panagenda
MWLUG 2014

2. Your Speakers
• Julian Robichaux
- Developer at panagenda, an Austrian based IT management
software provider
- Purveyor of nsftools.com
- Writer of open-­‐source (and commercial) software
- Speaker at Lotusphere since 2006, many LUG and View
conferences in various countries
- Doing almost exclusively Java and Eclipse development (often
Notes related) for the past few years
- Not as pretty as Paul Calhoun, but still easy on the eyes

3. Your Speakers
• Paul T. Calhoun
- I have been working with Lotus/IBM technologies for the past 25
years.
- I have been certified in both Administration and Development on
every major release of Notes/Domino since release 3.
- My focus has always been on training and enablement of
Administrators and Developers, but I also provide mentoring and
consulting services.
- Most recently I have been focused on XPages consulting, training
and enablement.
- When not working, my passion is spending all my spare time with
my awesomely wonderful grand-­‐kids. (WARNING: I have
pictures !)

4. Agenda
• Debugging
- Java Agents
- XPages
• Logging
• Using FindBugs
• Using JConsole and Dump Analysis
• Q & A

5. Debugging Java Agents
• There are several steps to enable a Java debugger to debug a
Java agent running on the Notes client:
- Enable Java debugging on the Notes client.
- Enable debugging in the agent (compile code with debug info).
- Add thread sleep commands to the Java agent.
- Set breakpoints in the Java code.
- Create a Debug Configuration in the Java Perspective in Domino
Designer to connect to the Notes client.
- Add the agent source to the debug configuration.
- Run the debugger in Domino Designer client.
- Start the Notes agent to debug.
- Debug the code as needed.
• Detach the debugger from Notes.

6. Enable Debugger
• Follow these steps to enable remote Java debugging on the
Notes client:
- Choose Tools | Java Debugging Preferences from the menu.
- In the Java Debugging Preferences dialog, choose the appropriate
types of agents to debug and set the port values.
- Port values must be unique when enabling debugging for more
than one type of agent.
- Click [OK] to save the settings and close the dialog.

7. Enable Debugger
• Variables in the NOTES.INI file are added to reflect the port
settings and agent type selections
- JavaDebugClientForeground=1
- JavaDebugClientForegroundPort=8701
- JavaDebugClientScheduled=1
- JavaDebugClientScheduledPort=8702
- JavaDebugClientWebPreview=1
- JavaDebugClientWebPreviewPort=8703
!
• Notes must be restarted for the NOTES.INI changes to take
effect. Click [OK] when prompted to restart Notes.
• Exit and restart Notes.

8. Enable Agent for Debugging
• Follow these steps to enable debugging for a Java agent:
- Open the Java agent's Properties.
- From the Basics tab of the Properties view
• Enable the 'Compile Java code with debugging information' option.
• This will add line numbers and information about variables to the
compiled code to allow the agent to be debugged.

9. Enable Breakpoints
• Follow these steps to add breakpoints to an agent:
- Open the agent in Designer.
- Breakpoints can be added by either:
• Right-­‐clicking in the left margin of the code editor and choosing
• Toggle Breakpoint
• Double-­‐clicking in the margin next to the code to add/remove the
breakpoint.
- The breakpoints are indicated by a blue dots in the bar beside the
editor margin.

10. Create Debug Configuration
• This only has to be done once
• In Designer:
- Click the down arrow next to the blue “Debug” icon
- Choose “Manage Debug Configurations”

11. Create Debug Configuration
• In the Debug Configurations Dialog
- Right Click on “Remote Java Application”
- Choose “New”

12. Set Debug Configuration
• Provide a Name
• Change Port to Port set in Notes.ini
- Default is 8701 but can be any open port
• Click “Apply” and “Close” to save
• Click “Apply” and “Debug” to start the debugger listening on
the defined port

13. Switch to the Debug Perspective
• If not already in the Debug Perspective a dialog will open
prompting to switch to that perspective
- Choose “Yes”
• Optionally choose “Remember my decision” to suppress this
dialog from popping up every time

14. The Debug Perspective

15. Debugging
• The Code will stop at the first breakpoint
• From there you can…
- Inspect (and change) variables
- Step through the code using the icons

16. Stop the Debugger
• After the debugging session is complete the debugger is still
“listening” on the open port
• In the debug perspective make sure to “Disconnect” from the
listening port to turn debugging off

17. Debugging Java XPages
• If XPages make calls to Java Code elements then that Java
code can be debugged using a similar process to debugging
agents.
• In order to debug Java code called by an XPage add the
following lines to the notes.ini
- JavaEnableDebug=1
- JavaDebugOptions=transport=dt_socket,server=y,suspend=n,addr
ess=<port to listen on>
• GOTCHA !!
- If the previous ports are enabled for Java agent debugging then
“connecting” to the port above will never work
- Disable the Java Debug Ports and restart your Notes, Designer
client prior to debugging Java called from XPages
- Your firewall might block the port

18. Debug Configuration
• Create a new Debug Configuration to “listen” on the port
that was set in the notes.ini file
• Preview the XPage to start the local web preview
• Start the debugger

19. Execute XPage Action
• Set breakpoints in the Java code to be debugged
• In the XPage in the browser
- Execute the action that will call the Java code
• If not already in the Debug perspective you will be prompted
to switch to that perspective
• Step through your code reviewing the variable values at each
breakpoint
• Clear breakpoints and resume to end
• Disconnect from Debugger when done.

20. WARNING
DO NOT EVER ENABLE
DEBUGGING ON A
PRODUCTION SERVER

21. Demo

22. Agenda
• Debugging
- Java Agents
- XPages
• Logging
• Using FindBugs
• Using JConsole and Dump Analysis
• Q & A

23. Logging

24. Logging Basics
• System.out.println() is NOT logging!
• Java has native logging classes
- java.util.logging
- There are other popular logging frameworks, which we will ignore
for the sake of brevity
- The native Java classes work just fine
• The big picture:
- Each bit of code can have its own logger, or loggers can be shared
- Java manages all the different loggers being used
- The logging level determines the severity of the message
- Different handlers send messages to the console or to files

25. Logging Example
create logger
log to various
levels
easy
Exception
handling

26. Logging Levels
• Log Levels, in order of highest to lowest:
- SEVERE – serious failure, horrible stuff
- WARNING – potential problem
- INFO – informational, non-­‐technical
- CONFIG – system/program configuration
- FINE – simplest trace messages
- FINER – medium trace messages
- FINEST – detailed trace messages

27. Interpreting Levels
• What level you use for your messages is a matter of personal
preference. Here’s a general guideline/suggestion:
- Showstopping issues go to SEVERE
- Errors your admin (and nosy users) should see go to WARNING
- Unobtrusive “heartbeat” kinds of messages go to INFO
- Start and stop messages go to CONFIG
- Debug info goes to FINE/ER/EST, with FINEST being very tedious
and verbose
• These are “trace” levels
• Generally only enabled for troubleshooting purposes

28. Log Handlers
• Typical log handlers write to either the console or a file
• There are default handlers for different contexts
- Java agents: Java console (client), server console (server)
- XPages: server console (server), files (client and server)
- Plugins: OSGi console, files
• Handlers also have levels
- Console normally only displays log messages for SEVERE,
WARNING, and INFO
- File logs often display log messages for all levels

29. Logging Example
where did
this one go?

30. Log Levels and Output
• Loggers often have a default level of INFO
• This means that by default, any messages with a level lower
than INFO will not be logged anywhere
- So messages at level CONFIG and FINE/ER/EST will never be sent
to a Handler because the Logger discards them
• Messages get filtered by the Logger before they are passed to
the Handler
- Even a Handler at Level.FINEST will not get any FINE/ER/EST
messages if the Logger is set to Level.INFO

31. Adjusting Log Levels
• Adjusting log levels programmatically
- setLevel() for Logger and Handler
- This will cause a SecurityException!
- You can get around this by editing jvm/lib/security/java.policy
- permission java.util.logging.LoggingPermission "control"
• Adjusting log levels with config files
- Notes agents: {notes}jvmliblogging.properties
- XPages: {domino}datadominoworkspace.configrcpinstall.properties

32. Example Log Level Adjustment
Don’t forget
the .level!

33. Logging Quirks in Notes
• Prior to 8.5.3 FP2, Java logging in Notes worked just fine
- Although logging in local agents didn’t log anywhere by default
• In 9.0.0 and 8.5.3 FP2, FP3, and FP4 Java logging gave errors
- java.security.AccessControlException
- Add LoggingPermission to jvm/lib/security/java.policy
• In 9.0.1 and 8.5.3 FP5, Java logging worked again
- NEW: Local Java agents log to the Java Agent Console by default
- NEW: if you still need the LoggingPermission so you can set
custom Handlers or Levels, you also need to add a special
SecurityPermission
• http://www-­‐01.ibm.com/support/docview.wss?uid=swg21669594

34. XPages Log Output
• XPages logging trace log files
- Server 8.5.1: none, server console only
- Server 8.5.2+: {domino}datadominoworkspacelogs
• SEVERE messages still show up on the console too
- XPiNC: {notes}dataworkspacelogs
• Default logger level is WARNING, not INFO
• Prior to 8.5.3, non-­‐logged XPages exceptions also used to
show up in the trace logs. Now they are in
{notes}dataIBM_TECHNICAL_SUPPORTxpages_exc_*.log
- http://www-­‐01.ibm.com/support/docview.wss?uid=swg1LO66802

35. Viewing Trace Log Files
• Trying to view the trace log files as raw XML is… painful
• A better way is to right-­‐click them in the “logs” folder and
open them in a browser
- There is an XSLT file that will format them nicely for you
• HOWEVER, if the client or server is still running, the XSLT file
won’t work because the closing XML tag is missing
- Make a copy, open in a text editor, and add a closing
</CommonBaseEvents> tag to the end. Then open in a browser.
- Make sure you open the copy from within the “logs” folder;
otherwise the XSLT file will not be found
- Sometimes log messages don’t show up immediately in the log
files! Everything gets flushed when you shut down the server.

36. XPages Logging Example

37. XPages Logging Example

38. XPages Logging Example

39. Logging Best Practices
• Use the same logger for your entire package (maybe even
your entire library)
• Most messages should be logged to FINE/ER/EST; don’t
clutter up the console with INFO messages
• Log actions, but not every step of a method or every single
get/set – logging is NOT a replacement for a debugger
• DO NOT EVER log sensitive data
- Personal user information, passwords, system “secrets”
- “No one will ever look at those anyway” is bad security

40. Agenda
• Debugging
- Java Agents
- XPages
• Logging
• Using FindBugs
• Using JConsole and Dump Analysis
• Q & A

41. FindBugs
• FindBugs is a program that uses static analysis to look for
bugs in Java code.
- Current version is 3.x
• Source and Documentation at
- http://findbugs.cs.umd.edu/eclipse
• Latest version of FindBugs CANNOT be installed into DDE
natively
- Either install in a stand alone version of Eclipse
- Use FindBugs for Domino Designer from OpenNTF
- FYI, This is a version 2.x implementation of FindBugs
• http://www.openntf.org/main.nsf/project.xsp?r=project/FindBugs
%20for%20DominoDesigner

42. FindBugs
• Available as an Eclipse plug-­‐in, install instructions at
- http://findbugs.cs.umd.edu/eclipse
• Agents contained in an NSF are not “readable” during the
FindBugs analysis.
- The agent code will need to be exported and imported into Eclipse
for analysis.

43. Change FindBugs Properties
• Change the FindBugs Properties to show all bugs

44. Running FindBugs
• In Domino Designer open the Package Explorer view
- If this view is not in the current perspective, either
• Switch to the XPages perspective
• Add it to the current working perspective
• Right click over the code folder to be analyzed and choose
“Run FindBugs”

45. FindBugs Perspective
• After analyzing code, switch to the FindBugs perspective
• Or add the Bug Explorer view to the current perspective
• Remediate the found bugs

46. Demo

47. Agenda
• Debugging
- Java Agents
- XPages
• Logging
• Using FindBugs
• Using JConsole and Dump Analysis
• Q & A

48. My Notes Client is Slow!
• Troubleshooting slow performance is tricky
- How do you define “slow”?
- How do you know it’s your code and not something else?
- Task Manager only tells you about high-­‐level processes
• Much easier to troubleshoot on a client, rather than a server
- Not disruptive to restart
- Easy to change parameters
- Crash at will!
• Whenever possible, port your Java code to a local Notes
agent or a local XPage for troubleshooting

49. A Tale of Two VMs
• The Notes client actually has two Java VMs for running code
- Java agents
- The rest of the Notes client (including local XPages)
• The VM for Java agents is adjusted with:
- Notes.ini settings
- {notes}jvmlib properties files
• The VM for everything else (on the client) is adjusted with:
- {notes}frameworkrcpdeployjvm.properties
- {data}workspace.configrcpinstall.properties

50. JConsole
• JConsole is a tool for monitoring performance and resource
usage in a running JVM
- http://docs.oracle.com/javase/6/docs/technotes/guides/
management/jconsole.html
• I wrote an article on setting this up for local XPages and
plugins (non-­‐Java-­‐agents) here:
- http://www.socialbizug.org/blogs/2ec5d0ed-­‐
d04e-­‐4b18-­‐9610-­‐9819fcebca79/entry/
using_jconsole_to_monitor_your_ibm_notes_client
• Quick demo…

51. JConsole Screenshot

52. JConsole for Local Java Agents
• To use JConsole to monitor local Java agents, there’s a trick
- Add JavaUserOptionsFile to your Notes.ini file:
• JavaUserOptionsFile=c:IBMNotesjava.options.txt
- In your Java options file add these lines:
• -­‐Dcom.sun.management.jmxremote.port=9876
-­‐Dcom.sun.management.jmxremote.ssl=false
-­‐Dcom.sun.management.jmxremote.authenticate=false
• Connect JConsole to the jmxremote.port you defined
- This port won’t be active until AFTER you run a Java agent
- Make sure nothing else is using that port
- Make sure the firewall allows that port to be used

53. JavaUserOptionsFile

54. Analyzing Dump Files
• If your Notes client (or Domino server) crashes due to a Java
problem, it will usually generate a core dump file
- default location is workspacelogs
- Just a text file
- Shows what threads were running, system info, etc.
• IBM Thread and Memory Dump Analyzer does a good job of
parsing the core dump file for you
- https://www.ibm.com/developerworks/community/groups/
service/html/communityview?communityUuid=2245aa39-­‐
fa5c-­‐4475-­‐b891-­‐14c205f7333c
- run like this: java -Xmx500m -jar jca455.jar

55. IBM Memory Dump Analyzer

56. Heap Dumps
• Heap Dumps are a snapshot of the JVM threads and
processes at a specific point in time, while it’s running
- Like a slice of what you see in JConsole
• Used for:
- Tracking down memory leaks
- Finding high-­‐memory-­‐use objects (and arrays)
- Finding objects that are unexpectedly still in memory
• IBM Heap Analyzer is a nice tool for viewing the information
- https://www.ibm.com/developerworks/community/alphaworks/tech/
heapanalyzer
- http://www-­‐01.ibm.com/support/docview.wss?uid=swg27006624&aid=1

57. How to Generate a Heap Dump
• Generating a Heap Dump file (no crash required!)
- Domino XPages
• tell http xsp heap dump
• XPages Toolbox ( http://www.openntf.org/p/XPages%20Toolbox )
- Notes Client (from a command prompt)
• notesframeworkrcprcplauncher.exe -­‐com.ibm.rcp.core.logger#dump heap -­‐dumps heapdump
• writes to {data}workspacelogsheapdump.###.phd by default
• You can also do a core (thread) dump with:
• notesframeworkrcprcplauncher.exe -­‐com.ibm.rcp.core.logger#dump threads -­‐dumps javacore
- Java Agent
• com.ibm.jvm.Dump.HeapDump();
• writes to {notes}frameworkheapdump.###.phd

58. IBM Heap Analyzer Screenshot

59. IBM Heap Analyzer Screenshot

60. IBM Heap Analyzer Screenshot
2318 Notes Documents
2318 Notes Items
1 Notes View
1 Notes Database
1 Notes Session
1 ???
———————————
4640 NotesWeakReferences

61. Agenda
• Debugging
- Java Agents
- XPages
• Logging
• Using FindBugs
• Using JConsole and Dump Analysis
• Q & A

62. Q & A
Questions?
Anyone?

63. THANK YOU
Julian Robichaux
panagenda
jrobichaux@panagenda.com
MWLUG 2014
Paul Calhoun
panagenda
paul.calhoun@panagenda.com