Learn right tools, tricks and patterns to identify root cause of complex java problems in seconds (not even in minutes). Here is the presentation of Modern Engineer’s Troubleshooting Tools, Techniques & Tricks delivered at Confoo 2018

1. Modern Engineer’s
troubleshooting tools, techniques
& tricks
Ram Lakshmanan
Architect – GCeasy.io & fastThread.io
Confoo - 2018

2. Problems
1. Functional
2. Non-Functional

3. What types of non-functional Problems you have
faced?

4. Have you experienced sudden CPU spikes?
• This Problem happened in major B2B Travel app

5. jstack -l <pid> > /tmp/threadDump.txt
jcmd <pid> Thread.print > /tmp/threadDump.txt
Jstack (since Java 5) kill -3 jVisualVM
JMC Windows (Ctrl + Break) ThreadMXBean
APM Tools Jcmd (since Java 7)
1. 2. 3.
4. 5. 6.
7. 8.
How to capture thread dumps? – 8 options
https://blog.fastthread.io/2016/06/06/how-to-take-thread-dumps-7-options/

6. 1. Infinite looping
while (aCondition()) {
doSomething();
}
1
Diagnose sudden CPU spikes?
2

7. Thread Mill Pattern
https://blog.fastthread.io/2016/03/16/thread-dump-analysis-pattern-thread-mill/

8. Other Patterns
• ATHLETE
• ATHEROSCLEROSIS
• TRAFFIC JAM
• ALL ROADS LEAD TO ROME
• LEPRECHAUN TRAP
• SEVERAL SCAVENGERS
• STOCK TICKER
• More Patterns - https://blog.fastthread.io/category/thread-dump-
patterns/

9. Have you experienced unresponsive DB/NoSQL?
• Demo - Real Problem happened in a major Banking application.

10. RSI Pattern
https://blog.fastthread.io/2016/02/22/thread-dump-analysis-pattern-repetitive-strain-injury-rsi/

11. Thread dump analyzers
• fastThread.io
• IBM Thread and Monitor Dump Analyzer
• Samurai
• TDA

12. 2. Consecutive Full GCs
2
Diagnose sudden CPU spikes
1. Infinite looping
while (aCondition()) {
doSomething();
}

13. 2. Consecutive Full GCs
Diagnose sudden CPU spikes
1. Infinite looping
while (aCondition()) {
doSomething();
}

14. Welcome to the world of Garbage
Collection
Garbage Collection is automatic. But not free.

15. How to enable Garbage Collection Log?
Till Java 8:
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<file-path>
From Java 9:
-Xlog:gc*:file=<file-path>

16. Demo - healthy application’s GC Log
• http://gceasy.io/my-gc-
report.jsp?p=c2hhcmVkLzIwMTgvMDMvOC8tLTAuZmlyc3RHQy5sb2cu
emlwLS0xOC0xMC00

17. Demo - sick application’s GC Log
• http://gceasy.io/diamondgc-report.jsp?oTxnId_value=97fff4aa-b22f-
424e-b112-fab579369ccb

18. How to reduce long GC Pause?
• 7 strategies.
<<Will be discussed in detail in tomorrow’s 2pm session “Become a Garbage
Collection Hero”>>
• First & foremost strategy: Reduce memory consumption

19. Welcome to the world of Heap dump
analysis

20. Demo - Let’s analyze Heap Dump

21. Have you experienced
OutOfMemoryError?

22. Types of OutOfMemoryErrors?
1. Java Heap Space
2. GC overhead limit exceeded
3. Requested array size exceeds VM limit
4. Permgen space
5. Metaspace
6. Unable to create new native thread
7. Kill process or sacrifice child
8. Reason stack_trace_with_native_method

23. OutOfMemoryError: Unable to create new native
thread?
Java Heap
Physical memory4 GB (-Xms, -Xmx)
8 GB
Java Heap
Physical memory4 GB (-Xms, -Xmx)
8 GB
Process-1 Process-2
OutOfMemo
ryError
Key point: Threads are created outside heap
threads
Solution:
1. Fix thread leak
2. Increase the Thread Limits Set at
Operating System(ulimit –u)
3. Reduce Java Heap Size
4. Kills other processes
5. Increase physical memory size
6. Reduce thread stack size (-Xss).
Note: can cause StackOverflowError

24. jmap -dump:live,file=<file-path> <pid>
jmap System Property jVisualVM
JMX IBM Administrative Consol Program
1. 2. 3.
4. 5. 6.
7.
How to capture Heap dumps? – 7 options
https://blog.fastthread.io/2016/06/06/how-to-take-thread-dumps-7-options/
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=<file-path>
jcmd <pid> GC.heap_dump <file-path>
Jcmd (since Java 7)

25. Heap Dump analyzers
• HeapHero.io
• Eclipse MAT
• jHat

26. Small gift for you!
Beautiful 1 page document on OutOfMemoryErrors:
https://bloggceasy.files.wordpress.com/2015/05/outofmemoryerror2.pdf

27. Thank you!!
On-site Training – for Developers
QA Engineers, DevOps
ram@tier1app.comhttps://www.linkedin.com/in/ramlakshman
Services Tools
Intelligent Thread dump Analyzer
Universal Garbage Collection log analyzer
Powerful Heap dump Analyzer