This document discusses various tools for diagnosing performance problems in Java applications running on Linux. It introduces the concept of "The Box" which represents all of the components that can affect application performance, including traffic, code, JVM, OS, virtualization, and hardware. It emphasizes the importance of understanding how the application is used, its code and algorithms, JVM configuration and garbage collection, OS limits and configuration, and underlying hardware. It also recommends using tools like Linux perf and following Brendan Gregg's USE method of analyzing utilization, saturation, and errors of key resources to help identify bottlenecks. The key takeaway is that GNU/Linux has tools that can help determine what is impacting performance at each level from traffic

1. Wprowadzenie do
strojenia Javy na
GNU/Linux dla zajętego
programisty
Tomasz Borek, Symentis

2. • Infected in childhood
• From me parents
• Amstrad, ElWro Junior
• Games! Doh!
• Mem pages in DOS anyone?
• In IT
• ETL, crawlers, web-app, archi
• IAAS, SAAS, own servers
• Java 4 – 7, GNU/Linux, SQLs
• Ardent “activist”
Tomasz Borek

3. Can be found in the net! :P
https://lafkblogs.wordpress.com/
https://twitter.com/lafk_pl
https://pl.linkedin.com/in/tjborek
GeeCON, ChamberConf, Confitura, Lambda Days, LambdaCon,
Java Developer Days, JavaDay Lviv, JavaDay Kiev

4. Tomek in IT groups
.com
http://java.pl Kraków
Poznań
Praga
Sopot
http://geecon.org

5. Prod hits rock bottom
● Everybody is kung-fu fighting
● Must be quick like lightning

6. So! Being a dev, you need:
1.To know your traffic and your app and your infra

7. So! Being a dev, you need:
1.To know your traffic and your app and your infra
2.To know recent changes

8. So! Being a dev, you need:
1.To know your traffic and your app and your infra
2.To know recent changes
3.To know your limits (sometimes comes from point 1)

9. So! Being a dev, you need:
1.To know your traffic and your app and your infra
2.To know recent changes
3.To know your limits (sometimes comes from point 1)
4.To have access to monitoring

10. So! Being a dev, you need:
1.To know your traffic and your app and your infra
2.To know recent changes
3.To know your limits (sometimes comes from point 1)
4.To have access to monitoring
5.To adjust pooling (both thread and connection)

11. So! Being a dev, you need:
1.To know your traffic and your app and your infra
2.To know recent changes
3.To know your limits (sometimes comes from point 1)
4.To have access to monitoring
5.To adjust pooling (both thread and connection)
6.To peruse logs and pull out anomalies

12. So! Being a dev, you need:
1.To know your traffic and your app and your infra
2.To know recent changes
3.To know your limits (sometimes comes from point 1)
4.To have access to monitoring
5.To adjust pooling (both thread and connection)
6.To peruse logs and pull out anomalies
7.TO TUNE JVM

13. So! Being a dev, you need:
1.To know your traffic and your app and your infra
2.To know recent changes
3.To know your limits (sometimes comes from point 1)
4.To have access to monitoring
5.To adjust pooling (both thread and connection)
6.To peruse logs and pull out anomalies
7.TO TUNE JVM

14. JVM? Meaning?
• Sun Hotspot
• IBM
• JRockit
• IceTea
• OpenJDK
• TCK decides

15. JVM? Meaning?
• Sun Hotspot
• IBM
• JRockit
• IceTea
• OpenJDK
• TCK decides

16. JVM? Meaning?
• Sun Hotspot – but which version?
• IBM
• JRockit
• IceTea
• OpenJDK
• TCK decides

17. JVM? Meaning?
• Sun Hotspot – but which version? And architecture?
• IBM
• JRockit
• IceTea
• OpenJDK
• TCK decides

18. So! JVM tuning?
1.To adjust GC, based on it's logs

19. So! JVM tuning?
1.To adjust GC, based on it's logs
2.To adjust (native) heap

20. So! JVM tuning?
1.To adjust GC, based on it's logs
2.To adjust (native) heap
3.Tweaking JVM starting flags

21. So! JVM tuning?
1.To adjust GC, based on it's logs
2.To adjust (native) heap
3.Tweaking JVM starting flags
4.To use TOOLS

22. Today!
● JVM tuning
● Diagnosing performance problems
● Tools
● All in a rush

23. JVM tuning

24. Takeaway #1
JVM is a process

25. Being a process means
● OS architecture enforces JVM architecture
● Your memory is limited by OS and other processes
– Heap is one
– C-heap is another!
● IO-wise and thread-wise:
– Your threads and files opened are limited by ulimit
● File descriptors!

26. OOM flavours?

27. OOM flavours
● Out of HeapSpace

28. OOM flavours
● Out of HeapSpace
● PermGen error

29. OOM flavours
● Out of HeapSpace
● PermGen error
● Unable to create native thread

30. OOM flavours
● Out of HeapSpace
● PermGen error
● Unable to create native thread
● There are others, these are most popular

31. API says OutOfMemoryError is
● Thrown when the Java Virtual Machine cannot
allocate an object because it is out of memory, and
no more memory could be made available by the
garbage collector. OutOfMemoryError objects may
be constructed by the virtual machine as if
suppression were disabled and/or the stack trace
was not writable.

32. API says OutOfMemoryError is
● Thrown when the Java Virtual Machine cannot
allocate an object because it is out of memory,
and no more memory could be made available
by the garbage collector. OutOfMemoryError
objects may be constructed by the virtual machine
as if suppression were disabled and/or the stack
trace was not writable.

33. API says OutOfMemoryError is
● Due to failure in allocating an object after garbage
collection.
● Nothing about threads.
● Nothing about PermGen.

34. How much memory JVM uses?
● Say you start it with Xmx == Xms == 1GB

35. How much memory JVM uses?
● Say you start it with Xmx == Xms == 1GB
● 1GB?

36. How much memory JVM uses?
● Say you start it with Xmx == Xms == 1GB
● 1GB?
● Around 1GB?

37. How much memory JVM uses?
● Say you start it with Xmx == Xms == 1GB
● 1GB?
● Around 1GB?
● If we talk about Java HEAP

38. Memory management
Entire RAM
● 32 bits: can address less than 4GB.
● 64 bits: can address… theoretically 16 EB.
– Still – much more!
● There's much more to 32vs64 but that's for another time

39. Memory management
Entire RAM
What's left of RAM – C-HEAPJVM heap
-Xmx
● -Xms – minimal heap size
● -Xmx – maximal heap size
● When both should be set to same value?

40. Memory management
Entire RAM
What's left of RAM – C-HEAPJVM heap
What's left – C-HEAPJVM opsJVM heap
-Xmx
PermGen

41. PermGen
● Permanent Generation
– Early Java
– for JDK classes
– and Strings
● Back then – good idea
● Now removed
– JDK 7 – Strings no longer in PermGen
– JDK 8 – no PermGen at all

42. PermGen
● Permanent Generation
– Early Java
– for JDK classes
– and Strings
● Back then – good idea
● Now removed
– JDK 7 – Strings no longer in PermGen
– JDK 8 – no PermGen at all
Quick fix?
Increase PermGen size.

43. Memory management
What's left – C-HEAPJVM opsJVM heap
-Xmx
PermGen

44. GC
● When it runs?

45. GC
● When it runs?
● Minor collection?

46. GC
● When it runs?
● Minor collection?
● Major collection?

47. GC
● When it runs?
● Minor collection?
● Major collection?
● How can a dev tune GC?

48. Generational GC
Tenured – where long-lived are promotedEden – infants that die quickly
● Studies showed:
– most objects die young
– Some live really long
● Ergo: short- and long-lived division
● Minor collection: Eden
● Major collection: whole heap

49. Generational GC - mechanics
● In fact Eden has also 2 Survivor spaces
– To handle locality
– Helps compress after freeing space
– First promotion to survivor, then to tenured
● Flags tell:
– How many GCs object must survive to be promoted
– How large Eden / Tenured / Survivors spaces are
– What is logged (how details GC logs are)
– Many, MANY more

50. Memory management trade-offs
● Large heap – large full GC – small native and C-heap?
● Smaller heap – minor GC may be enough
● Make sure your objects die young and only really long
lived reach tenured

51. JVM tuning
1.To adjust GC, based on it's logs
2.To adjust (native) heap
3.Tweaking JVM starting flags
4.To use TOOLS – later

52. Takeaway #2
Log GC

53. Takeaway #3
GC tuning is a trade-off

54. Diagnosing performance problems

55. „The Box”
• Heinz Kabutz, Kirk Pepperdine
• Top - bottom

56. „The Box”
• TRAFFIC: ? TRAFFIC

57. „The Box”
• TRAFFIC: how is app used? TRAFFIC: people,
automatic

58. „The Box”
• TRAFFIC: how is app used?
• CODE: ?
TRAFFIC: people,
automatic
Application CODE

59. „The Box”
• TRAFFIC: how is app used?
• CODE: threads, algos, data structures...
TRAFFIC: people,
automatic
CODE: threads, data
structs, algo

60. „The Box”
• TRAFFIC: how is app used?
• CODE: threads, algos, data structures...
• JVM: ?
TRAFFIC: people,
automatic
CODE: threads, data
structs, algo
JVM

61. „The Box”
• TRAFFIC: how is app used?
• CODE: threads, algos, data structures...
• JVM: ?
TRAFFIC: people,
automatic
CODE: threads, data
structs, algo
JVM: flags, GC

62. „The Box”
• TRAFFIC: how is app used?
• CODE: threads, algos, data structures...
• JVM: starting flags, GC
• OS: ?
TRAFFIC: people,
automatic
CODE: threads, data
structs, algo
JVM: flags, GC
OS

63. „The Box”
• TRAFFIC: how is app used?
• CODE: threads, algos, data structures...
• JVM: starting flags, GC
• OS: ulimit, FS, config, archi, other procs…
TRAFFIC: people,
automatic
CODE: threads, data
structs, algo
JVM: flags, GC
OS: FS, config, limits

64. „The Box”
• TRAFFIC: how is app used?
• CODE: threads, algos, data structures...
• JVM: starting flags, GC
• OS: ulimit, FS, config, archi, other procs…
• VIRT: ?
TRAFFIC: people,
automatic
CODE: threads, data
structs, algo
JVM: flags, GC
OS: FS, config, limits
VIRT

65. „The Box”
• TRAFFIC: how is app used?
• CODE: threads, algos, data structures...
• JVM: starting flags, GC
• OS: ulimit, FS, config, archi, other procs…
• VIRT: hell depends!
TRAFFIC: people,
automatic
CODE: threads, data
structs, algo
JVM: flags, GC
OS: FS, config, limits
VIRT

66. „The Box”
• TRAFFIC: how is app used?
• CODE: threads, algos, data structures...
• JVM: starting flags, GC
• OS: ulimit, FS, config, archi, other procs…
• VIRT: hell depends!
• HARDWARE: ?
TRAFFIC: people,
automatic
CODE: threads, data
structs, algo
JVM: flags, GC
OS: FS, config, limits
VIRT
HARDWARE

67. „The Box”
• TRAFFIC: how is app used?
• CODE: threads, algos, data structures...
• JVM: starting flags, GC
• OS: ulimit, FS, config, archi, other procs…
• VIRT: hell depends!
• HARDWARE: 32 vs 64, RAM, BIOS, drivers…
TRAFFIC: people,
automatic
CODE: threads, data
structs, algo
JVM: flags, GC
OS: FS, config, limits
VIRT
HARDWARE

68. Brendan Gregg – USE
• Utilization, Saturation, Errors
• Use – how much in use is your resource
• Saturation – how many requests await (waiting queue)
• Errors – what errors are thrown by resource
• http://www.brendangregg.com/usemethod.html
• Create a checklist for each important resource, for
finding out Utilization, Saturation and Errors and you'll
know how to soon find out what is going on with
resource

69. Takeaway #1
The Box

70. Takeaway #2
GNU/Linux perf? Brendan Gregg

71. TOOLS

74. Takeaway #1
GNU/Linux surely has a tool for that

75. How to find your java process?
● ps … | grep java
●

76. How to find your java process?
● ps … | grep java
● pgrep java

77. How to find your java process?
● ps … | grep java
● pgrep java
● jps

78. How to change flags on-the-fly?
● jinfo

79. How to dump threads or heap?
● kill -3

80. How to dump threads or heap?
● kill -3
● jstack

81. How to dump threads or heap?
● kill -3
● jstack
● jhat (heap)

82. jvisualvm
● Deserves it's own slide
● Profiler, sampler
● Monitor (heap, threads, etc.)
● Calls GC, does dumps

83. SUMMARIZING
● For being rushed – prepare in advance
– Monitoring, logs, ceilings, etc.
● Log GC
● JVM is a process – all process limits in your OS apply
● The Box
● Brendan Gregg
● GNU/Linux – tools unparalleled

84. ?