Profilers are absolute beasts. And profilers might prove useful to pinpoint the performance issues in your Java applications. By using profilers, developers are fortunate to find the root cause of an issue at hand. However, it requires effort to actually comprehend the data collected by the profiler. Due to the inherent complexity of the data, one has to understand how this data is collected. And thus understand how the profiler actually works. During this talk we will go through the classic profiler features. What is a hotspot? What is the difference between sampling and instrumentation from the profiler point of view? What are the problems with either of those methods? What is the time budget of the application? And more! I will also showcase a new kid on the block among the profiling tools: XRebel. This tool provides insight into application behavior and permits the developers to discover application level issues

1. Is your profiler speaking the
same language as you?
Simon Maple
@sjmaple

2. @sjmapleSimon Maple -

3. Agenda
• Performance Tools
• Performance by numbers
• Sampling vs Tracing
• XRebel
3

5. Performance Tools
• Java Monitoring Tools
• Java Profilers
• Java Testing Tools
5

6. Performance report
6

7. RebelLabs reports
7

8. Who are you?
8

9. Your Application
9

10. When do you test?
10

11. Tools
11

12. Finding bugs Exist
12

13. Symptoms
13

14. Root Causes
14

15. Do you affect Users?
15

16. Who are the heroes?
16

17. Profiling Frequency
17

18. How long is your app
release cycle?
18

19. How much time do you
spend testing?
19

20. Did it work?
20

21. The Journey of Performance
21

22. https://twitter.com/shipilev/status/578193813946134529

23. A
B
C
D E
Performance
Code complexity

24. Improving
Optimizing
Last 5%
Last 1%
A
B
C
D E

25. Profiling
CPU
Tracing Sampling
Memory
Usage Allocation
We are only talking about this part today

26. JVM has ability to produce thread dump:
Press Ctrl+Break on Windows
kill -3 PID on *nix
"http-nio-8080-exec-1"#40 daemon prio=5 os_prio=31 tid=0x00007fd7057ed800 nid=0x7313 runnable
java.lang.Thread.State: RUNNABLE
at o.s.s.p.w.OwnerController.processFindForm(OwnerController.java:89)
at s.r.NativeMethodAccessorImpl.invoke0(Native Method)
at s.r.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at s.r.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at j.l.r.Method.invoke(Method.java:497)
What is a sample?

27. Sampling interval : 20 ms
Sample count: 4

28. Sampling interval : 1 ms
Sample count: 100+

29. Sampling
main()
foo()
bar()
baz()
never captured :(

30. Safepoint bias
Safepoints
main()
foo()
bar()
baz()

31. Safepoint bias
Safepoints
main()
foo()
bar()
baz()

32. Taming safepoint bias
Java Mission Control
Proprietary protocol
Java 7u40
Honest Profiler
https://github.com/RichardWarburton/honest-profiler
AsyncGetCallTrace
JVMTI
NB! not documented

33. java.util is hot? :)

34. java.util is hot? :)

35. public
void
businessMethod()
{
long
start
=
System.currentTimeMillis();
work();
Profiler.log(System.currentTimeMillis()-­‐start);
}
Tracing
(Instrumentation)

36. Tracing
public
void
businessMethod()
{
long
start
=
System.nanoTime();
work();
Profiler.log(System.nanoTime()-­‐start);
}

37. Tracing
public
void
businessMethod()
{
Profiler.start(“businessMethod");
try
{
work();
}
finally
{
Profiler.log("businessMethod");
}
}

38. System.nanoTime
System.currentTimeMillis
Parallel thread updating easily accessible
memory location
sleep-wakeup-update
yield-wakeup-update
busy-loop-update

39. class
Profiler
{
Loop
loop;
public
static
void
start(String
method)
{
long
now
=
loop.getTime();
…
}

40. public
class
Loop
implements
Runnable
{
private
volatile
long
time;
public
void
run()
{
while
(running)
{
time
=
System.nanoTime();
sleep();
}
}
public
final
long
getTime()
{
return
time;
}
Busy Loop

41. Nano seconds put into perspective
Reading memory is not free.
It takes cycles = nanoseconds
Each (software) layer is not free.
JVM,JNI,OS,HW
http://shipilev.net/blog/2014/nanotrusting-nanotime/
Nanotrusting the NanoTime

42. Nano seconds put into perspective
http://shipilev.net/blog/2014/nanotrusting-nanotime/
Nanotrusting the NanoTime
granularity_nanotime: 26.300 +-0.205 ns
latency_nanotime: 25.542 +-0.024 ns
granularity_nanotime: 29.322 +-1.293 ns
latency_nanotime: 29.910 +-1.626 ns
granularity_nanotime: 371,419 +-1,541 ns
latency_nanotime: 14,415 +-0,389 ns
Linux
Solaris
Windows

43. Sleep timer: time-slicing & scheduling
Minimum sleep times:
Win8: 1.7 ms (1.0 ms using JNI + socket poll)
Linux: 0.1 ms
OS X: 0.05 ms
VirtualBox + Linux: don’t ask :)
Still uses 25-50% of CPU core
Yield?
Windows scheduler skips yielded threads in case of
CPU starvation

44. public
class
Loop
implements
Runnable
{
private
volatile
long
time;
public
void
run()
{
while
(running)
{
time
=
System.nanoTime();
sleep();
}
}
private
void
sleep()
{
if
(!MiscUtil.isWindows())
{
Thread.yield();
}
}
Busy Loop

45. public
class
Loop
implements
Runnable
{
private
volatile
long
time;
public
void
run()
{
while
(running)
{
time
=
System.nanoTime();
sleep();
}
}
private
void
sleep()
{
if
(!MiscUtil.isWindows())
{
Thread.yield();
}
}
Busy Loop

46. Busy Loop
public
class
Loop
implements
Runnable
{
private
volatile
long
time;
public
void
run()
{
while
(running)
{
time
=
System.nanoTime();
sleep();
}
}
private
void
sleep()
{
if
(!MiscUtil.isWindows())
{
Thread.yield();
}
}

47. public
class
Loop
implements
Runnable
{
private
volatile
long
time;
public
void
run()
{
while
(running)
{
time
=
System.nanoTime();
sleep();
}
}
private
void
sleep()
{
if
(!MiscUtil.isWindows())
{
Thread.yield();
}
}
Busy Loop

48. main()
foo()
bar()
baz()
Tracing
relatively higher overhead for fast methods :(

49. Tracing
main()
foo()
bar()
baz()
2
3
4

50. Tracing
main()
foo()
bar()
baz()
2
3
5
boo()
Balance shift due to tracing overhead

53. Database access
Web Services
RMI
Various application layers
3rd-party components
HTTP session
Exceptions
Caches
File system access
View rendering
Serialization
GC
What other performance
considerations should we have?

54. Questions you should be asking
……..
……..
……..
……..

55. Questions you should be asking
Where was most of the time spent?
How many SQL queries were executed?
How much time did the external calls take?
What’s my HTTP session state?
Is caching working properly?
Is there any excessive work done by the app?

56. Most improvement gained here!
This is where we should focus first!
A
B
C
D E