So You Want To Write Your Own Benchmark
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

So You Want To Write Your Own Benchmark

  • 8,590 views
Uploaded on

Performance has always been a major concern in software development and should not be taken lightly even when commodity computers have multicore CPUs and a few gigabytes of RAM. One of the most......

Performance has always been a major concern in software development and should not be taken lightly even when commodity computers have multicore CPUs and a few gigabytes of RAM. One of the most handy, simple tools for performance testing are microbenchmarks. Unfortunately, developing correct Java microbenchmarks is a complex task with many pitfalls on the way. This presentation is about the Do's and Don'ts of Java microbenchmarking and about what tools are out there to help with this tricky task.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
8,590
On Slideshare
8,501
From Embeds
89
Number of Embeds
13

Actions

Shares
Downloads
138
Comments
0
Likes
8

Embeds 89

http://drorbr.blogspot.com 53
http://www.slideshare.net 21
http://www.slideee.com 2
http://drorbr.blogspot.be 2
http://drorbr.blogspot.co.il 2
http://feeds.feedburner.com 2
http://drorbr.blogspot.com.es 1
http://drorbr.blogspot.co.uk 1
http://www.linkedin.com 1
http://drorbr.blogspot.fr 1
http://www.techgig.com 1
http://drorbr.blogspot.in 1
https://www.linkedin.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. yo ur y r ite w ark tsk to m zni er e a nt ch Dr or B w en ou rob y ic So m n ow 8t h2 008 er 1 mb Dece
  • 2. Agenda • Introduction • Java™ micro benchmarking pitfalls • Writing your own benchmark • Micro benchmarking tools • Summary 2
  • 3. Microbenchmark – simple definition 1. Start the 2. Run the code 3. Stop the 4. Report clock clock 3
  • 4. Better microbenchmark definition • Small program • Goal: Measure something about a few lines of code • All other variables should be removed • Returns some kind of a numeric result 4
  • 5. Why do I need microbenchmarks? • Discover something about my code: • How fast is it • Calculate throughput – TPS, KB/s • Measure the result of changing my code: • Should I replace a HashMap with a TreeMap? • What is the cost of synchronizing a method? 5
  • 6. Why are you talking about this? • It’s hard to write a robust microbenchmark • it’s even harder to do it in Java™ • There are not enough Java microbenchmarking tools • There are too many flawed microbenchmarks out there 6
  • 7. Agenda • Introduction • Java micro benchmarking pitfalls • Writing your own benchmark • Micro benchmarking tools • Summary 7
  • 8. A microbenchmark story: the problem The boss asks you to solve a performance issue in one of the components Blah, blah … 8
  • 9. A microbenchmark story: the cause You find out that the cause is excessive use of Math.sqrt() 9
  • 10. A microbenchmark story: a solution? • You decide to develop a state of the art square root approximation • After developing the square root approximation you want to benchmark it against the java.lang.Math implementation 10
  • 11. SQRT approximation microbenchmark Let’s run this little piece of code in a loop and see what happens … public static void main(String[] args) { long start = System.currentTimeMillis(); // start the clock for (double i = 0; i < 10 * 1000 * 1000; i++) { mySqrt(i); // little piece of code } long end = System.currentTimeMillis(); // stop the clock long duration = end - start; System.out.format(quot;Test duration: %d (ms) %nquot;, duration); } 11
  • 12. SQRT microbenchmark results Wow, this is really fast ! Test duration: 0 (ms) 12
  • 13. Flawed microbenchmark 13
  • 14. SQRT microbenchmark: what’s wrong? Dynamic optimizations Garbage collection Dead code elimination The Java™ HotSpot virtual machine Classloading Dynamic Compilation On Stack Replacement 14
  • 15. The HotSpot: a mixed mode system 2 Code is 1 interpreted Profiling 3 Interpreted again Dynamic or recompiled Compilation 5 Stuff 4 Happen 15
  • 16. Dynamic compilation • Dynamic compilation is unpredictable • Don’t know when the compiler will run • Don’t know how long the compiler will run • Same code may be compiled more than once • The JVM can switch to compiled code at will 16
  • 17. Dynamic compilation cont. • Dynamic compilation can seriously influence microbenchmark results Continuous recompilation Steady-state Interpreted execution + Compiled / Interpreted code Dynamic compilation + ≠ execution Compiled code execution 17
  • 18. Dynamic optimizations • The HotSpot server compiler performs large variety of optimizations: • loop unrolling • range check elimination • dead-code elimination • code hoisting … 18
  • 19. Code hoisting ? Did he just said “code hoisting”? 19
  • 20. What the heck is code hoisting ? • Hoist = to raise or lift • Size optimization • Eliminate duplicated pieces of code in method bodies by hoisting expressions or statements 20
  • 21. Code hoisting example a + b is a busy After hoisting the expression expression a + b. A new local variable t has been introduced Optimizing Java for Size: Compiler Techniques for Code Compaction, Samuli Heilala 21
  • 22. Dynamic optimizations cont. • Most of the optimizations are performed at runtime • Profiling data is used by the compiler to improve optimization decisions • You don’t have access to the dynamically compiled code 22
  • 23. Example: Very fast square root? 10,000,000 calls to Math.sqrt() ~ 4 ms public static void main(String[] args) { long start = System.nanoTime(); int result = 0; for (int i = 0; i < 10 * 1000 * 1000; i++) { result += Math.sqrt(i); } long duration = (System.nanoTime() - start) / 1000000; System.out.format(quot;Test duration: %d (ms) %nquot;, duration); } 23
  • 24. Example: not so fast? Now it takes ~ 2000 ms ?!? public static void main(String[] args) { long start = System.nanoTime(); int result = 0; for (int i = 0; i < 10 * 1000 * 1000; i++) { result += Math.sqrt(i); Single line of code } added System.out.format(quot;Result: %d %nquot;, result); long duration = (System.nanoTime() - start) / 1000000; System.out.format(quot;Test duration: %d (ms) %nquot;, duration); } 24
  • 25. DCE - Dead Code Elimination • Dead code - code that has no effect on the outcome of the program execution public static void main(String[] args) { long start = System.nanoTime(); int result = 0; for (int i = 0; i < 10 * 1000 * 1000; i++) { result += Math.sqrt(i); } Dead Code long duration = (System.nanoTime() - start) / 1000000; System.out.format(quot;Test duration: %d (ms) %nquot;, duration); } 25
  • 26. OSR - On Stack Replacement • Methods are HOT if they cumulatively execute more than 10,000 of loop iterations • Older JVM versions did not switch to the compiled version until the method exited and was re-entered • OSR - switch from interpretation to compiled code in the middle of a loop 26
  • 27. OSR and microbenchmarking • OSR’d code may be less performant • Some optimizations are not performed • OSR usually happen when you put everything into one long method • Developers tend to write long main() methods when benchmarking • Real life applications are hopefully divided into more fine grained methods 27
  • 28. Classloading • Classes are usually loaded only when they are first used • Class loading takes time • I/O • Parsing • Verification • May flow your benchmark results 28
  • 29. Garbage Collection • JVM automatically claim resources by • Garbage collection • Objects finalization • Outside of developer’s control • Unpredictable • Should be measured if invoked as a result of the benchmarked code 29
  • 30. Time measurement How long is one millisecond? public static void main(String[] args) throws InterruptedException { long start = System.currentTimeMillis(); Thread.sleep(1); final long end = System.currentTimeMillis(); final long duration = (end - start); System.out.format(quot;Test duration: %d (ms) %nquot;, duration); } Test duration: 16 (ms) 30
  • 31. System.curremtTimeMillis() • Accuracy varies with platform Resolution Platform Source 55 ms Windows 95/98 Java Glossary 10 – 15 ms Windows NT, 2K, XP, 2003 David Holmes 1 ms Mac OS X Java Glossary 1 ms Linux – 2.6 kernel Markus Kobler 31
  • 32. Wrong target platform • Choosing the wrong platform for your microbenchmark • Benchmarking on Windows when your target platform is Linux • Benchmarking a highly threaded application on a single core machine • Benchmarking on a Sun JVM when the target platform is Oracle (BEA) JRockit 32
  • 33. Caching • Caching • Hardware – CPU caching • Operating System – File system caching • Database – query caching 33
  • 34. Caching: CPU L1 and L2 caches • The more the data accessed are far from the CPU, the more the delays are high • Size of dataset affects access cost Array size Time (us) Cost (ns) 16k 413451 9.821 8192K 5743812 136.446 Jcachev2 results for Intel® core™2 duo T8300, L1 = 32 KB, L2 = 3 MB 34
  • 35. Busy environment • Running in a busy environment – CPU, IO, Memory 35
  • 36. Agenda • Introduction • Java micro benchmarking pitfalls • Writing your own benchmark • Micro benchmarking tools • Summary 36
  • 37. Warm-up your code 37
  • 38. Warm-up up your code • Let the JVM reach steady state execution profile before you start benchmarking • All classes should be loaded before benchmarking • Usually executing your code for ~10 seconds should be enough 38
  • 39. Warm-up up your code – cont. • Detect JIT compilations by using • CompilationMXBean. getTotalCompilationTime() • -XX:+PrintCompilation • Measure classloading time • Use the ClassLoadingMXBean 39
  • 40. CompilationMXBean usage import java.lang.management.ManagementFactory; import java.lang.management.CompilationMXBean; long compilationTimeTotal; CompilationMXBean compBean = ManagementFactory.getCompilationMXBean(); if (compBean.isCompilationTimeMonitoringSupported()) compilationTimeTotal = compBean.getTotalCompilationTime(); 40
  • 41. Dynamic optimizations • Avoid on stack replacement • Don’t put all your benchmark code in one big main() method • Avoid dead code elimination • Print the final result • Report unreasonable speedups 41
  • 42. Garbage Collection • Measure garbage collection time • Force garbage collection and finalization before benchmarking • Perform enough iteration to reach garbage collection steady state • Gather gc stats: -XX:PrintGCTimeStamps -XX:PrintGCDetails 42
  • 43. Time measurement • Use System.nanoTime() • Microseconds accuracy on modern operating systems and hardware • Not worse than currentTimeMillis() • Notice: Windows users • executes in microseconds • don’t overuse ! 43
  • 44. JVM configuration • Use similar JVM options to your target environment: • -server or –client JVM • Enough heap space (-Xmx) • Garbage collection options • Thread stack size (-Xss) • JIT compiling options 44
  • 45. Other issues • Use fixed size data sets • Too large data sets can cause L1 cache blowout • Notice system load • Don’t play GTA while benchmarking ! 45
  • 46. Agenda • Introduction • Java micro benchmarking pitfalls • Writing your own benchmark • Micro benchmarking tools • Summary 46
  • 47. Java™ benchmarking tools • Various specialized benchmarks • SPECjAppServer ® • SPECjvm™ • CaffeineMark 3.0™ • SciMark 2.0 • Only a few benchmarking frameworks 47
  • 48. Japex Micro-Benchmark framework • Similar in spirit to JUnit • Measures throughput – work over time • Transactions Per Second (Default) • KBs per second • XML based configuration • XML/HTML reports 48
  • 49. Japex: Drivers • Encapsulates knowledge about a specific algorithm implementation • Must extend JapexDriverBase public interface JapexDriver extends Runnable { public void initializeDriver(); public void prepare(TestCase testCase); public void warmup(TestCase testCase); public void run(TestCase testCase); public void finish(TestCase testCase); public void terminateDriver(); } 49
  • 50. Japex: Writing your own driver public class SqrtNewtonApproxDriver extends JapexDriverBase { private long tmp; … @Override public void warmup(TestCase testCase) { tmp += sqrt(getNextRandomNumber()); } … } 50
  • 51. Japex: Test suite <testSuite name=quot;SQRT Test Suitequot; xmlns=http://www.sun.com/japex/testSuite …> <param name=quot;libraryDirquot; value=quot;C:/java/japex/libquot;/> <param name=quot;japex.classPathquot; value=quot;./target/classesquot;/> <param name=quot;japex.runIterationsquot; value=quot;1000000quot; /> <driver name=quot;SqrtApproxNewtonDriverquot;> <param name=quot;Descriptionquot; value=quot;Newton Driverquot;/> <param name=quot;japex.driverClass“ value=quot;com.alphacsp.javaedge.benchmark. japex.driver.SqrtNewtonApproxDriverquot;/> </driver> <testCase name=quot;testcase1quot;/> </testSuite> 51
  • 52. Japex: HTML Reports 52
  • 53. Japex: more chart types Scatter chart Line chart 53
  • 54. Japex: pros and cons • Pros • Similar to JUnit • Nice HTML reports • Cons • Last stable release on March 2007 • HotSpot issues are not handled • XML configuration 54
  • 55. Brent Boyer’s Benchmark framework • Part of the “Robust Java benchmarking” article by Brent Boyer • Automate as many aspects as possible: • Resource reclamation • Class loading • Dead code elimination • Statistics 55
  • 56. Benchmark framework example Benchmark.Params params = new Benchmark.Params(true); params.setExecutionTimeGoal(0.5); params.setNumberMeasurements(50); Runnable task = new Runnable() { public void run() { sqrt(getNextRandomNumber()); } }; Benchmark benchmark = new Benchmark(task, params); System.out.println(benchmark.toString()); 56
  • 57. Benchmark single line summary Benchmark output: first = 25.702 us, mean = 91.070 ns (CI deltas: -115.591 ps, +171.423 ps) sd = 1.451 us (CI deltas: -461.523 ns, +676.964 ns) WARNING: execution times have mild outliers, SD VALUES MAY BE INACCURATE 57
  • 58. Outlier and serial correlation issues • Records outlier and serial correlation issues • Outliers indicate that a major measurement error happened • Large outliers - some other activity started on the computer during measurement • Small outliers might hint that DCE occurred • Serial correlation indicates that the JVM has not reached its steady-state performance profile 58
  • 59. Benchmark : pros and cons • Pros • Handles HotSpot related issues • Detailed statistics • Cons • Each run takes a lot of time • Not a formal project • Lacks documentation 59
  • 60. Agenda • Introduction • Java micro benchmarking pitfalls • Writing your own benchmark • Micro benchmarking tools • Summary 60
  • 61. Summary 1 • Micro benchmarking is hard when it comes to Java™ • Define what you want to measure and how want to do it, pick your goals • Know what you are doing • Always warm-up your code • Handle DCE, OSR, GC issues • Use fixed size data sets and fixed work 61
  • 62. Summary 2 • Do not rely solely on microbenchmark results • Sanity check results • Use a profiler • Test your code in real life scenarios under realistic load (macro-benchmark) 62
  • 63. Summary: resources • http://www.ibm.com/developerworks/java/librar y/j-benchmark1.html • http://www.azulsystems.com/events/javaone_20 02/microbenchmarks.pdf • https://japex.dev.java.net/ • http://www.ibm.com/developerworks/java/librar y/j-jtp12214/ • http://www.dei.unipd.it/~bertasi/jcache/ 63
  • 64. Thank You ! 64