Performance Tuning: Pulling a Rabbit From a Hat - Atlassian Summit 2010

6,839 views
6,707 views

Published on

Performance Tuning: Pulling a Rabbit From a Hat

George Barnett of Atlassian

Published in: Technology
1 Comment
1 Like
Statistics
Notes
  • Guys, you have many graphs not based at zero here (e.g. slide 46). This is extremely misleading and hard to read. It is a well-known best practice to always start graphs like this at zero, and a well-known 'malpractice' to not do so (e.g. see book 'How to lie with statistics').
    Other than that, great presentation!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
6,839
On SlideShare
0
From Embeds
0
Number of Embeds
1,501
Actions
Shares
0
Downloads
94
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Performance Tuning: Pulling a Rabbit From a Hat - Atlassian Summit 2010

  1. 1. 1
  2. 2. Pulling a rabbit from a hat How to make your apps run faster George Barnett, Performance Engineer, Atlassian 2 2
  3. 3. Why are we here? • How to make your Atlassian software run faster • What doesnʼt work • What Atlassian does to improve performance • No development HOWTOs or code! 3 3
  4. 4. Who can improve performance? • Atlassian • YOU! (Thanks for watching!) 4 4
  5. 5. Performance Engineering • Isolate performance issues before they become headaches. • Make sure these issues get fixed. 5 5
  6. 6. Simple code lifecycle Developer writes code commit Unit & Functional tests Reporting tests pass Software artefact Performance Testing Reporting 6 6
  7. 7. Performance Testing Scripts • Apache JMeter • In beta for JIRA • Ask support@atlassian.com • Available for Confluence • http://confluence.atlassian.com/display/DOC/Performance+Testing+Scripts 7 7
  8. 8. Performance Testing • Real world data and load • Public Atlassian instances • Performance test suite • JMeter & scripts • Automation • Maven & Bamboo 8 8
  9. 9. Performance Testing • Daily performance tests • Most products & libraries • Weekly soak tests • Run for much longer than daily tests • Reporting for developers • Build screens, dashboards, notifications 9 9
  10. 10. Reporting 10 10
  11. 11. Does it work? Jira 3.13 Jira 4.0 Confluence 2.10 Confluence 3.0 90 600 68 450 45 300 23 150 0 0 Browse Issue (ms) Browse Page (ms) 11 11
  12. 12. Upgrade your Atlassian Software! • Dozens of Performance issues fixed • Release criteria includes performance 12 12
  13. 13. You are here • How I did the testing • Hardware & Software • Must have improvements • Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection • Worth doing • Gigabit vs 100mbit, Heap size tuning, use an SSD • Sadly, not useful • Tune MySQL, Replace MySQL 13 13
  14. 14. The contenders • Dell PE 850 - “WALL-E” • Xeon X3220 2.4Ghz Quad Core • Quad Core, 2 x 4Mb L3 • 8Gb DDR2 RAM • 2 x 7.2K SAS Disks • 1 x Intel X25-E 32GB SSD • ~$1500 14 14
  15. 15. The contenders • Dell PE 2950 - “Johnny 5” • 2 x Xeon E5405 2.0Ghz • Quad Core, 2 x 6M L3 • 32Gb RAM • 2 x 72Gb 15K Drives • ~$4000 15 15
  16. 16. The contenders • Dell PE R610 - “EVE” • 2 x Xeon E5520 2.2Ghz • Quad Core, 8M “SmartCache” • 32Gb Ram • 2 x 146G 15K drives • ~$4000 16 16
  17. 17. The Workload • JIRA 4, MySQL 5, RHEL 5 • ~70,000 Issues, ~80 projects • ~30 reqs/sec, JMeter 2.3.4 (patched, JSR-223) • Traffic modelled on http://jira.atlassian.com with higher writes 17 17
  18. 18. About the graphs • Most show Average Response Time in Milliseconds • Includes time to generate AND deliver the response • Does not include any browser time • Arrows show if lower or higher is better! 18 18
  19. 19. You are here • How I did the testing • Hardware & Software • Must have improvements • Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection • Worth doing • Gigabit vs 100mbit, Heap size tuning, use an SSD • Sadly, not useful • Tune MySQL, Replace MySQL 19 19
  20. 20. Must. Have. 20 20
  21. 21. Upgrade Java 1.5 to 1.6 • Advances in compiler / VM technology • Produces better runtime code • Able to use modern instructions, eg SSE • Able to use modern hardware, eg NUMA 21 21
  22. 22. Java 1.5 vs 1.6 (Johnny 5) Java 1.5.0_18 Java 1.6.0_16 1500 1125 750 375 0 Average Response Time (ms) 6.3X 22 22
  23. 23. Java 1.5 vs 1.6 (Johnny 5) Java 1.5.0_18 Java 1.6.0_16 380 285 190 95 0 Average CPU (%) 26% 23 23
  24. 24. Upgrade Java to 1.6 • 3 x HotSpot VM updates in 1.6.0_XX • Advancing technology • Compressed OOPS on 64bit • Escape Analysis • G1 Garbage Collector • NUMA 24 24
  25. 25. Java 1.6 vs 1.6 (Eve) Java 1.6.0_7 Java 1.6.0_16 148 146 143 141 138 7% Average Response Time (ms) 25 25
  26. 26. Hardware Upgrades • Hyper-Threading • QuickPath Interconnect • Nehalem E5520 (Eve) have 3X on die DDR3 Memory Controllers • Memory throughput improvements - good for Java! Technology Speed DDR2-800 (dual channel) 6.4 GB/sec DDR3-1333 (dual channel) 21.3 GB/sec 26 26
  27. 27. FSB architecture • Shared bus • 1333 MT/s = 1333M Transfers / sec • 4 transfers/clock = 266Mhz. RAM CPU FSB North Bridge CPU South I/O: SATA, USB, etc Bridge 27 27
  28. 28. Quick Path Interconnect • 3.2Ghz, 6.4 GT/sec • 25.6 GB/sec CPU RAM QPI IOH RAM CPU 28 28
  29. 29. Hyper Threading Thread 1 CPU Thread 2 Thread 1 runs until it hits a cache miss. Thread 2 runs on the same CPU while T1 is awaiting data 29 29
  30. 30. Hardware Upgrades 2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve) 2000 1500 1000 500 0 8-12X Average Response Time (ms) 30 30
  31. 31. Improved GC Throughput 2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve) 30000 22500 15000 7500 0 2-3X GC Throughput (MB/sec) 31 31
  32. 32. Garbage Collection • Javaʼs Virtual Machine manages memory • New objects are continually created during runtime • GC is the process of collecting dead objects to reclaim memory 32 32
  33. 33. Tune Garbage Collection • Getting your GC tuning wrong is bad • How bad? • Are the defaults sane? 33 33
  34. 34. Tune Garbage Collection CMS ParNew ParOld 4000 3000 2000 1000 0 Wall-E (ms) Johnny 5 (ms) Average Response Time Eve (ms) 2X 34 34
  35. 35. Tune Garbage Collection CMS ParNew ParOld 500 375 250 125 0 Johnny 5 (ms) Average Response Time Eve (ms) 2X 35 35
  36. 36. Tune Garbage Collection • Defaults are sane • Parallel New + Parallel Old is fastest • CMS prioritises low pauses over CPU usage • CMS does not compact - can lead to heap fragmentation 36 36
  37. 37. VMWare • Does being in a VM hurt? • What if the risks are managed? 37 37
  38. 38. Remove VMWare Real ESX 3.5 ESX 4i 170 128 85 43 0 Average Response Time (ms) 43% 38 38
  39. 39. Remove VMWare • On average ~30% slower • Under extreme resource starvation, up to 100% slower • Avoid VMWare for high traffic instances • See the Atlassian docs for recommendations • http://confluence.atlassian.com/display/DOC/Running+Confluence+in+a+Virtualised+Environment • http://confluence.atlassian.com/display/JIRA/Running+JIRA+in+a+Virtualised+Environment 39 39
  40. 40. Upgrade your browsers! • Use a modern browser • IE8, Safari 4 or Firefox 3.6 • Pick the browser thats right for you • Avoid IE6 (unsupported and slow) • http://sixrevisions.com/infographics/performance-comparison-of-major-web-browsers/ 40 40
  41. 41. You are here • How I did the testing • Hardware & Software • Must have improvements • Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection • Worth doing • Gigabit vs 100mbit, Heap size tuning, use an SSD • Sadly, not useful • Tune MySQL, Replace MySQL 41 41
  42. 42. Maybe. 42 42
  43. 43. Gigabit Network vs 100Mbit • Gigabit is faster, sure • But 100Mb is OK if youʼre not doing big transfers, RIGHT?! 43 43
  44. 44. Use Gigabit 100mb/sec 1000mb/sec 1900 1825 1750 1675 1600 Average Response Time (ms) 10% 44 44
  45. 45. Use Gigabit • Lower RTT on Gigabit • Unlikely to hit throughput limit • If you have enough CPU power, keep the DB on localhost 45 45
  46. 46. Use Gigabit 100Mbit 1000Mbit 200 188 175 163 150 ReIndex Time (seconds) 26% 46 46
  47. 47. Heap size tuning • Memory starvation is bad, but can we drown the system with too much memory 47 47
  48. 48. Heap size - Wall-E 1.5G 3G 6G 2000 1500 1000 500 0 Average Response Time 23% 48 48
  49. 49. Heap size - Johnny 5 1.5G 3G 6G 15G 290 218 145 73 0 Average Response Time 49% 49 49
  50. 50. Heap size - Eve 1.5G 3G 6G 15G 170 128 85 43 0 Average Response Time 28% 50 50
  51. 51. Heap size tuning • Heap too small - bad • Heap too big - not bad! • Bigger heaps give the JVM more room to work • Diminishing returns from really big heaps 51 51
  52. 52. Buy an SSD • SSDs are FAST! • SSDs are EXPENSIVE! • SSDs are all the rage! • DBAs are finally happy (almost!) 52 52
  53. 53. Buy an SSD SATA SSD 800 600 400 200 0 Browse Issue Create Issue 33% 53 53
  54. 54. Buy an SSD • Fast writes • 0.1ms Avg Service Time • Most improvement comes from SSD on the DB • Writing to the Lucene index means re-opening IndexReaders • Much faster on SSD • Lucene Search Term Cache fits in OS buffers 54 54
  55. 55. You are here • How I did the testing • Hardware & Software • Must have improvements • Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection • Worth doing • Gigabit vs 100mbit, Heap size tuning, use an SSD • Sadly, not useful • Tune MySQL, Replace MySQL 55 55
  56. 56. Meh. 56 56
  57. 57. Replace MySQL with MySQL • MySQL 5.0.45 - RHEL 5 • MySQL 5.0.84 Percona B18. • Percona HighPerf patches fix contention points • Improved I/O • Better scaling 57 57
  58. 58. Replace MySQL with MySQL MySQL 5.0.45 Percona MySQL 5.0.85 1687 1515 1344 1172 1000 0.4% Average Response Time (ms) 58 58
  59. 59. Replace MySQL with MySQL MySQL 5.0.45 Percona MySQL 5.0.84 25 19 13 6 0 0% Average CPU(%) 59 59
  60. 60. Replace MySQL with MySQL • JIRA doesnʼt do enough DB queries • JIRA uses Lucene for many queries • Percona Highperf MySQL really shines at high query rates • Try this if you share your DB server with other apps! 60 60
  61. 61. Tune MySQL • InnoDB-Heavy-4G.conf sort_buffer_size = 8M join_buffer_size = 8M read_buffer_size = 2M read_rnd_buffer_size = 16M thread_concurrecy = 16 61 61
  62. 62. Tune MySQL Defaults Tuned 1710 1533 1355 1178 1% 1000 Average Response Time (ms) 62 62
  63. 63. Tune MySQL MyISAM ALL Engines join_buffer_size - Large joins without Indexes key_cache_age_threshold, key_cache_block_size, sort_buffer_size - Used by key_cache_division_limit, ORDER BY / GROUP BY key_buffer_size read_buffer_size, query_cache_size read_rnd_buffer_size query_cache_limit bulk_insert_buffer_size query_cache_min_res_unit 63 63
  64. 64. Tune MySQL • Be sure to set innodb_buffer_pool_size • Note! innodb_log_file_size will affect shutdown speed 64 64
  65. 65. Remember! • Upgrade! • Atlassian software • Java to 1.6 • Your server • Use Parallel Garbage Collection with a BIG heap • Remove VMWare if you can • Use 1000mbit network • Buy an SSD for write workloads 65 65
  66. 66. A final word • Donʼt resist change but TEST EVERYTHING! 66 66
  67. 67. Thanks! Questions? • Come chat to me in the hallway! • Charlie Lounge, 10 June, 3pm - 4pm • http://blogs.atlassian.com/developer Flickr Attributions: Siesta in Denmark - http://www.flickr.com/photos/jakecaptive/27123533/ Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/ Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/ 67 67

×