0
1
Pulling a rabbit from a hat
How to make your apps run faster


George Barnett, Performance Engineer, Atlassian




       ...
Why are we here?
• How to make your Atlassian software run faster
• What doesnʼt work
• What Atlassian does to improve per...
Who can improve performance?
• Atlassian
• YOU! (Thanks for watching!)




                                4
             ...
Performance Engineering
• Isolate performance issues before they become headaches.
• Make sure these issues get fixed.




...
Simple code lifecycle
   Developer writes code

               commit


   Unit & Functional tests   Reporting

          ...
Performance Testing Scripts
• Apache JMeter
• In beta for JIRA
 • Ask support@atlassian.com

• Available for Confluence
 • ...
Performance Testing
• Real world data and load
 • Public Atlassian instances

• Performance test suite
 • JMeter & scripts...
Performance Testing
• Daily performance tests
 • Most products & libraries

• Weekly soak tests
 • Run for much longer tha...
Reporting




            10
             10
Does it work?
       Jira 3.13                   Jira 4.0    Confluence 2.10       Confluence 3.0


  90                    ...
Upgrade your Atlassian Software!
• Dozens of Performance issues fixed
• Release criteria includes performance




         ...
You are here
• How I did the testing
 • Hardware & Software

• Must have improvements
 • Upgrade Java, Hardware, Remove VM...
The contenders
• Dell PE 850 - “WALL-E”
• Xeon X3220 2.4Ghz Quad Core
 • Quad Core, 2 x 4Mb L3

• 8Gb DDR2 RAM
• 2 x 7.2K ...
The contenders
• Dell PE 2950 - “Johnny 5”
• 2 x Xeon E5405 2.0Ghz
 • Quad Core, 2 x 6M L3

• 32Gb RAM
• 2 x 72Gb 15K Driv...
The contenders
• Dell PE R610 - “EVE”
• 2 x Xeon E5520 2.2Ghz
 • Quad Core, 8M “SmartCache”

• 32Gb Ram
• 2 x 146G 15K dri...
The Workload
• JIRA 4, MySQL 5, RHEL 5
• ~70,000 Issues, ~80 projects
• ~30 reqs/sec, JMeter 2.3.4 (patched, JSR-223)
• Tr...
About the graphs
• Most show Average Response Time in Milliseconds
 • Includes time to generate AND deliver the response
 ...
You are here
• How I did the testing
 • Hardware & Software

• Must have improvements
 • Upgrade Java, Hardware, Remove VM...
Must. Have.




              20
               20
Upgrade Java 1.5 to 1.6
• Advances in compiler / VM technology
 • Produces better runtime code
 • Able to use modern instr...
Java 1.5 vs 1.6 (Johnny 5)
              Java 1.5.0_18                 Java 1.6.0_16

       1500


       1125


        ...
Java 1.5 vs 1.6 (Johnny 5)
             Java 1.5.0_18                 Java 1.6.0_16


       380


       285


       190...
Upgrade Java to 1.6
• 3 x HotSpot VM updates in 1.6.0_XX
• Advancing technology
 • Compressed OOPS on 64bit
 • Escape Anal...
Java 1.6 vs 1.6 (Eve)
                Java 1.6.0_7                 Java 1.6.0_16

          148


          146


        ...
Hardware Upgrades
• Hyper-Threading
• QuickPath Interconnect
• Nehalem E5520 (Eve) have 3X on die DDR3 Memory Controllers
...
FSB architecture
• Shared bus
• 1333 MT/s = 1333M Transfers / sec
• 4 transfers/clock = 266Mhz.
                          ...
Quick Path Interconnect
• 3.2Ghz, 6.4 GT/sec
• 25.6 GB/sec



                       CPU
        RAM

                    ...
Hyper Threading

                           Thread 1
                                               CPU
                  ...
Hardware Upgrades
    2.4Ghz Xeon (Wall-E)    2.0Ghz Xeon (Johnny 5)      2.2Ghz Xeon (Eve)


             2000


        ...
Improved GC Throughput
    2.4Ghz Xeon (Wall-E)   2.0Ghz Xeon (Johnny 5)   2.2Ghz Xeon (Eve)


             30000


      ...
Garbage Collection
• Javaʼs Virtual Machine manages memory
• New objects are continually created during runtime
• GC is th...
Tune Garbage Collection
• Getting your GC tuning wrong is bad
• How bad?
• Are the defaults sane?




                    ...
Tune Garbage Collection
                  CMS                 ParNew ParOld

   4000


   3000


   2000


   1000


     ...
Tune Garbage Collection
                  CMS                ParNew ParOld

      500


      375


      250


      125
...
Tune Garbage Collection
• Defaults are sane
• Parallel New + Parallel Old is fastest
• CMS prioritises low pauses over CPU...
VMWare
• Does being in a VM hurt?
• What if the risks are managed?




                                   37
             ...
Remove VMWare
            Real             ESX 3.5            ESX 4i

      170


      128


       85


       43


    ...
Remove VMWare
• On average ~30% slower
• Under extreme resource starvation, up to 100% slower
• Avoid VMWare for high traf...
Upgrade your browsers!
• Use a modern browser
  • IE8, Safari 4 or Firefox 3.6

• Pick the browser thats right for you
• A...
You are here
• How I did the testing
 • Hardware & Software

• Must have improvements
 • Upgrade Java, Hardware, Remove VM...
Maybe.




         42
          42
Gigabit Network vs 100Mbit
• Gigabit is faster, sure
• But 100Mb is OK if youʼre not doing big transfers, RIGHT?!




    ...
Use Gigabit
               100mb/sec                 1000mb/sec

        1900


        1825


        1750


        1675...
Use Gigabit
• Lower RTT on Gigabit
• Unlikely to hit throughput limit
• If you have enough CPU power, keep the DB on local...
Use Gigabit
              100Mbit               1000Mbit

        200


        188


        175


        163


        ...
Heap size tuning
• Memory starvation is bad, but can we drown the system with too much
  memory




                      ...
Heap size - Wall-E
               1.5G            3G         6G

       2000


       1500


       1000


        500


 ...
Heap size - Johnny 5
               1.5G   3G             6G       15G


         290


         218


         145


    ...
Heap size - Eve
              1.5G      3G             6G    15G


        170


        128


         85


         43

...
Heap size tuning
• Heap too small - bad
• Heap too big - not bad!
• Bigger heaps give the JVM more room to work
• Diminish...
Buy an SSD
• SSDs are FAST!
• SSDs are EXPENSIVE!
• SSDs are all the rage!
• DBAs are finally happy (almost!)




         ...
Buy an SSD
                        SATA       SSD


     800


     600


     400


     200


       0
             Brow...
Buy an SSD
• Fast writes
 • 0.1ms Avg Service Time

• Most improvement comes from SSD on the DB
• Writing to the Lucene in...
You are here
• How I did the testing
 • Hardware & Software

• Must have improvements
 • Upgrade Java, Hardware, Remove VM...
Meh.




       56
        56
Replace MySQL with MySQL
• MySQL 5.0.45 - RHEL 5
• MySQL 5.0.84 Percona B18.
• Percona HighPerf patches fix contention poin...
Replace MySQL with MySQL
             MySQL 5.0.45                    Percona MySQL 5.0.85


      1687


      1515


   ...
Replace MySQL with MySQL
           MySQL 5.0.45              Percona MySQL 5.0.84


      25


      19


      13


    ...
Replace MySQL with MySQL
• JIRA doesnʼt do enough DB queries
• JIRA uses Lucene for many queries
• Percona Highperf MySQL ...
Tune MySQL
• InnoDB-Heavy-4G.conf


     sort_buffer_size = 8M
     join_buffer_size = 8M
     read_buffer_size = 2M
     ...
Tune MySQL
             Defaults                Tuned

    1710


    1533


    1355


    1178

                        ...
Tune MySQL

    MyISAM                       ALL Engines

                                 join_buffer_size - Large joins
...
Tune MySQL
• Be sure to set innodb_buffer_pool_size
• Note! innodb_log_file_size will affect shutdown speed




           ...
Remember!
• Upgrade!
 • Atlassian software
 • Java to 1.6
 • Your server

• Use Parallel Garbage Collection with a BIG hea...
A final word
• Donʼt resist change but TEST EVERYTHING!




                                             66
               ...
Thanks! Questions?
• Come chat to me in the hallway!
• Charlie Lounge, 10 June, 3pm - 4pm
• http://blogs.atlassian.com/dev...
Upcoming SlideShare
Loading in...5
×

Performance Tuning: Pulling a Rabbit From a Hat - Atlassian Summit 2010

5,831

Published on

Performance Tuning: Pulling a Rabbit From a Hat

George Barnett of Atlassian

Published in: Technology
1 Comment
0 Likes
Statistics
Notes
  • Guys, you have many graphs not based at zero here (e.g. slide 46). This is extremely misleading and hard to read. It is a well-known best practice to always start graphs like this at zero, and a well-known 'malpractice' to not do so (e.g. see book 'How to lie with statistics').
    Other than that, great presentation!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total Views
5,831
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
85
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Performance Tuning: Pulling a Rabbit From a Hat - Atlassian Summit 2010"

  1. 1. 1
  2. 2. Pulling a rabbit from a hat How to make your apps run faster George Barnett, Performance Engineer, Atlassian 2 2
  3. 3. Why are we here? • How to make your Atlassian software run faster • What doesnʼt work • What Atlassian does to improve performance • No development HOWTOs or code! 3 3
  4. 4. Who can improve performance? • Atlassian • YOU! (Thanks for watching!) 4 4
  5. 5. Performance Engineering • Isolate performance issues before they become headaches. • Make sure these issues get fixed. 5 5
  6. 6. Simple code lifecycle Developer writes code commit Unit & Functional tests Reporting tests pass Software artefact Performance Testing Reporting 6 6
  7. 7. Performance Testing Scripts • Apache JMeter • In beta for JIRA • Ask support@atlassian.com • Available for Confluence • http://confluence.atlassian.com/display/DOC/Performance+Testing+Scripts 7 7
  8. 8. Performance Testing • Real world data and load • Public Atlassian instances • Performance test suite • JMeter & scripts • Automation • Maven & Bamboo 8 8
  9. 9. Performance Testing • Daily performance tests • Most products & libraries • Weekly soak tests • Run for much longer than daily tests • Reporting for developers • Build screens, dashboards, notifications 9 9
  10. 10. Reporting 10 10
  11. 11. Does it work? Jira 3.13 Jira 4.0 Confluence 2.10 Confluence 3.0 90 600 68 450 45 300 23 150 0 0 Browse Issue (ms) Browse Page (ms) 11 11
  12. 12. Upgrade your Atlassian Software! • Dozens of Performance issues fixed • Release criteria includes performance 12 12
  13. 13. You are here • How I did the testing • Hardware & Software • Must have improvements • Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection • Worth doing • Gigabit vs 100mbit, Heap size tuning, use an SSD • Sadly, not useful • Tune MySQL, Replace MySQL 13 13
  14. 14. The contenders • Dell PE 850 - “WALL-E” • Xeon X3220 2.4Ghz Quad Core • Quad Core, 2 x 4Mb L3 • 8Gb DDR2 RAM • 2 x 7.2K SAS Disks • 1 x Intel X25-E 32GB SSD • ~$1500 14 14
  15. 15. The contenders • Dell PE 2950 - “Johnny 5” • 2 x Xeon E5405 2.0Ghz • Quad Core, 2 x 6M L3 • 32Gb RAM • 2 x 72Gb 15K Drives • ~$4000 15 15
  16. 16. The contenders • Dell PE R610 - “EVE” • 2 x Xeon E5520 2.2Ghz • Quad Core, 8M “SmartCache” • 32Gb Ram • 2 x 146G 15K drives • ~$4000 16 16
  17. 17. The Workload • JIRA 4, MySQL 5, RHEL 5 • ~70,000 Issues, ~80 projects • ~30 reqs/sec, JMeter 2.3.4 (patched, JSR-223) • Traffic modelled on http://jira.atlassian.com with higher writes 17 17
  18. 18. About the graphs • Most show Average Response Time in Milliseconds • Includes time to generate AND deliver the response • Does not include any browser time • Arrows show if lower or higher is better! 18 18
  19. 19. You are here • How I did the testing • Hardware & Software • Must have improvements • Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection • Worth doing • Gigabit vs 100mbit, Heap size tuning, use an SSD • Sadly, not useful • Tune MySQL, Replace MySQL 19 19
  20. 20. Must. Have. 20 20
  21. 21. Upgrade Java 1.5 to 1.6 • Advances in compiler / VM technology • Produces better runtime code • Able to use modern instructions, eg SSE • Able to use modern hardware, eg NUMA 21 21
  22. 22. Java 1.5 vs 1.6 (Johnny 5) Java 1.5.0_18 Java 1.6.0_16 1500 1125 750 375 0 Average Response Time (ms) 6.3X 22 22
  23. 23. Java 1.5 vs 1.6 (Johnny 5) Java 1.5.0_18 Java 1.6.0_16 380 285 190 95 0 Average CPU (%) 26% 23 23
  24. 24. Upgrade Java to 1.6 • 3 x HotSpot VM updates in 1.6.0_XX • Advancing technology • Compressed OOPS on 64bit • Escape Analysis • G1 Garbage Collector • NUMA 24 24
  25. 25. Java 1.6 vs 1.6 (Eve) Java 1.6.0_7 Java 1.6.0_16 148 146 143 141 138 7% Average Response Time (ms) 25 25
  26. 26. Hardware Upgrades • Hyper-Threading • QuickPath Interconnect • Nehalem E5520 (Eve) have 3X on die DDR3 Memory Controllers • Memory throughput improvements - good for Java! Technology Speed DDR2-800 (dual channel) 6.4 GB/sec DDR3-1333 (dual channel) 21.3 GB/sec 26 26
  27. 27. FSB architecture • Shared bus • 1333 MT/s = 1333M Transfers / sec • 4 transfers/clock = 266Mhz. RAM CPU FSB North Bridge CPU South I/O: SATA, USB, etc Bridge 27 27
  28. 28. Quick Path Interconnect • 3.2Ghz, 6.4 GT/sec • 25.6 GB/sec CPU RAM QPI IOH RAM CPU 28 28
  29. 29. Hyper Threading Thread 1 CPU Thread 2 Thread 1 runs until it hits a cache miss. Thread 2 runs on the same CPU while T1 is awaiting data 29 29
  30. 30. Hardware Upgrades 2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve) 2000 1500 1000 500 0 8-12X Average Response Time (ms) 30 30
  31. 31. Improved GC Throughput 2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve) 30000 22500 15000 7500 0 2-3X GC Throughput (MB/sec) 31 31
  32. 32. Garbage Collection • Javaʼs Virtual Machine manages memory • New objects are continually created during runtime • GC is the process of collecting dead objects to reclaim memory 32 32
  33. 33. Tune Garbage Collection • Getting your GC tuning wrong is bad • How bad? • Are the defaults sane? 33 33
  34. 34. Tune Garbage Collection CMS ParNew ParOld 4000 3000 2000 1000 0 Wall-E (ms) Johnny 5 (ms) Average Response Time Eve (ms) 2X 34 34
  35. 35. Tune Garbage Collection CMS ParNew ParOld 500 375 250 125 0 Johnny 5 (ms) Average Response Time Eve (ms) 2X 35 35
  36. 36. Tune Garbage Collection • Defaults are sane • Parallel New + Parallel Old is fastest • CMS prioritises low pauses over CPU usage • CMS does not compact - can lead to heap fragmentation 36 36
  37. 37. VMWare • Does being in a VM hurt? • What if the risks are managed? 37 37
  38. 38. Remove VMWare Real ESX 3.5 ESX 4i 170 128 85 43 0 Average Response Time (ms) 43% 38 38
  39. 39. Remove VMWare • On average ~30% slower • Under extreme resource starvation, up to 100% slower • Avoid VMWare for high traffic instances • See the Atlassian docs for recommendations • http://confluence.atlassian.com/display/DOC/Running+Confluence+in+a+Virtualised+Environment • http://confluence.atlassian.com/display/JIRA/Running+JIRA+in+a+Virtualised+Environment 39 39
  40. 40. Upgrade your browsers! • Use a modern browser • IE8, Safari 4 or Firefox 3.6 • Pick the browser thats right for you • Avoid IE6 (unsupported and slow) • http://sixrevisions.com/infographics/performance-comparison-of-major-web-browsers/ 40 40
  41. 41. You are here • How I did the testing • Hardware & Software • Must have improvements • Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection • Worth doing • Gigabit vs 100mbit, Heap size tuning, use an SSD • Sadly, not useful • Tune MySQL, Replace MySQL 41 41
  42. 42. Maybe. 42 42
  43. 43. Gigabit Network vs 100Mbit • Gigabit is faster, sure • But 100Mb is OK if youʼre not doing big transfers, RIGHT?! 43 43
  44. 44. Use Gigabit 100mb/sec 1000mb/sec 1900 1825 1750 1675 1600 Average Response Time (ms) 10% 44 44
  45. 45. Use Gigabit • Lower RTT on Gigabit • Unlikely to hit throughput limit • If you have enough CPU power, keep the DB on localhost 45 45
  46. 46. Use Gigabit 100Mbit 1000Mbit 200 188 175 163 150 ReIndex Time (seconds) 26% 46 46
  47. 47. Heap size tuning • Memory starvation is bad, but can we drown the system with too much memory 47 47
  48. 48. Heap size - Wall-E 1.5G 3G 6G 2000 1500 1000 500 0 Average Response Time 23% 48 48
  49. 49. Heap size - Johnny 5 1.5G 3G 6G 15G 290 218 145 73 0 Average Response Time 49% 49 49
  50. 50. Heap size - Eve 1.5G 3G 6G 15G 170 128 85 43 0 Average Response Time 28% 50 50
  51. 51. Heap size tuning • Heap too small - bad • Heap too big - not bad! • Bigger heaps give the JVM more room to work • Diminishing returns from really big heaps 51 51
  52. 52. Buy an SSD • SSDs are FAST! • SSDs are EXPENSIVE! • SSDs are all the rage! • DBAs are finally happy (almost!) 52 52
  53. 53. Buy an SSD SATA SSD 800 600 400 200 0 Browse Issue Create Issue 33% 53 53
  54. 54. Buy an SSD • Fast writes • 0.1ms Avg Service Time • Most improvement comes from SSD on the DB • Writing to the Lucene index means re-opening IndexReaders • Much faster on SSD • Lucene Search Term Cache fits in OS buffers 54 54
  55. 55. You are here • How I did the testing • Hardware & Software • Must have improvements • Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection • Worth doing • Gigabit vs 100mbit, Heap size tuning, use an SSD • Sadly, not useful • Tune MySQL, Replace MySQL 55 55
  56. 56. Meh. 56 56
  57. 57. Replace MySQL with MySQL • MySQL 5.0.45 - RHEL 5 • MySQL 5.0.84 Percona B18. • Percona HighPerf patches fix contention points • Improved I/O • Better scaling 57 57
  58. 58. Replace MySQL with MySQL MySQL 5.0.45 Percona MySQL 5.0.85 1687 1515 1344 1172 1000 0.4% Average Response Time (ms) 58 58
  59. 59. Replace MySQL with MySQL MySQL 5.0.45 Percona MySQL 5.0.84 25 19 13 6 0 0% Average CPU(%) 59 59
  60. 60. Replace MySQL with MySQL • JIRA doesnʼt do enough DB queries • JIRA uses Lucene for many queries • Percona Highperf MySQL really shines at high query rates • Try this if you share your DB server with other apps! 60 60
  61. 61. Tune MySQL • InnoDB-Heavy-4G.conf sort_buffer_size = 8M join_buffer_size = 8M read_buffer_size = 2M read_rnd_buffer_size = 16M thread_concurrecy = 16 61 61
  62. 62. Tune MySQL Defaults Tuned 1710 1533 1355 1178 1% 1000 Average Response Time (ms) 62 62
  63. 63. Tune MySQL MyISAM ALL Engines join_buffer_size - Large joins without Indexes key_cache_age_threshold, key_cache_block_size, sort_buffer_size - Used by key_cache_division_limit, ORDER BY / GROUP BY key_buffer_size read_buffer_size, query_cache_size read_rnd_buffer_size query_cache_limit bulk_insert_buffer_size query_cache_min_res_unit 63 63
  64. 64. Tune MySQL • Be sure to set innodb_buffer_pool_size • Note! innodb_log_file_size will affect shutdown speed 64 64
  65. 65. Remember! • Upgrade! • Atlassian software • Java to 1.6 • Your server • Use Parallel Garbage Collection with a BIG heap • Remove VMWare if you can • Use 1000mbit network • Buy an SSD for write workloads 65 65
  66. 66. A final word • Donʼt resist change but TEST EVERYTHING! 66 66
  67. 67. Thanks! Questions? • Come chat to me in the hallway! • Charlie Lounge, 10 June, 3pm - 4pm • http://blogs.atlassian.com/developer Flickr Attributions: Siesta in Denmark - http://www.flickr.com/photos/jakecaptive/27123533/ Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/ Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/ 67 67
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×