Performance tuning for JavaapplicationsAndrew PrenticeAtlassian                              2
Topics• How to make your apps run faster• Tuning ideas                                     3
A note• X vs Y will always depend on context• Test with your application!                                         4
You are here• How I did the testing • Hardware & software• Must have improvements • Jave, Hardware, Virtualization and Gar...
The contenders• Dell PE 860 - “WALL-E”• Xeon X3220 2.4 Ghz Quad core • Quad Core, 2 x 4Mb L3• 8 Gb DDR2 RAM• 2 x 7.2K SAS ...
The contenders• Dell PE 2950 - “Johnny 5”• 2 x Xeon E5 405 2.0 Ghz • Quad Core, 2 x 6M L3• 32 Gb RAM• 2 x 72 Gb 15 K Drive...
The contenders• Dell PE R610 - “EVE”• 2 x Xeon E5520 2.2Ghz • Quad Core, 8M “SmartCache”• 32 Gb RAM• 2 x 146G 15K drives• ...
The Workload• JIRA 4, MySQL 5, RHEL 5• Java 1.6, 3G heap, ParNew, ParOld GC• ~70,000 Issues, ~80 projects• ~30 reqs/sec, J...
About the graphs• Most show Average Response Time in milliseconds • Includes time to generate AND deliver the response • D...
You are here• How I did the testing • Hardware & software• Must have improvements • Jave, Hardware, Virtualization and Gar...
Must. Have.              12
Upgrade Java 1.5 to 1.6• Advances in complier/VM technology • Produces better runtime code • Able to use modern instructio...
Java 1.5 vs 1.6 (Johnny 5)             Java 1.5                    Java 1.6.0     1500     1125      750      375         ...
Java 1.5 vs 1.6 (Johnny 5)           Java 1.5.0                Java 1.6.0     380     285     190      95                 ...
Upgrade Java to 1.6• HotSpot VM updates in 1.6• Advancing technology • Compressed OOPS on 64bit • Escape Analysis • NUMA• ...
Java 1.6 vs 1.6 (Eve)           Java 1.6.0_7                       Java 1.6.0_16     148     111      74      37          ...
Hardware Upgrades• Hyper-Threading• QuickPath Interconnect• Nehalem (Eve) have on die DDR3 Memory Controllers• Memory thro...
Hyper Threading                  Thread 1                                              CPU                  Thread 2 Threa...
Hardware Upgrades   2.4Ghz Xeon (Wall-E)                2.0Ghz Xeon (Johnny 5)   2.2Ghz Xeon (Eve)  2000  1500  1000   500...
Improved GC Throughput      2.4Ghz Xeon (Wall-E)                2.0Ghz Xeon (Johnny 5)   2.2Ghz Xeon (Eve)  30000  22500  ...
Garbage Collection• Java’s Virtual Machine manages memory• New objects are continually created during runtime• GC is the p...
Tune Garbage Collection• Getting your GC tuning wrong is bad• How bad?• Are the defaults sane?                            ...
Tune Garbage Collection                     CMS                       ParNew ParOld4000300020001000                       ...
Tune Garbage Collection               CMS                            ParNew ParOld500375250125                            ...
Tune Garbage Collection• Defaults are sane• Parallel New + Parallel Old is fastest• CMS prioritizes low pauses over CPU us...
Virtualisation• Does being in a VM hurt?• What if the risks are managed?                                   27
Virtualisation      Real   VMWare ESX 3.5               VMWare ESX 4i  170.0  127.5   85.0   42.5                         ...
Virtualisation• On average ~30% slower• Under extreme resource starvation, up to 100% slower• Avoid virtualization for hig...
You are here• How I did the testing • Hardware & software• Must have improvements • Java, Hardware, Virtualization and Gar...
Maybe.         31
Gigabit Network vs 100Mbit• Gigabit is faster, sure• But 100Mb is OK if you’re not doing big transfers,  RIGHT?!          ...
Use Gigabit          100mb/sec                           1000mb/sec   1900   1425    950    475                           ...
Use Gigabit• Lower RTT on Gigabit• If you have enough CPU power, keep the DB on  localhost• Check your development environ...
Use Gigabit         100Mbit                      1000Mbit   200   150   100    50                                         ...
Heap size tuning• Memory starvation is bad, but can we “drown” Java  with too much memory?                                ...
Heap size - Wall-E        1.5G             3G            6G 2000 1500 1000  500                                           ...
Heap size - Johnny 5         1.5G     3G                    6G   15G 290.0 217.5 145.0  72.5                              ...
Heap size - Eve         1.5G   3G                      6G   15G 170.0 127.5  85.0  42.5                                   ...
Heap size tuning• Heap too small - bad• Heap too big - not bad!• Bigger heaps give the JVM more room to work• Diminishing ...
Buy an SSD• SSDs are FAST!• SSDs are EXPENSIVE!• SSDs are all the rage!• DBAs are finally happy (almost)                  ...
Buy an SSD             SATA      SSD800600400200                                 33%  0      Browse Issue   Create Issue  ...
Buy an SSD• Fast writes • 0.1ms Avg Service Time• Most improvement comes from SSD on the DB• Writing to the Lucene index m...
Remember!• Upgrade! • Java to 1.6 • Your server• Use Parallel Garbage Collection with a big heap• Remove virtualization if...
...Part Two:Environment andJoanna ThurmannJIRA Systems Administrator                             2
Don’t let this happen to you!                                3
“Step1 - We admitted we were powerless -our JIRA had become unmanageable.”                       JIRAcholics Anonymous    ...
Contents• Performance Considerations• Hardware & Software Upgrade• JIRA and Plugin Upgrades• Structure for Scale          ...
Our JIRA           6
Our JIRA• 210,000 issues                   6
Our JIRA• 210,000 issues• 142 projects                   6
Our JIRA• 210,000 issues• 142 projects• 66 workflows                   6
Our JIRA• 210,000 issues• 142 projects• 66 workflows• 320 custom fields                      6
Our JIRA• 210,000 issues• 142 projects• 66 workflows• 320 custom fields• 6,000 user directory (~400 concurrent)           ...
Our JIRA• 210,000 issues• 142 projects• 66 workflows• 320 custom fields• 6,000 user directory (~400 concurrent)• 24x7 usag...
Considerations forPerformance and Usability• Total Issue Count• User Concurrency• # and Complexity of Configurations• Inte...
In General…• JIRA uses the disk much more than db (due to Lucene  indexing)• JIRA needs considerable memory               ...
Our Upgrade Summary               Before                                         After HP ProLiant DL580 G4              ...
Our Upgrade – Disk and CPU• Chose Raid10 over Raid5 for better performance• Upgraded Controller from 512 MB cache to 1GB c...
Our Upgrade – Memory                       11
Our Upgrade – Memory• 3.5GB  6GB                       11
Our Upgrade – Memory• 3.5GB  6GB• Full garbage collections times lower and less frequent                                 ...
Our Upgrade – Memory• 3.5GB  6GB• Full garbage collections times lower and less frequent• Free up more memory            ...
Our Upgrade – Memory• 3.5GB  6GB• Full garbage collections times lower and less frequent• Free up more memory• Eliminate ...
Our Upgrade - MemoryBefore (3.5 GB JIRA, 8 GB Total)      After (6 GB JIRA, 32 GB Total)Full GC: 11-50s, 1x/hour         ...
Our Upgrade - MemoryBefore (3.5 GB JIRA, 8 GB Total)      After (6 GB JIRA, 32 GB Total)Full GC: 11-50s, 1x/hour         ...
Our Upgrade - Software• SQL Server 2008 R2• OK to run JIRA and DB on the same server• Allocate sufficient memory to the db...
What’s the bottom line?                          14
What’s the bottom line?                          14
Results – backend                       Then        Now       Percentage                                               fas...
Results – front end                       Then     Now      Percentage                                           faster  S...
Plugin challenges we faced• Memory leak• End of support of *key* workflow plugin• Odd behavior in JIRA 4.1.1 upgrade• Perf...
Plugin Install & Upgrades                            18
Plugin Install & Upgrades• Scrutinize Plugins                            18
Plugin Install & Upgrades• Scrutinize Plugins   Ask yourself…                                       18
Plugin Install & Upgrades• Scrutinize Plugins   Ask yourself…                       • Is the plugin reputable?            ...
Plugin Install & Upgrades• Scrutinize Plugins   Ask yourself…                       • Is the plugin reputable?            ...
Plugin Install & Upgrades• Scrutinize Plugins   Ask yourself…                       • Is the plugin reputable?            ...
Plugin Install & Upgrades• Scrutinize Plugins   Ask yourself…                       • Is the plugin reputable?            ...
Plugin Install & Upgrades• Scrutinize Plugins   Ask yourself…                       • Is the plugin reputable?            ...
Plugin Install & Upgrades• Scrutinize Plugins   Ask yourself…                       • Is the plugin reputable?            ...
Plugin Install & Upgrades• Scrutinize Plugins       Ask yourself…                           • Is the plugin reputable?• Ge...
Plugin Install & Upgrades• Scrutinize Plugins       Ask yourself…                           • Is the plugin reputable?• Ge...
Plugin Install & Upgrades• Scrutinize Plugins       Ask yourself…                           • Is the plugin reputable?• Ge...
Why Upgrade JIRA Regularly?                              19
Why Upgrade JIRA Regularly?• Performance improvements & key fixes in each release for  BOTH end-users AND admins          ...
Why Upgrade JIRA Regularly?• Performance improvements & key fixes in each release for  BOTH end-users AND admins• Example:...
My Weekend Upgrade                     20
My Weekend Upgrade     Before                     20
My Weekend Upgrade     Before                     20
My Weekend Upgrade     Before          After                             20
My Weekend Upgrade     Before          After                             20
Why else?            21
Why else?• Fewer internal support calls                                 21
Why else?• Fewer internal support calls• Less time to resolve them                                 21
Why else?• Fewer internal support calls• Less time to resolve them• Easier & faster for Atlassian to support you          ...
Why else?• Fewer internal support calls• Less time to resolve them• Easier & faster for Atlassian to support you• Early bi...
Our last JIRA upgrade rocked!                                22
Our last JIRA upgrade rocked!              • Resolved 4 bugs found internally                                             ...
Our last JIRA upgrade rocked!              • Resolved 4 bugs found internally              • Delivered 5 must-have feature...
Our last JIRA upgrade rocked!              • Resolved 4 bugs found internally              • Delivered 5 must-have feature...
Our last JIRA upgrade rocked!              • Resolved 4 bugs found internally              • Delivered 5 must-have feature...
Structure – For Startup JIRAs                                23
Structure – For Startup JIRAs• Configurations  Less is more since JIRA caches them                                       ...
Structure – For Startup JIRAs• Configurations  Less is more since JIRA caches them• Use fewer issue types, resolutions, s...
Structure – For Startup JIRAs• Configurations  Less is more since JIRA caches them• Use fewer issue types, resolutions, s...
Structure – For Startup JIRAs• Configurations  Less is more since JIRA caches them• Use fewer issue types, resolutions, s...
Structure – Project Roles                            24
Structure – Project Roles• Use Project Roles vs individual names & groups in  permission schemes                          ...
Structure – Project Roles• Use Project Roles vs individual names & groups in  permission schemes                       Bef...
Structure – Project Roles• Use Project Roles vs individual names & groups in  permission schemes                       Bef...
Structure – Project Roles• Use Project Roles vs individual names & groups in  permission schemes                       Bef...
Structure – Project Roles• Use Project Roles vs individual names & groups in  permission schemes                       Bef...
Structure – For Epic JIRAs                             25
Structure – For Epic JIRAs• Consolidate any and all schemes                                    25
Structure – For Epic JIRAs• Consolidate any and all schemes• Use Scheme Tools: e.g. Scheme Comparison Tool,  Group to Proj...
Structure – For Epic JIRAs• Consolidate any and all schemes• Use Scheme Tools: e.g. Scheme Comparison Tool,  Group to Proj...
We Declutter               26
We Declutter• Created project naming conventions                                       26
We Declutter• Created project naming conventions• Eliminated & consolidated custom fields (25 out of 345)                 ...
We Declutter• Created project naming conventions• Eliminated & consolidated custom fields (25 out of 345)• Took custom fie...
We Declutter• Created project naming conventions• Eliminated & consolidated custom fields (25 out of 345)• Took custom fie...
We Tidy          27
We Tidy• Super User (SU) plugin                           27
We Tidy• Super User (SU) plugin• Easier to troubleshoot user issues                                       27
We Tidy• Super User (SU) plugin• Easier to troubleshoot user issues• Helps us “terminate” JIRA accounts  •Delete their das...
We Tidy• Super User (SU) plugin• Easier to troubleshoot user issues• Helps us “terminate” JIRA accounts  •Delete their das...
We Train           28
We Train• Limit General Permissions & train the users on how/when to use                                                  ...
We Train• Limit General Permissions & train the users on how/when to use    • Bulk Change                                 ...
We Train• Limit General Permissions & train the users on how/when to use    • Bulk Change    • Create Shared Objects (filt...
We Train• Limit General Permissions & train the users on how/when to use    • Bulk Change    • Create Shared Objects (filt...
We Train• Limit General Permissions & train the users on how/when to use    • Bulk Change    • Create Shared Objects (filt...
We Do More             29
We Do More• Use db-specific vs. XML-backup                                   29
We Do More• Use db-specific vs. XML-backup• Schedule regular maintenance                                   29
We Do More• Use db-specific vs. XML-backup• Schedule regular maintenance• Regularly check logs for errors                 ...
We Do More• Use db-specific vs. XML-backup• Schedule regular maintenance• Regularly check logs for errors• Read the JIRA p...
30
We     Keep            Going                    30
“Step12 - we promised to help otherJIRAcholics, and to always practice what wepreach                          JIRAcholics ...
A word from my sponsors….
A word from my sponsors….
A word from my sponsors…. Polycom, Inc. (Nasdaq: PLCM) is a global leader in unified communicationssolutions with industry-...
blogs.atlassian.com/developerFlickr Attributions:Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/Blue Streak - h...
blogs.atlassian.com/developerFlickr Attributions:Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/Blue Streak - h...
Structure for scale: Dialing in your apps for optimal performance
Structure for scale: Dialing in your apps for optimal performance
Upcoming SlideShare
Loading in …5
×

Structure for scale: Dialing in your apps for optimal performance

2,467 views
2,408 views

Published on

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,467
On SlideShare
0
From Embeds
0
Number of Embeds
1,742
Actions
Shares
0
Downloads
43
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Structure for scale: Dialing in your apps for optimal performance

    1. 1. Performance tuning for JavaapplicationsAndrew PrenticeAtlassian 2
    2. 2. Topics• How to make your apps run faster• Tuning ideas 3
    3. 3. A note• X vs Y will always depend on context• Test with your application! 4
    4. 4. You are here• How I did the testing • Hardware & software• Must have improvements • Jave, Hardware, Virtualization and Garbage Collection• Worth doing • Gigabit vs 100 mbit, Heap size tuning, use an SSD 5
    5. 5. The contenders• Dell PE 860 - “WALL-E”• Xeon X3220 2.4 Ghz Quad core • Quad Core, 2 x 4Mb L3• 8 Gb DDR2 RAM• 2 x 7.2K SAS Disks• 1 x Intel X25-E 32 GB SSD 6
    6. 6. The contenders• Dell PE 2950 - “Johnny 5”• 2 x Xeon E5 405 2.0 Ghz • Quad Core, 2 x 6M L3• 32 Gb RAM• 2 x 72 Gb 15 K Drives• ~$4000 7
    7. 7. The contenders• Dell PE R610 - “EVE”• 2 x Xeon E5520 2.2Ghz • Quad Core, 8M “SmartCache”• 32 Gb RAM• 2 x 146G 15K drives• ~$4000 8
    8. 8. The Workload• JIRA 4, MySQL 5, RHEL 5• Java 1.6, 3G heap, ParNew, ParOld GC• ~70,000 Issues, ~80 projects• ~30 reqs/sec, JMeter 2.3.4 (patched, JSR-223)• Traffic modeled on http://jira.atlassian.com with higher writes 9
    9. 9. About the graphs• Most show Average Response Time in milliseconds • Includes time to generate AND deliver the response • Does not include browser time• Arrows show if lower or higher is better 10
    10. 10. You are here• How I did the testing • Hardware & software• Must have improvements • Jave, Hardware, Virtualization and Garbage Collection• Worth doing • Gigabit vs 100 mbit, Heap size tuning, use an SSD 11
    11. 11. Must. Have. 12
    12. 12. Upgrade Java 1.5 to 1.6• Advances in complier/VM technology • Produces better runtime code • Able to use modern instructions • Able to use modern hardware 13
    13. 13. Java 1.5 vs 1.6 (Johnny 5) Java 1.5 Java 1.6.0 1500 1125 750 375 6.3X 0 Average Response Time (ms) 14
    14. 14. Java 1.5 vs 1.6 (Johnny 5) Java 1.5.0 Java 1.6.0 380 285 190 95 26% 0 Average CPU (%) 15
    15. 15. Upgrade Java to 1.6• HotSpot VM updates in 1.6• Advancing technology • Compressed OOPS on 64bit • Escape Analysis • NUMA• http://java.sun.com/performance/reference/ whitepapers/6_performance.html 16
    16. 16. Java 1.6 vs 1.6 (Eve) Java 1.6.0_7 Java 1.6.0_16 148 111 74 37 7% 0 Average Response Time (ms) 17
    17. 17. Hardware Upgrades• Hyper-Threading• QuickPath Interconnect• Nehalem (Eve) have on die DDR3 Memory Controllers• Memory throuput improvements - good for Java! Technology Speed DDR2-800 (dual channel) 6.4 GB/sec DDR3-1333 (dual channel) 21.3 GB/sec 18
    18. 18. Hyper Threading Thread 1 CPU Thread 2 Thread 1 runs until it hits a cache miss. Thread 2 runs on the same CPU while T1 is awaiting data 19
    19. 19. Hardware Upgrades 2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve) 2000 1500 1000 500 8-12X 0 Average Response Time (ms) 20
    20. 20. Improved GC Throughput 2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve) 30000 22500 15000 2-3X 7500 0 GC Throughput (MB/sec) 21
    21. 21. Garbage Collection• Java’s Virtual Machine manages memory• New objects are continually created during runtime• GC is the process of collecting dead objects to reclaim memory 22
    22. 22. Tune Garbage Collection• Getting your GC tuning wrong is bad• How bad?• Are the defaults sane? 23
    23. 23. Tune Garbage Collection CMS ParNew ParOld4000300020001000 2X 0 Wall-E (ms) Johnny 5 (ms) Eve (ms) Average Response Time 24
    24. 24. Tune Garbage Collection CMS ParNew ParOld500375250125 2X 0 Johnny 5 (ms) Eve (ms) Average Response Time 25
    25. 25. Tune Garbage Collection• Defaults are sane• Parallel New + Parallel Old is fastest• CMS prioritizes low pauses over CPU usage• CMS does not compact - can lead to heap fragmentation 26
    26. 26. Virtualisation• Does being in a VM hurt?• What if the risks are managed? 27
    27. 27. Virtualisation Real VMWare ESX 3.5 VMWare ESX 4i 170.0 127.5 85.0 42.5 43% 0 Average Response Time (ms) 28
    28. 28. Virtualisation• On average ~30% slower• Under extreme resource starvation, up to 100% slower• Avoid virtualization for high traffic instances• See the Atlassian docs for recommendations • http://confluence.atlassian.com/display/DOC/Running +Confluence+in+a+Virtualised+Environment 29 • http://confluence.atlassian.com/display/JIRA/Running+JIRA+in
    29. 29. You are here• How I did the testing • Hardware & software• Must have improvements • Java, Hardware, Virtualization and Garbage Collection• Worth doing • Gigabit vs 100 mbit, Heap size tuning, use an SSD 30
    30. 30. Maybe. 31
    31. 31. Gigabit Network vs 100Mbit• Gigabit is faster, sure• But 100Mb is OK if you’re not doing big transfers, RIGHT?! 32
    32. 32. Use Gigabit 100mb/sec 1000mb/sec 1900 1425 950 475 10% 0 Average Response Time (ms) 33
    33. 33. Use Gigabit• Lower RTT on Gigabit• If you have enough CPU power, keep the DB on localhost• Check your development environment 34
    34. 34. Use Gigabit 100Mbit 1000Mbit 200 150 100 50 26% 0 ReIndex Time (seconds) 35
    35. 35. Heap size tuning• Memory starvation is bad, but can we “drown” Java with too much memory? 36
    36. 36. Heap size - Wall-E 1.5G 3G 6G 2000 1500 1000 500 23% 0 Average Response Time 37
    37. 37. Heap size - Johnny 5 1.5G 3G 6G 15G 290.0 217.5 145.0 72.5 49% 0 Average Response Time 38
    38. 38. Heap size - Eve 1.5G 3G 6G 15G 170.0 127.5 85.0 42.5 28% 0 Average Response Time 39
    39. 39. Heap size tuning• Heap too small - bad• Heap too big - not bad!• Bigger heaps give the JVM more room to work• Diminishing returns from really big heaps 40
    40. 40. Buy an SSD• SSDs are FAST!• SSDs are EXPENSIVE!• SSDs are all the rage!• DBAs are finally happy (almost) 41
    41. 41. Buy an SSD SATA SSD800600400200 33% 0 Browse Issue Create Issue 42
    42. 42. Buy an SSD• Fast writes • 0.1ms Avg Service Time• Most improvement comes from SSD on the DB• Writing to the Lucene index means re-opening IndexReaders • Much faster on SSD 43• Lucene Search Term Cache fits in OS buffers
    43. 43. Remember!• Upgrade! • Java to 1.6 • Your server• Use Parallel Garbage Collection with a big heap• Remove virtualization if you can• Use 1000mbit network 44
    44. 44. ...Part Two:Environment andJoanna ThurmannJIRA Systems Administrator 2
    45. 45. Don’t let this happen to you! 3
    46. 46. “Step1 - We admitted we were powerless -our JIRA had become unmanageable.” JIRAcholics Anonymous 4
    47. 47. Contents• Performance Considerations• Hardware & Software Upgrade• JIRA and Plugin Upgrades• Structure for Scale 5
    48. 48. Our JIRA 6
    49. 49. Our JIRA• 210,000 issues 6
    50. 50. Our JIRA• 210,000 issues• 142 projects 6
    51. 51. Our JIRA• 210,000 issues• 142 projects• 66 workflows 6
    52. 52. Our JIRA• 210,000 issues• 142 projects• 66 workflows• 320 custom fields 6
    53. 53. Our JIRA• 210,000 issues• 142 projects• 66 workflows• 320 custom fields• 6,000 user directory (~400 concurrent) 6
    54. 54. Our JIRA• 210,000 issues• 142 projects• 66 workflows• 320 custom fields• 6,000 user directory (~400 concurrent)• 24x7 usage 6
    55. 55. Considerations forPerformance and Usability• Total Issue Count• User Concurrency• # and Complexity of Configurations• Integrations• Plugins 7
    56. 56. In General…• JIRA uses the disk much more than db (due to Lucene indexing)• JIRA needs considerable memory 8
    57. 57. Our Upgrade Summary Before After HP ProLiant DL580 G4  HP ProLiant DL380 G7 4 x Dual-Core processors - Intel Xeon  2 x 6-CORE processors - Intel Xeon X5680 7140M 3.4GHz 3.33GHz RAID5 array  5 x HP 300GB 10K HDD  RAID10 array  11 x HP 300GB 10K HDD 512 MB cache (25%R-75%W)  1GB cache (50%R-50%W) 3.5 GB for JIRA (8 GB total)  6 GB for JIRA (32 GB total) Windows Server 2003 Std Edition 64-bit  Windows Server 2008 R2 Std Edition 64-bit SQL Server 2005 Std Edition  SQL Server 2008 R2 Std Edition 9
    58. 58. Our Upgrade – Disk and CPU• Chose Raid10 over Raid5 for better performance• Upgraded Controller from 512 MB cache to 1GB cache• Chose 12 CPUs for concurrency 10
    59. 59. Our Upgrade – Memory 11
    60. 60. Our Upgrade – Memory• 3.5GB  6GB 11
    61. 61. Our Upgrade – Memory• 3.5GB  6GB• Full garbage collections times lower and less frequent 11
    62. 62. Our Upgrade – Memory• 3.5GB  6GB• Full garbage collections times lower and less frequent• Free up more memory 11
    63. 63. Our Upgrade – Memory• 3.5GB  6GB• Full garbage collections times lower and less frequent• Free up more memory• Eliminate out-of-memory errors 11
    64. 64. Our Upgrade - MemoryBefore (3.5 GB JIRA, 8 GB Total) After (6 GB JIRA, 32 GB Total)Full GC: 11-50s, 1x/hour Full GC: 3-7s, <5x/day2.5 – 3.25 GB avg. usage 3-5 GB avg. usage20-30% free 50-75% freeOut-of-memory: 1x/month Out-of-memory: 0 12
    65. 65. Our Upgrade - MemoryBefore (3.5 GB JIRA, 8 GB Total) After (6 GB JIRA, 32 GB Total)Full GC: 11-50s, 1x/hour Full GC: 3-7s, <5x/day2.5 – 3.25 GB avg. usage 3-5 GB avg. usage20-30% free 50-75% freeOut-of-memory: 1x/month Out-of-memory: 0 12
    66. 66. Our Upgrade - Software• SQL Server 2008 R2• OK to run JIRA and DB on the same server• Allocate sufficient memory to the db• Use the latest JDBC driver & JDK 13
    67. 67. What’s the bottom line? 14
    68. 68. What’s the bottom line? 14
    69. 69. Results – backend Then Now Percentage faster Total
Reindex 60
mins 25
mins 59% XML
Backup 30
mins 4
½
mins 85%SQL
Server
Backup 7
mins 2
mins
 71% Metrics
service 20
mins 6
mins 70%
Index
opNmizaNon 7+
mins 25
secs 94% Index
excepNons 3‐4
per
day 0 15
    70. 70. Results – front end Then Now Percentage faster Search
all
issues 3
secs 2
secs 34% Create
/
Edit
issue 4
secs 2
secs 50% 16
    71. 71. Plugin challenges we faced• Memory leak• End of support of *key* workflow plugin• Odd behavior in JIRA 4.1.1 upgrade• Performance issue with plugin 17
    72. 72. Plugin Install & Upgrades 18
    73. 73. Plugin Install & Upgrades• Scrutinize Plugins 18
    74. 74. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… 18
    75. 75. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… • Is the plugin reputable? 18
    76. 76. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… • Is the plugin reputable? • Will it be supported? 18
    77. 77. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… • Is the plugin reputable? • Will it be supported? • How widely used is it? 18
    78. 78. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… • Is the plugin reputable? • Will it be supported? • How widely used is it? • Does it perform well? 18
    79. 79. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… • Is the plugin reputable? • Will it be supported? • How widely used is it? • Does it perform well? • Do you really need it? 18
    80. 80. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… • Is the plugin reputable? • Will it be supported? • How widely used is it? • Does it perform well? • Do you really need it? • Does it introduce other bugs? 18
    81. 81. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… • Is the plugin reputable?• Get the latest release • Will it be supported? • How widely used is it? • Does it perform well? • Do you really need it? • Does it introduce other bugs? 18
    82. 82. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… • Is the plugin reputable?• Get the latest release • Will it be supported? • How widely used is it?• Test in QA environment • Does it perform well? • Do you really need it? • Does it introduce other bugs? 18
    83. 83. Plugin Install & Upgrades• Scrutinize Plugins Ask yourself… • Is the plugin reputable?• Get the latest release • Will it be supported? • How widely used is it?• Test in QA environment • Does it perform well? • Do you really need it?• Test! Test! Test! • Does it introduce other bugs? 18
    84. 84. Why Upgrade JIRA Regularly? 19
    85. 85. Why Upgrade JIRA Regularly?• Performance improvements & key fixes in each release for BOTH end-users AND admins 19
    86. 86. Why Upgrade JIRA Regularly?• Performance improvements & key fixes in each release for BOTH end-users AND admins• Example: in-place DB upgrades in 4.3 19
    87. 87. My Weekend Upgrade 20
    88. 88. My Weekend Upgrade Before 20
    89. 89. My Weekend Upgrade Before 20
    90. 90. My Weekend Upgrade Before After 20
    91. 91. My Weekend Upgrade Before After 20
    92. 92. Why else? 21
    93. 93. Why else?• Fewer internal support calls 21
    94. 94. Why else?• Fewer internal support calls• Less time to resolve them 21
    95. 95. Why else?• Fewer internal support calls• Less time to resolve them• Easier & faster for Atlassian to support you 21
    96. 96. Why else?• Fewer internal support calls• Less time to resolve them• Easier & faster for Atlassian to support you• Early bird gets the worm 21
    97. 97. Our last JIRA upgrade rocked! 22
    98. 98. Our last JIRA upgrade rocked! • Resolved 4 bugs found internally 22
    99. 99. Our last JIRA upgrade rocked! • Resolved 4 bugs found internally • Delivered 5 must-have features 22
    100. 100. Our last JIRA upgrade rocked! • Resolved 4 bugs found internally • Delivered 5 must-have features • Resulted in ~25% fewer support calls 22
    101. 101. Our last JIRA upgrade rocked! • Resolved 4 bugs found internally • Delivered 5 must-have features • Resulted in ~25% fewer support calls • Saved ~3hrs/week with JQL instead of SQL 22
    102. 102. Structure – For Startup JIRAs 23
    103. 103. Structure – For Startup JIRAs• Configurations  Less is more since JIRA caches them 23
    104. 104. Structure – For Startup JIRAs• Configurations  Less is more since JIRA caches them• Use fewer issue types, resolutions, statuses, custom fields 23
    105. 105. Structure – For Startup JIRAs• Configurations  Less is more since JIRA caches them• Use fewer issue types, resolutions, statuses, custom fields• Use custom field contexts vs. new fields 23
    106. 106. Structure – For Startup JIRAs• Configurations  Less is more since JIRA caches them• Use fewer issue types, resolutions, statuses, custom fields• Use custom field contexts vs. new fields• Use a simple workflow and apply to many projects 23
    107. 107. Structure – Project Roles 24
    108. 108. Structure – Project Roles• Use Project Roles vs individual names & groups in permission schemes 24
    109. 109. Structure – Project Roles• Use Project Roles vs individual names & groups in permission schemes Before Permission Scheme for Project 1 Administer Projects Group (“”Project 1 Admins”) Edit Issues Group(“Project 1 Team”) 24
    110. 110. Structure – Project Roles• Use Project Roles vs individual names & groups in permission schemes Before After Global Permission Scheme Permission Scheme for Project 1 Administer Projects Project Role (Administrators) Administer Projects Group (“”Project 1 Admins”) Edit Issues Project Role (Developers) Edit Issues Group(“Project 1 Team”) 24
    111. 111. Structure – Project Roles• Use Project Roles vs individual names & groups in permission schemes Before After Global Permission Scheme Permission Scheme for Project 1 Administer Projects Project Role (Administrators) Administer Projects Group (“”Project 1 Admins”) Edit Issues Project Role (Developers) Edit Issues Group(“Project 1 Team”) Project Roles for Project 1 Administrators “Project 1 Admins” Developers “Project 1 Team” 24
    112. 112. Structure – Project Roles• Use Project Roles vs individual names & groups in permission schemes Before After Global Permission Scheme Permission Scheme for Project 1 Administer Projects Project Role (Administrators) Administer Projects Group (“”Project 1 Admins”) Edit Issues Project Role (Developers) Edit Issues Group(“Project 1 Team”) Project Roles for Project 1 Administrators “Project 1 Admins” Developers “Project 1 Team” 24
    113. 113. Structure – For Epic JIRAs 25
    114. 114. Structure – For Epic JIRAs• Consolidate any and all schemes 25
    115. 115. Structure – For Epic JIRAs• Consolidate any and all schemes• Use Scheme Tools: e.g. Scheme Comparison Tool, Group to Project Role Mapping Tool 25
    116. 116. Structure – For Epic JIRAs• Consolidate any and all schemes• Use Scheme Tools: e.g. Scheme Comparison Tool, Group to Project Role Mapping Tool• ‘Archive’ projects - hide them from view of users, so that they don’t clutter the menus. Create ‘Archived’ config schemes 25
    117. 117. We Declutter 26
    118. 118. We Declutter• Created project naming conventions 26
    119. 119. We Declutter• Created project naming conventions• Eliminated & consolidated custom fields (25 out of 345) 26
    120. 120. We Declutter• Created project naming conventions• Eliminated & consolidated custom fields (25 out of 345)• Took custom fields out of global context 26
    121. 121. We Declutter• Created project naming conventions• Eliminated & consolidated custom fields (25 out of 345)• Took custom fields out of global context• Eliminated issue types, status, resolution choices (15-30) 26
    122. 122. We Tidy 27
    123. 123. We Tidy• Super User (SU) plugin 27
    124. 124. We Tidy• Super User (SU) plugin• Easier to troubleshoot user issues 27
    125. 125. We Tidy• Super User (SU) plugin• Easier to troubleshoot user issues• Helps us “terminate” JIRA accounts •Delete their dashboards •Delete their personal filters 27
    126. 126. We Tidy• Super User (SU) plugin• Easier to troubleshoot user issues• Helps us “terminate” JIRA accounts •Delete their dashboards •Delete their personal filters •Delete subscriptions (the “silent killer”) 27
    127. 127. We Train 28
    128. 128. We Train• Limit General Permissions & train the users on how/when to use 28
    129. 129. We Train• Limit General Permissions & train the users on how/when to use • Bulk Change 28
    130. 130. We Train• Limit General Permissions & train the users on how/when to use • Bulk Change • Create Shared Objects (filters + dashboards) 28
    131. 131. We Train• Limit General Permissions & train the users on how/when to use • Bulk Change • Create Shared Objects (filters + dashboards) • Manage Group Filter Subscriptions 28
    132. 132. We Train• Limit General Permissions & train the users on how/when to use • Bulk Change • Create Shared Objects (filters + dashboards) • Manage Group Filter Subscriptions• Train project administrators / super-users 28
    133. 133. We Do More 29
    134. 134. We Do More• Use db-specific vs. XML-backup 29
    135. 135. We Do More• Use db-specific vs. XML-backup• Schedule regular maintenance 29
    136. 136. We Do More• Use db-specific vs. XML-backup• Schedule regular maintenance• Regularly check logs for errors 29
    137. 137. We Do More• Use db-specific vs. XML-backup• Schedule regular maintenance• Regularly check logs for errors• Read the JIRA performance doc 29
    138. 138. 30
    139. 139. We Keep Going 30
    140. 140. “Step12 - we promised to help otherJIRAcholics, and to always practice what wepreach JIRAcholics Anonymous 31
    141. 141. A word from my sponsors….
    142. 142. A word from my sponsors….
    143. 143. A word from my sponsors…. Polycom, Inc. (Nasdaq: PLCM) is a global leader in unified communicationssolutions with industry-leading telepresence, video, voice, and infrastructure solutions built on open standards. Polycom powers smarter conversations, transforming lives and businesses worldwide. Please visit www.polycom.com for more information.
    144. 144. blogs.atlassian.com/developerFlickr Attributions:Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/
    145. 145. blogs.atlassian.com/developerFlickr Attributions:Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/

    ×