ICT role in 21st century education and its challenges
The (not so) Dark Art of Atlassian Performance Tuning
1. The (Not So) Dark Art
of Atlassian Performance Tuning
Standard horizontal view logo.
Purposes: Website, Blogs, External PRs,
External websites, Event websites.
Dan Hardiker • CTO • @adaptavist
Sma
Purpose
4. Who do I think you are?!
• Users
• Application Administrators
• System Administrators
Logo centralis
Purpose: Events, stall head
5. Out of Scope
• Performance Testing
• Load / Capacity
• Tools in Action
• List of Solutions
... and lots more!
Logo centralis
Purpose: Events, stall head
24. In Summary
• Build a Usage Profile
• Identify power & automated users
• Understand the business processes
that the applications support
Logo centralis
Purpose: Events, stall head
39. In Summary
• Use the Latest Supported JVM
• Integrate with the OS
• Monitor all the things
• Virtualize with Care
• Define the Scale Required
Logo centralis
Purpose: Events, stall head
56. Where is the time going?
• Caching
• Encryption
• Compression
• Transport
• Page Generation
• Connection Queuing
• Ordering
• Browser Rendering
Logo centralis
Purpose: Events, stall head
58. Averaging Page Generation Time
• Obtain from Access Logs
• Average over a time window
• Resolve into a Usage Profile
• Define use cases, measure APGT for each
• Use Usage Profile scale for load test setup
Logo centralis
Purpose: Events, stall head
62. Logging
• Access Logging
• Apache & Application
• Time-To-First-Byte timing
• Database Logging
• Log4JDBC
• Database Server Tooling
Logo centralis
Purpose: Events, stall head
63. Profiling
• Application Execution
• Atlassian Built-in Profiler
• Targetted Timing
• AOP or Plugins with Perf4J
• Sub-millisecond is usually unreliable
Logo centralis
Purpose: Events, stall head
73. Better Problem Statement
• Every so often the system becomes unresponsive.
• The system recovers given some time.
Logo centralis
Purpose: Events, stall head
74. System Analysis
• System shows kernel times are low
• jstat -gccause shows STW GC is active at pauses
Logo centralis
Purpose: Events, stall head
75. Garbage Collection Analysis
• GREP / SED / AWK
• Analysers
• HP JMeter
• GCViewer (https://github.com/chewiebug)
• jClarity Censum
Logo centralis
Purpose: Events, stall head
78. Findings
• Use the latest supported Oracle JVM
• Understand & Size the Generations
• Use appropriate collectors
• Use NUMA & TLAB, if possible
• LargePages/HugePages stop swapping
• Except when they don’t!
Logo centralis
Purpose: Events, stall head
79. Next Steps
• Upgrade OS to gain NUMA support
• Repeat testing to measure performance change
• Roll out one thing at a time
Logo centralis
Purpose: Events, stall head
81. Anti-patterns
• Stress is Stressful
• Read: why zebra’s don’t get ulcers
• Actions Taken Prematurely
• “Doing something is better than nothing”
Logo centralis
Purpose: Events, stall head
82. Shot in the Dark
• Conclusions drawn from partial data
• Changes implemented rashly
Logo centralis
Purpose: Events, stall head
83. Knee Jerk Analysis
• OutOfMemoryError seen in log
• System is GCing heavily
• It needs more memory!
• Action: double the JVM heap
• Outcome: no change - problem recurred as before
Logo centralis
Purpose: Events, stall head
84. What was missed?
• System had been stable for 2 weeks
• Usage Profile had not varied significantly
• All HTTP threads waiting on the database pool
• All above threads had a common code path
Logo centralis
Purpose: Events, stall head
85. A Better Approach
• Database connection pool monitoring
• DBCP
• C3P0
• Long running queries will tie up connections
• Unreturned / Unused connections can be leaks
Logo centralis
Purpose: Events, stall head
89. Results
• Team Calendars had been upgraded
• New JIRA integration increased database
activity
• System Requirements changed
• Larger DB pool size needed
• Pool increased from 30 to 250
• We didn’t just “make it bigger”!
Logo centralis
Purpose: Events, stall head
97. Sometimes …
• Nobody is available
• Time is too short
• Downtime is costing money
• You need an external opinion
• You need a framework to work by
• The problem is too big
Logo centralis
• You just need a helping hand
Purpose: Events, stall head
100. If you only remember 1 thing:
• Ensure you’ve JFDI
• Be Methodical
Logo centralis
Purpose: Events, stall head
101. If you only remember 2 things:
• Ensure you’ve JFDI
• Be Methodical
• Measure, Don’t Guess
Logo centralis
Purpose: Events, stall head
102. Amongst the things to remember
• Ensure you’ve JFDI
• Be Methodical
• Measure, Don’t Guess
Logo centralis
Purpose: Events, stall head
103. Einstein said something like:
“If I had an hour to solve a
problem, I’d spend 55 minutes
defining the problem, and 5
minutes thinking about the
solution”
Logo centralis
Purpose: Events, stall head
107. One Last Thing
If you knew all this and we bored you…
…or if you’re just really, really good…
We’re hiring in the USA, UK and Germany.
Visit www.adaptavist.com
Logo centralis
Purpose: Events, stall head
108. Thank you!
Standard horizontal view logo.
Purposes: Website, Blogs, External PRs,
External websites, Event websites.
Dan Hardiker • CTO • @adaptavist
Sma
Purpose
109. Thank you!
Standard horizontal view logo.
Purposes: Website, Blogs, External PRs,
External websites, Event websites.
Dan Hardiker • CTO • @adaptavist
Sma
Purpose
