My updated slides about the journey to hell and back to normality wrt automated tests at scale. Based on real 10+ years experience of JIRA development teams.
I delivered this talk at XPDays in Kiev in October 2013.
2. About me
• Coding for 30 years, now only in "free time"
• Agile Practices (inc.TDD) since 2003
• Dev Nerd,Tech Leader,Agile Coach,
Speaker, PHB
• 6 years with Atlassian (JIRA Dev Manager)
• Spartez Co-founder & CEO
19. Build Tiers and Policy
Tier A1 - green soon after all commits
Tier A2 - green at the end of the day
Tier A3 - green at the end of the iteration
unit tests and functional* tests
WebDriver and bundled plugins tests
supported platforms tests, compatibility tests
21. Extensive Training
• assertThat over assertTrue/False and
assertEquals
• avoiding races - Atlassian Selenium with its
TimedElement
• Favouring unit tests over functional tests
• Promoting Page Objects
• Brownbags, blogs, code reviews
28. Ditching - benefits
• Freed build agents - better system throughput
• Boosted morale
• Gazillion of developer hours saved
• Money saved on infrastructure
29. Ditching - due diligence
• conducting the audit - analysis of the
coverage we lost
• determining which tests needs to rewritten
(e.g. security related)
• rewriting some of the tests (good job for
new hires + a senior mentor)
30. Flaky Browser-based Tests
Races between test code and asynchronous page logic
Playing with "loading" CSS class does not really help
31. Races Removal with Tracing
// in the browser:
function mySearchClickHandler() {
doSomeXhr().always(function() {
// This executes when the XHR has completed (either success or failure)
JIRA.trace("search.completed");
});
}
// In production code JIRA.trace is a no-op
// in my page object:
@Inject
TraceContext traceContext;
public SearchResults doASearch() {
Tracer snapshot = traceContext.checkpoint();
getSearchButton().click(); // causes mySearchClickHandler to be invoked
// This waits until the "search.completed"
// event has been emitted, *after* previous snapshot
traceContext.waitFor(snapshot, "search.completed");
return pageBinder.bind(SearchResults.class);
}
42. Agent Availability/Setup
• starved builds due to
busy agents building
very long builds
• time synchronization
issue - NTPD problem
Fixes applied
43. • Proximity of SCM repo
• shallow git clones are not so fast and lightweight +
generating extra git server CPU load
• git clone per agent/plan + git pull + git clone per build
(hard links!)
• Atlassian Stash was thankful (queue)
SCM Update - Checkout time
2 min → 5 seconds
Trade disk space
for speed
46. Compilation
• Restructuring multi-pom maven project
and dependencies
• Maven 3 parallel compilation FTW
-T 1.5C
*optimal factor thanks to scientific trial and error research
7 min → 1 min
47. Unit Test Execution
• Splitting unit tests into 2 buckets: good and
legacy (much longer)
• Maven 3 parallel test execution (-T 1.5C)
7 min → 5 min
3000 poor tests
(5min)
11000 good tests
(1.5min)
48. Functional Tests
• Selenium 1 removal did help
• Faster reset/restore (avoid unnecessary
stuff, intercepting SQL operations for debug
purposes - building stacktraces is costly)
• Restoring via Backdoor REST API
• Using REST API for common setup/
teardown operations
50. Publishing Results
• Server log allocation per test → using now
Backdoor REST API (was Selenium)
• Bamboo DB performance degradation for
rich build history - to be addressed
1 min → 40 s
51. Unexpected Problem
• Stability issues with our CI server
• The bottleneck changed from I/O to CPU
• Too many agents per physical machine
53. Improvements Summary
Tests Before After Improvement %
Unit tests 29 min 17 min 41%
Functional tests 56 min 34 min 39%
WebDriver tests 39 min 21 min 46%
Overall 124 min 72 min 42%
* Additional ca. 5% improvement expected once new git clone
strategy is consistently rolled-out everywhere
57. Codebase Split - Problems
• Organizational concerns - understanding,
managing, integrating, releasing
• Mindset change - if something worked for
10 years why to change it?
• We damned ourselves with big buckets for
all tests - where do they belong to?
58. Splitting code base
• Step 0 - JIRA Importers Plugin (3.5 years ago)
• Step 1- New IssueView and Navigator
JIRA 6.0
59. We are still escaping hell.
Hell sucks in your soul.
60. Conclusions
• Visibility and problem awareness help
• Maintaing huge testbed is difficult and costly
• Measure the problem, measure improvements
• No prejudice - no sacred cows
• Automated tests are not one-off investment,
it's a continuous journey
• Performance is a damn important feature
62. Interested in such stuff?
Talk to me at the conference or visit http://www.spartez.com/careers
We are hiring in Gdańsk
63. • Turtle - by Jonathan Zander, CC-BY-SA-3.0
• Loading - by MatthewJ13, CC-SA-3.0
• Magic Potion - by Koolmann1, CC-BY-SA-2.0
• Merlin Tool - by By L. Mahin, CC-BY-SA-3.0
• Choose Pills - by *rockysprings, CC-BY-SA-3.0
• Flashing Red Light - bt Chris Phan, CC BY 2.0
Images - Credits