My updated slides about the journey to hell and back to normality wrt automated tests at scale. Based on real 10+ years experience of JIRA development teams. I delivered this talk at XPDays in Kiev in October 2013.

1. Automated Test Hell
Wojciech Seliga
wojciech.seliga@spartez.com
@wseliga
Our Journey

2. About me
• Coding for 30 years, now only in "free time"
• Agile Practices (inc.TDD) since 2003
• Dev Nerd,Tech Leader,Agile Coach,
Speaker, PHB
• 6 years with Atlassian (JIRA Dev Manager)
• Spartez Co-founder & CEO

3. XP Promise
CostofChange
Time
Waterfall
XP

4. 1.5 year ago

5. Almost 10 years
of accumulating
garbage automatic tests

6. About 20 000 tests
on all levels of abstraction

7. Very slow (even hours)
and fragile feedback loop

8. Serious performance and
reliability issues

9. Dispirited devs
accepting RED as a norm

10. Feedback
Speed
`
Test
Quality

11. Test Code is Not
Trash
Design
Maintain
Refactor
Share
Review
Prune
Respect
Discuss
Restructure

12. Test Pyramid
Unit Tests (including QUnit)
REST / HTML Tests
Selenium

13. Optimum Balance
Isolation Speed Coverage Level Access Effort

14. Dangerous to temper with
MaintainabilityQuality / Determinism

15. Now

16. People - Motivation
Making GREEN the norm

17. Shades of Red
Shades of Green

18. Pragmatic CI Health

19. Build Tiers and Policy
Tier A1 - green soon after all commits
Tier A2 - green at the end of the day
Tier A3 - green at the end of the iteration
unit tests and functional* tests
WebDriver and bundled plugins tests
supported platforms tests, compatibility tests

20. Wallboards:
Constant
Awareness

21. Extensive Training
• assertThat over assertTrue/False and
assertEquals
• avoiding races - Atlassian Selenium with its
TimedElement
• Favouring unit tests over functional tests
• Promoting Page Objects
• Brownbags, blogs, code reviews

22. Quality

23. Automatic Flakiness Detection
Quarantine
Re-run failed tests and see if they pass

24. Quarantine - Healing

25. SlowMo - expose races

26. Selenium 1

27. Selenium ditching
Sky did not fall in

28. Ditching - benefits
• Freed build agents - better system throughput
• Boosted morale
• Gazillion of developer hours saved
• Money saved on infrastructure

29. Ditching - due diligence
• conducting the audit - analysis of the
coverage we lost
• determining which tests needs to rewritten
(e.g. security related)
• rewriting some of the tests (good job for
new hires + a senior mentor)

30. Flaky Browser-based Tests
Races between test code and asynchronous page logic
Playing with "loading" CSS class does not really help

31. Races Removal with Tracing
// in the browser:
function mySearchClickHandler() {
doSomeXhr().always(function() {
// This executes when the XHR has completed (either success or failure)
JIRA.trace("search.completed");
});
}
// In production code JIRA.trace is a no-op
// in my page object:
@Inject
TraceContext traceContext;
public SearchResults doASearch() {
Tracer snapshot = traceContext.checkpoint();
getSearchButton().click(); // causes mySearchClickHandler to be invoked
// This waits until the "search.completed"
// event has been emitted, *after* previous snapshot
traceContext.waitFor(snapshot, "search.completed");
return pageBinder.bind(SearchResults.class);
}

32. Can we halve our build times?
Speed

33. Parallel Execution - Theory
End of Build
Batches
Start of Build Time

34. Parallel Execution
End of Build
Batches
Start of Build Time

35. Parallel Execution -
Reality Bites
End of Build
Batches
Start of Build
Agent
availability
Time

36. Dynamic Test Execution
Dispatch - Hallelujah

37. "You can't manage what
you can't measure."
not by W. Edwards Deming
If you believe just in it
you are doomed.

38. You can't improve something
if you can't measure it
Profiler, Build statistics, Logs, statsd → Graphite

39. Anatomy of Build*
Compilation
Packaging
Executing Tests
Fetching Dependencies
*Any resemblance to maven build is entirely accidental
SCM Update
Agent Availability/Setup
Publishing Results

40. JIRA Unit Tests Build
Compilation (7min)
Packaging (0min)
Executing Tests (7min)
Fetching Dependencies (1.5min)
SCM Update (2min)
Agent Availability/Setup (mean 10min)
Publishing Results (1min)

41. Decreasing
Test Execution Time
to ZERRO
alone would not let us
achieve our goal!

42. Agent Availability/Setup
• starved builds due to
busy agents building
very long builds
• time synchronization
issue - NTPD problem
Fixes applied

43. • Proximity of SCM repo
• shallow git clones are not so fast and lightweight +
generating extra git server CPU load
• git clone per agent/plan + git pull + git clone per build
(hard links!)
• Atlassian Stash was thankful (queue)
SCM Update - Checkout time
2 min → 5 seconds
Trade disk space
for speed

45. • Fix Predator
• Sandboxing/isolation agent trade-off:
rm -rf $HOME/.m2/repository/com/atlassian/*
into
find $HOME/.m2/repository/com/atlassian/
-name “*SNAPSHOT*” | xargs rm
• Network hardware failure found
(dropping packets)
Fetching Dependencies
1.5 min → 10 seconds

46. Compilation
• Restructuring multi-pom maven project
and dependencies
• Maven 3 parallel compilation FTW
-T 1.5C
*optimal factor thanks to scientific trial and error research
7 min → 1 min

47. Unit Test Execution
• Splitting unit tests into 2 buckets: good and
legacy (much longer)
• Maven 3 parallel test execution (-T 1.5C)
7 min → 5 min
3000 poor tests
(5min)
11000 good tests
(1.5min)

48. Functional Tests
• Selenium 1 removal did help
• Faster reset/restore (avoid unnecessary
stuff, intercepting SQL operations for debug
purposes - building stacktraces is costly)
• Restoring via Backdoor REST API
• Using REST API for common setup/
teardown operations

49. Functional Tests
We like this trend

50. Publishing Results
• Server log allocation per test → using now
Backdoor REST API (was Selenium)
• Bamboo DB performance degradation for
rich build history - to be addressed
1 min → 40 s

51. Unexpected Problem
• Stability issues with our CI server
• The bottleneck changed from I/O to CPU
• Too many agents per physical machine

52. JIRA Unit Tests Build Improved
Compilation (1min)
Packaging (0min)
Executing Tests (5min)
Fetching Dependencies (10sec)
SCM Update (5sec)
Agent Availability/Setup (3min)*
Publishing Results (40sec)

53. Improvements Summary
Tests Before After Improvement %
Unit tests 29 min 17 min 41%
Functional tests 56 min 34 min 39%
WebDriver tests 39 min 21 min 46%
Overall 124 min 72 min 42%
* Additional ca. 5% improvement expected once new git clone
strategy is consistently rolled-out everywhere

54. The Quality Follows

55. But that's still bad
We want CI feedback loop in a few minutes maximum

56. Splitting The Codebase

57. Codebase Split - Problems
• Organizational concerns - understanding,
managing, integrating, releasing
• Mindset change - if something worked for
10 years why to change it?
• We damned ourselves with big buckets for
all tests - where do they belong to?

58. Splitting code base
• Step 0 - JIRA Importers Plugin (3.5 years ago)
• Step 1- New IssueView and Navigator
JIRA 6.0

59. We are still escaping hell.
Hell sucks in your soul.

60. Conclusions
• Visibility and problem awareness help
• Maintaing huge testbed is difficult and costly
• Measure the problem, measure improvements
• No prejudice - no sacred cows
• Automated tests are not one-off investment,
it's a continuous journey
• Performance is a damn important feature

61. Revised XP Promise
CostofChange
Time
Waterfall
XP
Sad Reality

62. Interested in such stuff?
Talk to me at the conference or visit http://www.spartez.com/careers
We are hiring in Gdańsk

63. • Turtle - by Jonathan Zander, CC-BY-SA-3.0
• Loading - by MatthewJ13, CC-SA-3.0
• Magic Potion - by Koolmann1, CC-BY-SA-2.0
• Merlin Tool - by By L. Mahin, CC-BY-SA-3.0
• Choose Pills - by *rockysprings, CC-BY-SA-3.0
• Flashing Red Light - bt Chris Phan, CC BY 2.0
Images - Credits

64. ThankYou!
Tweet your feedback at @wseliga