Slides from my talk at 33rd Degree 2013 Conference in Warsaw. More than year ago we faced the fact that we are hitting the wall with our large scale automated testing in Atlassian JIRA. We analysed the problems and possible solutions and shared them with community at 33rd Degree in 2012. Since then we've implemented a lot of our ideas and come up with new, learnt new quite unexpected things and got rid of Selenium 1 completely. This session shows the learnings from our journey – escaping from Test Hell – back to the normality. If you are interested to hear what problems you can (and probably will) face if you have thousands of automated tests on on levels of abstractions (functional, integration, unit, UI, performance) and what solutions can be applied to remedy them – this presentation is for you.

1. Main sponsor
Escaping Automated Test Hell
One year later...
Wojciech Seliga

2. About me
• Coding for 30 years
• Agile Practices (inc. TDD) since 2003
• Dev Nerd, Tech Leader, Agile Coach,
Speaker
• 5+ years with Atlassian (JIRA Development
Team Lead)
• Spartez Co-founder

3. Year ago - recap

4. 18 000 tests on all levels
Very slow and fragile
feedback loop

5. Serious performance and
reliability issues

6. Feedback Test
Speed Quality
`

7. Respect Restructure
Design Share
Prune
Test Code is Not
Trash
Refactor
Maintain
Review
Discuss

8. Optimum Balance

9. Optimum Balance
Isolation

10. Optimum Balance
Isolation Speed

11. Optimum Balance
Isolation Speed Coverage

12. Optimum Balance
Isolation Speed Coverage Level

13. Optimum Balance
Isolation Speed Coverage Level Access

14. Optimum Balance
Isolation Speed Coverage Level Access Effort

15. Dangerous to temper with

16. Dangerous to temper with
Quality / Determinism

17. Dangerous to temper with
Quality / Determinism Maintainability

18. Splitting codebase is
key aspect of short
test feedback loop

19. Now

20. People - Motivation

21. Shades of Red

22. Pragmatic CI Health

23. Build Tiers and Policy
Tier A1 - green soon after all commits
unit tests and functional* tests
Tier A2 - green at the end of the day
WebDriver and bundled plugins tests
Tier A3 - green at the end of the iteration
supported platforms tests, compatibility tests

24. Wallboards:
Constant
Awareness

25. Training
• assertThat over assertTrue/False and
assertEquals
• avoiding races - Atlassian Selenium with its
TimedElement
• Unit tests over functional tests
• Brownbags, blogs, code reviews

26. Quality

27. Re-run failed tests and see if they pass
Automatic Flakiness Detection
Quarantine

28. Quarantine - Healing

29. SlowMo - expose races

30. Selenium 1

31. Selenium 1

32. Selenium ditching
Sky did not fall in

33. Ditching - benefits
• Freed build agents - better system
throughput
• Boosted morale
• Gazillion of developer hours saved
• Money saved on infrastructure

34. Ditching - due diligence
• conducting the audit - analysis of the
coverage we lost
• determining which tests needs to rewritten
(e.g. security related)
• rewriting the tests

35. Flaky Browser-based Tests
Races between test code and asynchronous page logic
Playing with "loading" CSS class does not really help

36. Races Removal with Tracing
// in the browser:
function mySearchClickHandler() {
doSomeXhr().always(function() {
// This executes when the XHR has completed (either success or failure)
JIRA.trace("search.completed");
});
}
// In production code JIRA.trace is a no-op
// in my page object:
@Inject
TraceContext traceContext;
public SearchResults doASearch() {
Tracer snapshot = traceContext.checkpoint();
getSearchButton().click(); // causes mySearchClickHandler to be invoked
// This waits until the "search.completed"
// event has been emitted, *after* previous snapshot
traceContext.waitFor(snapshot, "search.completed");
return pageBinder.bind(SearchResults.class);
}

37. Speed

38. Speed
Can we halve our build times?

39. Parallel Execution - Theory
Batches
Start of Build End of Build

40. Parallel Execution
Batches
Start of Build End of Build

41. Parallel Execution -
Agent
availability
Reality Bites
Batches
Start of Build End of Build

42. Dynamic Test Execution
Dispatch - Hallelujah

43. Dynamic Test Execution
Dispatch - Hallelujah

44. "You can't manage what
you can't measure."
W. Edwards Deming

45. If y
ou
you bel
ieve
"Youare manage what
can't just
doo i i
you can't measure."n
me t
d.
W. Edwards Deming

46. You can't improve something
if you can't measure it

47. You can't improve something
if you can't measure it
Profiler, Build statistics, Logs, statsd → Graphite

48. Compilation
Packaging
Executing Tests
Anatomy of Build*

49. Fetching Dependencies
Compilation
Packaging
Executing Tests
Anatomy of Build*

50. Fetching Dependencies
Compilation
Packaging
Executing Tests
Anatomy of Build*
*Any resemblance to maven build is entirely accidental

51. Fetching Dependencies
Compilation
Packaging
SCM Update Executing Tests
Anatomy of Build*
*Any resemblance to maven build is entirely accidental

52. Agent Availability/Setup
Fetching Dependencies
Compilation
Packaging
SCM Update Executing Tests
Anatomy of Build*
*Any resemblance to maven build is entirely accidental

53. Agent Availability/Setup
Fetching Dependencies
Compilation
Packaging Publishing Results
SCM Update Executing Tests
Anatomy of Build*
*Any resemblance to maven build is entirely accidental

54. Compilation (7min)
JIRA Unit Tests Build

55. Compilation (7min)
Packaging (0min)
JIRA Unit Tests Build

56. Compilation (7min)
Packaging (0min)
Executing Tests (7min)
JIRA Unit Tests Build

57. Compilation (7min)
Publishing Results (1min)
Packaging (0min)
Executing Tests (7min)
JIRA Unit Tests Build

58. Compilation (7min)
Publishing Results (1min)
Packaging (0min)
Executing Tests (7min)
Fetching Dependencies (1.5min)
JIRA Unit Tests Build

59. Compilation (7min)
Publishing Results (1min)
SCM Update (2min) Packaging (0min)
Executing Tests (7min)
Fetching Dependencies (1.5min)
JIRA Unit Tests Build

60. Agent Availability/Setup (mean 10min)
Compilation (7min)
Publishing Results (1min)
SCM Update (2min) Packaging (0min)
Executing Tests (7min)
Fetching Dependencies (1.5min)
JIRA Unit Tests Build

61. Decreasing Test
Execution Time to
ZERRO
alone would not let us
achieve our goal!

62. Agent Availability/Setup
• starved builds due to
busy agents building
very long builds
• time synchronization
issue - NTPD problem

63. SCM Update - Checkout time
• Proximity of SCM repo
• shallow git clones are not so fast and lightweight +
generating extra git server CPU load
• git clone per agent/plan + git pull + git clone per build
(hard links!)
• Stash was thankful (queue)

64. SCM Update - Checkout time
• Proximity of SCM repo
• shallow git clones are not so fast and lightweight +
generating extra git server CPU load
• git clone per agent/plan + git pull + git clone per build
(hard links!)
• Stash was thankful (queue)
2 min → 5 seconds

66. Fetching Dependencies
• Fix Predator
• Sandboxing/isolation agent trade-off:
rm -rf $HOME/.m2/repository/com/atlassian/*
into
find $HOME/.m2/repository/com/atlassian/
-name “*SNAPSHOT*” | xargs rm
• Network hardware failure found
(dropping packets)

67. Fetching Dependencies
• Fix Predator
• Sandboxing/isolation agent trade-off:
rm -rf $HOME/.m2/repository/com/atlassian/*
into
find $HOME/.m2/repository/com/atlassian/
-name “*SNAPSHOT*” | xargs rm
• Network hardware failure found
(dropping packets)
1.5 min → 10 seconds

68. Compilation
• Restructuring multi-pom maven project
and dependencies
• Maven 3 parallel compilation FTW
-T 1.5C
*optimal factor thanks to scientific trial and error research

69. Compilation
• Restructuring multi-pom maven project
and dependencies
• Maven 3 parallel compilation FTW
-T 1.5C
*optimal factor thanks to scientific trial and error research
7 min → 1 min

70. Unit Test Execution
• Splitting unit tests into 2 buckets: good and
legacy (much longer)
• Maven 3 parallel test execution (-T 1.5C)
3000 poor tests 11000 good tests
(5min) (1.5min)

71. Unit Test Execution
• Splitting unit tests into 2 buckets: good and
legacy (much longer)
• Maven 3 parallel test execution (-T 1.5C)
3000 poor tests 11000 good tests
(5min) (1.5min)
7 min → 5 min

72. Functional Tests
• Selenium 1 removal did help
• Faster reset/restore (avoid unnecessary
stuff, intercepting SQL operations for debug
purposes - building stacktraces is costly)
• Restoring via Backdoor REST API
• Using REST API for common setup/
teardown operations

73. Functional Tests

74. Publishing Results
• Server log allocation per test → using now
Backdoor REST API (was Selenium)
• Bamboo DB performance degradation for
rich build history - to be addressed

75. Publishing Results
• Server log allocation per test → using now
Backdoor REST API (was Selenium)
• Bamboo DB performance degradation for
rich build history - to be addressed
1 min → 40 s

76. Unexpected Problem
• Stability Issues with our CI server
• The bottleneck changed from I/O to CPU
• Too many agents per physical machine

77. Compilation (1min)
JIRA Unit Tests Build Improved

78. Compilation (1min)
Packaging (0min)
JIRA Unit Tests Build Improved

79. Compilation (1min)
Packaging (0min)
Executing Tests (5min)
JIRA Unit Tests Build Improved

80. Compilation (1min)
Packaging (0min)
Publishing Results (40sec)
Executing Tests (5min)
JIRA Unit Tests Build Improved

81. Compilation (1min)
Packaging (0min)
Publishing Results (40sec)
Executing Tests (5min)
Fetching Dependencies (10sec)
JIRA Unit Tests Build Improved

82. Compilation (1min)
Packaging (0min)
Publishing Results (40sec)
SCM Update (5sec) Executing Tests (5min)
Fetching Dependencies (10sec)
JIRA Unit Tests Build Improved

83. Agent Availability/Setup (3min)*
Compilation (1min)
Packaging (0min)
Publishing Results (40sec)
SCM Update (5sec) Executing Tests (5min)
Fetching Dependencies (10sec)
JIRA Unit Tests Build Improved

84. Improvements Summary
Tests Before After Improvement %
Unit tests 29 min 17 min 41%
Functional tests 56 min 34 min 39%
WebDriver tests 39 min 21 min 46%
Overall 124 min 72 min 42%
* Additional ca. 5% improvement expected once new git clone
strategy is consistently rolled-out everywhere

85. The Quality Follows

86. The Quality Follows

87. The Quality Follows

88. But that's still bad
We want CI feedback loop in a few minutes maximum

89. Splitting The Codebase

90. Resistance against splitting
The last attempt: Magic Machine
Decide with high confidence (e.g. > 95%) which subset of tests
to run basing on the committed changes

91. Magic Machine
• Looking at Bamboo history (analysing
correlation between changes and failures)
• Matching: package test/prod code and
transitive imports
• Code instrumentation (Clover, Emma, AspectJ)
• Run most often failing first

92. Inevitable Split - Fears
• Organizational concerns - understanding,
managing, integrating, releasing
• Mindset change - if something worked for
10 years why to change it?
• We damned ourselves with big buckets for
all tests - where do they belong to?

93. Magic Machine strikes back
With heavy use of brain, common sense
and expert judgement

94. Splitting code base
• Step 0 - JIRA Importers Plugin (3 years ago)
• Step 1- New Issue View and Navigator
JIRA
6 .0

95. We are still escaping hell.
Hell sucks in your soul.

96. Conclusions
• Visibility and problem awareness help
• Maintaing huge testbed is difficult and costly
• Measure the problem
• No prejudice - no sacred cows
• Automated tests are not one-off investment,
it's a continuous journey
• Performance is a damn important feature

97. Do you want to help?
We are hiring in Gdańsk
• Principal Java Developer
• Development Team Lead
• Java and Scala Developers
• UX Designer
• Front-End Developer
• QA Engineer
Visit us at the booth or apply at
http://www.atlassian.com/company/careers

98. Images - Credits
• Turtle - by Jonathan Zander, CC-BY-SA-3.0
• Loading - by MatthewJ13, CC-SA-3.0
• Magic Potion - by Koolmann1, CC-BY-SA-2.0
• Merlin Tool - by By L. Mahin, CC-BY-SA-3.0
• Choose Pills - by *rockysprings, CC-BY-SA-3.0

99. Thank You!