Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Escaping Automated Test Hell - One Year Later

1,782 views

Published on

Slides from my talk at 33rd Degree 2013 Conference in Warsaw.

More than year ago we faced the fact that we are hitting the wall with our large scale automated testing in Atlassian JIRA. We analysed the problems and possible solutions and shared them with community at 33rd Degree in 2012. Since then we've implemented a lot of our ideas and come up with new, learnt new quite unexpected things and got rid of Selenium 1 completely.

This session shows the learnings from our journey – escaping from Test Hell – back to the normality.

If you are interested to hear what problems you can (and probably will) face if you have thousands of automated tests on on levels of abstractions (functional, integration, unit, UI, performance) and what solutions can be applied to remedy them – this presentation is for you.

Published in: Technology
  • Be the first to comment

Escaping Automated Test Hell - One Year Later

  1. 1. Main sponsorEscaping Automated Test Hell One year later... Wojciech Seliga
  2. 2. About me• Coding for 30 years• Agile Practices (inc. TDD) since 2003• Dev Nerd, Tech Leader, Agile Coach, Speaker• 5+ years with Atlassian (JIRA Development Team Lead)• Spartez Co-founder
  3. 3. Year ago - recap
  4. 4. 18 000 tests on all levels Very slow and fragile feedback loop
  5. 5. Serious performance and reliability issues
  6. 6. Feedback Test Speed Quality `
  7. 7. Respect RestructureDesign Share Prune Test Code is Not TrashRefactor Maintain Review Discuss
  8. 8. Optimum Balance
  9. 9. Optimum BalanceIsolation
  10. 10. Optimum BalanceIsolation Speed
  11. 11. Optimum BalanceIsolation Speed Coverage
  12. 12. Optimum BalanceIsolation Speed Coverage Level
  13. 13. Optimum BalanceIsolation Speed Coverage Level Access
  14. 14. Optimum BalanceIsolation Speed Coverage Level Access Effort
  15. 15. Dangerous to temper with
  16. 16. Dangerous to temper withQuality / Determinism
  17. 17. Dangerous to temper withQuality / Determinism Maintainability
  18. 18. Splitting codebase is key aspect of short test feedback loop
  19. 19. Now
  20. 20. People - Motivation
  21. 21. Shades of Red
  22. 22. Pragmatic CI Health
  23. 23. Build Tiers and PolicyTier A1 - green soon after all commits unit tests and functional* testsTier A2 - green at the end of the day WebDriver and bundled plugins testsTier A3 - green at the end of the iteration supported platforms tests, compatibility tests
  24. 24. Wallboards: ConstantAwareness
  25. 25. Training• assertThat over assertTrue/False and assertEquals• avoiding races - Atlassian Selenium with its TimedElement• Unit tests over functional tests• Brownbags, blogs, code reviews
  26. 26. Quality
  27. 27. Re-run failed tests and see if they passAutomatic Flakiness Detection Quarantine
  28. 28. Quarantine - Healing
  29. 29. SlowMo - expose races
  30. 30. Selenium 1
  31. 31. Selenium 1
  32. 32. Selenium ditching Sky did not fall in
  33. 33. Ditching - benefits• Freed build agents - better system throughput• Boosted morale• Gazillion of developer hours saved• Money saved on infrastructure
  34. 34. Ditching - due diligence• conducting the audit - analysis of the coverage we lost• determining which tests needs to rewritten (e.g. security related)• rewriting the tests
  35. 35. Flaky Browser-based Tests Races between test code and asynchronous page logic Playing with "loading" CSS class does not really help
  36. 36. Races Removal with Tracing// in the browser:function mySearchClickHandler() {    doSomeXhr().always(function() {        // This executes when the XHR has completed (either success or failure)        JIRA.trace("search.completed");    });}// In production code JIRA.trace is a no-op// in my page object:@InjectTraceContext traceContext; public SearchResults doASearch() {    Tracer snapshot = traceContext.checkpoint();    getSearchButton().click(); // causes mySearchClickHandler to be invoked    // This waits until the "search.completed" // event has been emitted, *after* previous snapshot        traceContext.waitFor(snapshot, "search.completed");    return pageBinder.bind(SearchResults.class);}
  37. 37. Speed
  38. 38. SpeedCan we halve our build times?
  39. 39. Parallel Execution - Theory BatchesStart of Build End of Build
  40. 40. Parallel Execution BatchesStart of Build End of Build
  41. 41. Parallel Execution - Agentavailability Reality Bites Batches Start of Build End of Build
  42. 42. Dynamic Test Execution Dispatch - Hallelujah
  43. 43. Dynamic Test Execution Dispatch - Hallelujah
  44. 44. "You cant manage what you cant measure." W. Edwards Deming
  45. 45. If y ou you bel ieve "Youare manage what cant just doo i i you cant measure."n me t d. W. Edwards Deming
  46. 46. You cant improve something if you cant measure it
  47. 47. You cant improve something if you cant measure it Profiler, Build statistics, Logs, statsd → Graphite
  48. 48. Compilation Packaging Executing TestsAnatomy of Build*
  49. 49. Fetching Dependencies Compilation Packaging Executing Tests Anatomy of Build*
  50. 50. Fetching Dependencies Compilation Packaging Executing Tests Anatomy of Build* *Any resemblance to maven build is entirely accidental
  51. 51. Fetching Dependencies Compilation PackagingSCM Update Executing Tests Anatomy of Build* *Any resemblance to maven build is entirely accidental
  52. 52. Agent Availability/Setup Fetching Dependencies Compilation Packaging SCM Update Executing Tests Anatomy of Build* *Any resemblance to maven build is entirely accidental
  53. 53. Agent Availability/Setup Fetching Dependencies Compilation Packaging Publishing Results SCM Update Executing Tests Anatomy of Build* *Any resemblance to maven build is entirely accidental
  54. 54. Compilation (7min)JIRA Unit Tests Build
  55. 55. Compilation (7min) Packaging (0min)JIRA Unit Tests Build
  56. 56. Compilation (7min) Packaging (0min) Executing Tests (7min)JIRA Unit Tests Build
  57. 57. Compilation (7min) Publishing Results (1min) Packaging (0min) Executing Tests (7min)JIRA Unit Tests Build
  58. 58. Compilation (7min) Publishing Results (1min) Packaging (0min) Executing Tests (7min) Fetching Dependencies (1.5min)JIRA Unit Tests Build
  59. 59. Compilation (7min) Publishing Results (1min)SCM Update (2min) Packaging (0min) Executing Tests (7min) Fetching Dependencies (1.5min) JIRA Unit Tests Build
  60. 60. Agent Availability/Setup (mean 10min) Compilation (7min) Publishing Results (1min) SCM Update (2min) Packaging (0min) Executing Tests (7min) Fetching Dependencies (1.5min) JIRA Unit Tests Build
  61. 61. Decreasing Test Execution Time to ZERROalone would not let us achieve our goal!
  62. 62. Agent Availability/Setup• starved builds due to busy agents building very long builds• time synchronization issue - NTPD problem
  63. 63. SCM Update - Checkout time• Proximity of SCM repo• shallow git clones are not so fast and lightweight + generating extra git server CPU load• git clone per agent/plan + git pull + git clone per build (hard links!)• Stash was thankful (queue)
  64. 64. SCM Update - Checkout time• Proximity of SCM repo• shallow git clones are not so fast and lightweight + generating extra git server CPU load• git clone per agent/plan + git pull + git clone per build (hard links!)• Stash was thankful (queue) 2 min → 5 seconds
  65. 65. Fetching Dependencies• Fix Predator• Sandboxing/isolation agent trade-off: rm -rf $HOME/.m2/repository/com/atlassian/* into find $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm• Network hardware failure found (dropping packets)
  66. 66. Fetching Dependencies• Fix Predator• Sandboxing/isolation agent trade-off: rm -rf $HOME/.m2/repository/com/atlassian/* into find $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm• Network hardware failure found (dropping packets) 1.5 min → 10 seconds
  67. 67. Compilation• Restructuring multi-pom maven project and dependencies• Maven 3 parallel compilation FTW -T 1.5C *optimal factor thanks to scientific trial and error research
  68. 68. Compilation• Restructuring multi-pom maven project and dependencies• Maven 3 parallel compilation FTW -T 1.5C *optimal factor thanks to scientific trial and error research 7 min → 1 min
  69. 69. Unit Test Execution • Splitting unit tests into 2 buckets: good and legacy (much longer) • Maven 3 parallel test execution (-T 1.5C)3000 poor tests 11000 good tests (5min) (1.5min)
  70. 70. Unit Test Execution • Splitting unit tests into 2 buckets: good and legacy (much longer) • Maven 3 parallel test execution (-T 1.5C)3000 poor tests 11000 good tests (5min) (1.5min) 7 min → 5 min
  71. 71. Functional Tests• Selenium 1 removal did help• Faster reset/restore (avoid unnecessary stuff, intercepting SQL operations for debug purposes - building stacktraces is costly)• Restoring via Backdoor REST API• Using REST API for common setup/ teardown operations
  72. 72. Functional Tests
  73. 73. Publishing Results• Server log allocation per test → using now Backdoor REST API (was Selenium)• Bamboo DB performance degradation for rich build history - to be addressed
  74. 74. Publishing Results• Server log allocation per test → using now Backdoor REST API (was Selenium)• Bamboo DB performance degradation for rich build history - to be addressed 1 min → 40 s
  75. 75. Unexpected Problem• Stability Issues with our CI server• The bottleneck changed from I/O to CPU• Too many agents per physical machine
  76. 76. Compilation (1min)JIRA Unit Tests Build Improved
  77. 77. Compilation (1min) Packaging (0min)JIRA Unit Tests Build Improved
  78. 78. Compilation (1min) Packaging (0min) Executing Tests (5min)JIRA Unit Tests Build Improved
  79. 79. Compilation (1min) Packaging (0min) Publishing Results (40sec) Executing Tests (5min)JIRA Unit Tests Build Improved
  80. 80. Compilation (1min) Packaging (0min) Publishing Results (40sec) Executing Tests (5min)Fetching Dependencies (10sec)JIRA Unit Tests Build Improved
  81. 81. Compilation (1min) Packaging (0min) Publishing Results (40sec)SCM Update (5sec) Executing Tests (5min)Fetching Dependencies (10sec)JIRA Unit Tests Build Improved
  82. 82. Agent Availability/Setup (3min)* Compilation (1min) Packaging (0min) Publishing Results (40sec)SCM Update (5sec) Executing Tests (5min)Fetching Dependencies (10sec)JIRA Unit Tests Build Improved
  83. 83. Improvements Summary Tests Before After Improvement % Unit tests 29 min 17 min 41% Functional tests 56 min 34 min 39% WebDriver tests 39 min 21 min 46% Overall 124 min 72 min 42%* Additional ca. 5% improvement expected once new git clone strategy is consistently rolled-out everywhere
  84. 84. The Quality Follows
  85. 85. The Quality Follows
  86. 86. The Quality Follows
  87. 87. But thats still badWe want CI feedback loop in a few minutes maximum
  88. 88. Splitting The Codebase
  89. 89. Resistance against splitting The last attempt: Magic MachineDecide with high confidence (e.g. > 95%) which subset of tests to run basing on the committed changes
  90. 90. Magic Machine• Looking at Bamboo history (analysing correlation between changes and failures)• Matching: package test/prod code and transitive imports• Code instrumentation (Clover, Emma, AspectJ)• Run most often failing first
  91. 91. Inevitable Split - Fears• Organizational concerns - understanding, managing, integrating, releasing• Mindset change - if something worked for 10 years why to change it?• We damned ourselves with big buckets for all tests - where do they belong to?
  92. 92. Magic Machine strikes back With heavy use of brain, common sense and expert judgement
  93. 93. Splitting code base• Step 0 - JIRA Importers Plugin (3 years ago)• Step 1- New Issue View and Navigator JIRA 6 .0
  94. 94. We are still escaping hell.Hell sucks in your soul.
  95. 95. Conclusions• Visibility and problem awareness help• Maintaing huge testbed is difficult and costly• Measure the problem• No prejudice - no sacred cows• Automated tests are not one-off investment, its a continuous journey• Performance is a damn important feature
  96. 96. Do you want to help?We are hiring in Gdańsk• Principal Java Developer• Development Team Lead• Java and Scala Developers• UX Designer• Front-End Developer• QA Engineer Visit us at the booth or apply at http://www.atlassian.com/company/careers
  97. 97. Images - Credits• Turtle - by Jonathan Zander, CC-BY-SA-3.0• Loading - by MatthewJ13, CC-SA-3.0• Magic Potion - by Koolmann1, CC-BY-SA-2.0• Merlin Tool - by By L. Mahin, CC-BY-SA-3.0• Choose Pills - by *rockysprings, CC-BY-SA-3.0
  98. 98. Thank You!

×