Your SlideShare is downloading. ×
0
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Escaping Test Hell - Our Journey - XPDays Ukraine 2013

1,181

Published on

My updated slides about the journey to hell and back to normality wrt automated tests at scale. Based on real 10+ years experience of JIRA development teams. …

My updated slides about the journey to hell and back to normality wrt automated tests at scale. Based on real 10+ years experience of JIRA development teams.
I delivered this talk at XPDays in Kiev in October 2013.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,181
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. Automated Test Hell Wojciech Seliga wojciech.seliga@spartez.com @wseliga Our Journey
  2. About me • Coding for 30 years, now only in "free time" • Agile Practices (inc.TDD) since 2003 • Dev Nerd,Tech Leader,Agile Coach, Speaker, PHB • 6 years with Atlassian (JIRA Dev Manager) • Spartez Co-founder & CEO
  3. XP Promise CostofChange Time Waterfall XP
  4. 1.5 year ago
  5. Almost 10 years of accumulating garbage automatic tests
  6. About 20 000 tests on all levels of abstraction
  7. Very slow (even hours) and fragile feedback loop
  8. Serious performance and reliability issues
  9. Dispirited devs accepting RED as a norm
  10. Feedback Speed ` Test Quality
  11. Test Code is Not Trash Design Maintain Refactor Share Review Prune Respect Discuss Restructure
  12. Test Pyramid Unit Tests (including QUnit) REST / HTML Tests Selenium
  13. Optimum Balance Isolation Speed Coverage Level Access Effort
  14. Dangerous to temper with MaintainabilityQuality / Determinism
  15. Now
  16. People - Motivation Making GREEN the norm
  17. Shades of Red Shades of Green
  18. Pragmatic CI Health
  19. Build Tiers and Policy Tier A1 - green soon after all commits Tier A2 - green at the end of the day Tier A3 - green at the end of the iteration unit tests and functional* tests WebDriver and bundled plugins tests supported platforms tests, compatibility tests
  20. Wallboards: Constant Awareness
  21. Extensive Training • assertThat over assertTrue/False and assertEquals • avoiding races - Atlassian Selenium with its TimedElement • Favouring unit tests over functional tests • Promoting Page Objects • Brownbags, blogs, code reviews
  22. Quality
  23. Automatic Flakiness Detection Quarantine Re-run failed tests and see if they pass
  24. Quarantine - Healing
  25. SlowMo - expose races
  26. Selenium 1
  27. Selenium ditching Sky did not fall in
  28. Ditching - benefits • Freed build agents - better system throughput • Boosted morale • Gazillion of developer hours saved • Money saved on infrastructure
  29. Ditching - due diligence • conducting the audit - analysis of the coverage we lost • determining which tests needs to rewritten (e.g. security related) • rewriting some of the tests (good job for new hires + a senior mentor)
  30. Flaky Browser-based Tests Races between test code and asynchronous page logic Playing with "loading" CSS class does not really help
  31. Races Removal with Tracing // in the browser: function mySearchClickHandler() {     doSomeXhr().always(function() {         // This executes when the XHR has completed (either success or failure)         JIRA.trace("search.completed");     }); } // In production code JIRA.trace is a no-op // in my page object: @Inject TraceContext traceContext;   public SearchResults doASearch() {     Tracer snapshot = traceContext.checkpoint();     getSearchButton().click(); // causes mySearchClickHandler to be invoked     // This waits until the "search.completed" // event has been emitted, *after* previous snapshot         traceContext.waitFor(snapshot, "search.completed");     return pageBinder.bind(SearchResults.class); }
  32. Can we halve our build times? Speed
  33. Parallel Execution - Theory End of Build Batches Start of Build Time
  34. Parallel Execution End of Build Batches Start of Build Time
  35. Parallel Execution - Reality Bites End of Build Batches Start of Build Agent availability Time
  36. Dynamic Test Execution Dispatch - Hallelujah
  37. "You can't manage what you can't measure." not by W. Edwards Deming If you believe just in it you are doomed.
  38. You can't improve something if you can't measure it Profiler, Build statistics, Logs, statsd → Graphite
  39. Anatomy of Build* Compilation Packaging Executing Tests Fetching Dependencies *Any resemblance to maven build is entirely accidental SCM Update Agent Availability/Setup Publishing Results
  40. JIRA Unit Tests Build Compilation (7min) Packaging (0min) Executing Tests (7min) Fetching Dependencies (1.5min) SCM Update (2min) Agent Availability/Setup (mean 10min) Publishing Results (1min)
  41. Decreasing Test Execution Time to ZERRO alone would not let us achieve our goal!
  42. Agent Availability/Setup • starved builds due to busy agents building very long builds • time synchronization issue - NTPD problem Fixes applied
  43. • Proximity of SCM repo • shallow git clones are not so fast and lightweight + generating extra git server CPU load • git clone per agent/plan + git pull + git clone per build (hard links!) • Atlassian Stash was thankful (queue) SCM Update - Checkout time 2 min → 5 seconds Trade disk space for speed
  44. • Fix Predator • Sandboxing/isolation agent trade-off: rm -rf $HOME/.m2/repository/com/atlassian/* into find $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm • Network hardware failure found (dropping packets) Fetching Dependencies 1.5 min → 10 seconds
  45. Compilation • Restructuring multi-pom maven project and dependencies • Maven 3 parallel compilation FTW -T 1.5C *optimal factor thanks to scientific trial and error research 7 min → 1 min
  46. Unit Test Execution • Splitting unit tests into 2 buckets: good and legacy (much longer) • Maven 3 parallel test execution (-T 1.5C) 7 min → 5 min 3000 poor tests (5min) 11000 good tests (1.5min)
  47. Functional Tests • Selenium 1 removal did help • Faster reset/restore (avoid unnecessary stuff, intercepting SQL operations for debug purposes - building stacktraces is costly) • Restoring via Backdoor REST API • Using REST API for common setup/ teardown operations
  48. Functional Tests We like this trend
  49. Publishing Results • Server log allocation per test → using now Backdoor REST API (was Selenium) • Bamboo DB performance degradation for rich build history - to be addressed 1 min → 40 s
  50. Unexpected Problem • Stability issues with our CI server • The bottleneck changed from I/O to CPU • Too many agents per physical machine
  51. JIRA Unit Tests Build Improved Compilation (1min) Packaging (0min) Executing Tests (5min) Fetching Dependencies (10sec) SCM Update (5sec) Agent Availability/Setup (3min)* Publishing Results (40sec)
  52. Improvements Summary Tests Before After Improvement % Unit tests 29 min 17 min 41% Functional tests 56 min 34 min 39% WebDriver tests 39 min 21 min 46% Overall 124 min 72 min 42% * Additional ca. 5% improvement expected once new git clone strategy is consistently rolled-out everywhere
  53. The Quality Follows
  54. But that's still bad We want CI feedback loop in a few minutes maximum
  55. Splitting The Codebase
  56. Codebase Split - Problems • Organizational concerns - understanding, managing, integrating, releasing • Mindset change - if something worked for 10 years why to change it? • We damned ourselves with big buckets for all tests - where do they belong to?
  57. Splitting code base • Step 0 - JIRA Importers Plugin (3.5 years ago) • Step 1- New IssueView and Navigator JIRA 6.0
  58. We are still escaping hell. Hell sucks in your soul.
  59. Conclusions • Visibility and problem awareness help • Maintaing huge testbed is difficult and costly • Measure the problem, measure improvements • No prejudice - no sacred cows • Automated tests are not one-off investment, it's a continuous journey • Performance is a damn important feature
  60. Revised XP Promise CostofChange Time Waterfall XP Sad Reality
  61. Interested in such stuff? Talk to me at the conference or visit http://www.spartez.com/careers We are hiring in Gdańsk
  62. • Turtle - by Jonathan Zander, CC-BY-SA-3.0 • Loading - by MatthewJ13, CC-SA-3.0 • Magic Potion - by Koolmann1, CC-BY-SA-2.0 • Merlin Tool - by By L. Mahin, CC-BY-SA-3.0 • Choose Pills - by *rockysprings, CC-BY-SA-3.0 • Flashing Red Light - bt Chris Phan, CC BY 2.0 Images - Credits
  63. ThankYou! Tweet your feedback at @wseliga

×