Your SlideShare is downloading. ×
0
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Escaping Test Hell - Our Journey - XPDays Ukraine 2013
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Escaping Test Hell - Our Journey - XPDays Ukraine 2013

1,179

Published on

My updated slides about the journey to hell and back to normality wrt automated tests at scale. Based on real 10+ years experience of JIRA development teams. …

My updated slides about the journey to hell and back to normality wrt automated tests at scale. Based on real 10+ years experience of JIRA development teams.
I delivered this talk at XPDays in Kiev in October 2013.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,179
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Automated Test Hell Wojciech Seliga wojciech.seliga@spartez.com @wseliga Our Journey
  • 2. About me • Coding for 30 years, now only in "free time" • Agile Practices (inc.TDD) since 2003 • Dev Nerd,Tech Leader,Agile Coach, Speaker, PHB • 6 years with Atlassian (JIRA Dev Manager) • Spartez Co-founder & CEO
  • 3. XP Promise CostofChange Time Waterfall XP
  • 4. 1.5 year ago
  • 5. Almost 10 years of accumulating garbage automatic tests
  • 6. About 20 000 tests on all levels of abstraction
  • 7. Very slow (even hours) and fragile feedback loop
  • 8. Serious performance and reliability issues
  • 9. Dispirited devs accepting RED as a norm
  • 10. Feedback Speed ` Test Quality
  • 11. Test Code is Not Trash Design Maintain Refactor Share Review Prune Respect Discuss Restructure
  • 12. Test Pyramid Unit Tests (including QUnit) REST / HTML Tests Selenium
  • 13. Optimum Balance Isolation Speed Coverage Level Access Effort
  • 14. Dangerous to temper with MaintainabilityQuality / Determinism
  • 15. Now
  • 16. People - Motivation Making GREEN the norm
  • 17. Shades of Red Shades of Green
  • 18. Pragmatic CI Health
  • 19. Build Tiers and Policy Tier A1 - green soon after all commits Tier A2 - green at the end of the day Tier A3 - green at the end of the iteration unit tests and functional* tests WebDriver and bundled plugins tests supported platforms tests, compatibility tests
  • 20. Wallboards: Constant Awareness
  • 21. Extensive Training • assertThat over assertTrue/False and assertEquals • avoiding races - Atlassian Selenium with its TimedElement • Favouring unit tests over functional tests • Promoting Page Objects • Brownbags, blogs, code reviews
  • 22. Quality
  • 23. Automatic Flakiness Detection Quarantine Re-run failed tests and see if they pass
  • 24. Quarantine - Healing
  • 25. SlowMo - expose races
  • 26. Selenium 1
  • 27. Selenium ditching Sky did not fall in
  • 28. Ditching - benefits • Freed build agents - better system throughput • Boosted morale • Gazillion of developer hours saved • Money saved on infrastructure
  • 29. Ditching - due diligence • conducting the audit - analysis of the coverage we lost • determining which tests needs to rewritten (e.g. security related) • rewriting some of the tests (good job for new hires + a senior mentor)
  • 30. Flaky Browser-based Tests Races between test code and asynchronous page logic Playing with "loading" CSS class does not really help
  • 31. Races Removal with Tracing // in the browser: function mySearchClickHandler() {     doSomeXhr().always(function() {         // This executes when the XHR has completed (either success or failure)         JIRA.trace("search.completed");     }); } // In production code JIRA.trace is a no-op // in my page object: @Inject TraceContext traceContext;   public SearchResults doASearch() {     Tracer snapshot = traceContext.checkpoint();     getSearchButton().click(); // causes mySearchClickHandler to be invoked     // This waits until the "search.completed" // event has been emitted, *after* previous snapshot         traceContext.waitFor(snapshot, "search.completed");     return pageBinder.bind(SearchResults.class); }
  • 32. Can we halve our build times? Speed
  • 33. Parallel Execution - Theory End of Build Batches Start of Build Time
  • 34. Parallel Execution End of Build Batches Start of Build Time
  • 35. Parallel Execution - Reality Bites End of Build Batches Start of Build Agent availability Time
  • 36. Dynamic Test Execution Dispatch - Hallelujah
  • 37. "You can't manage what you can't measure." not by W. Edwards Deming If you believe just in it you are doomed.
  • 38. You can't improve something if you can't measure it Profiler, Build statistics, Logs, statsd → Graphite
  • 39. Anatomy of Build* Compilation Packaging Executing Tests Fetching Dependencies *Any resemblance to maven build is entirely accidental SCM Update Agent Availability/Setup Publishing Results
  • 40. JIRA Unit Tests Build Compilation (7min) Packaging (0min) Executing Tests (7min) Fetching Dependencies (1.5min) SCM Update (2min) Agent Availability/Setup (mean 10min) Publishing Results (1min)
  • 41. Decreasing Test Execution Time to ZERRO alone would not let us achieve our goal!
  • 42. Agent Availability/Setup • starved builds due to busy agents building very long builds • time synchronization issue - NTPD problem Fixes applied
  • 43. • Proximity of SCM repo • shallow git clones are not so fast and lightweight + generating extra git server CPU load • git clone per agent/plan + git pull + git clone per build (hard links!) • Atlassian Stash was thankful (queue) SCM Update - Checkout time 2 min → 5 seconds Trade disk space for speed
  • 44. • Fix Predator • Sandboxing/isolation agent trade-off: rm -rf $HOME/.m2/repository/com/atlassian/* into find $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm • Network hardware failure found (dropping packets) Fetching Dependencies 1.5 min → 10 seconds
  • 45. Compilation • Restructuring multi-pom maven project and dependencies • Maven 3 parallel compilation FTW -T 1.5C *optimal factor thanks to scientific trial and error research 7 min → 1 min
  • 46. Unit Test Execution • Splitting unit tests into 2 buckets: good and legacy (much longer) • Maven 3 parallel test execution (-T 1.5C) 7 min → 5 min 3000 poor tests (5min) 11000 good tests (1.5min)
  • 47. Functional Tests • Selenium 1 removal did help • Faster reset/restore (avoid unnecessary stuff, intercepting SQL operations for debug purposes - building stacktraces is costly) • Restoring via Backdoor REST API • Using REST API for common setup/ teardown operations
  • 48. Functional Tests We like this trend
  • 49. Publishing Results • Server log allocation per test → using now Backdoor REST API (was Selenium) • Bamboo DB performance degradation for rich build history - to be addressed 1 min → 40 s
  • 50. Unexpected Problem • Stability issues with our CI server • The bottleneck changed from I/O to CPU • Too many agents per physical machine
  • 51. JIRA Unit Tests Build Improved Compilation (1min) Packaging (0min) Executing Tests (5min) Fetching Dependencies (10sec) SCM Update (5sec) Agent Availability/Setup (3min)* Publishing Results (40sec)
  • 52. Improvements Summary Tests Before After Improvement % Unit tests 29 min 17 min 41% Functional tests 56 min 34 min 39% WebDriver tests 39 min 21 min 46% Overall 124 min 72 min 42% * Additional ca. 5% improvement expected once new git clone strategy is consistently rolled-out everywhere
  • 53. The Quality Follows
  • 54. But that's still bad We want CI feedback loop in a few minutes maximum
  • 55. Splitting The Codebase
  • 56. Codebase Split - Problems • Organizational concerns - understanding, managing, integrating, releasing • Mindset change - if something worked for 10 years why to change it? • We damned ourselves with big buckets for all tests - where do they belong to?
  • 57. Splitting code base • Step 0 - JIRA Importers Plugin (3.5 years ago) • Step 1- New IssueView and Navigator JIRA 6.0
  • 58. We are still escaping hell. Hell sucks in your soul.
  • 59. Conclusions • Visibility and problem awareness help • Maintaing huge testbed is difficult and costly • Measure the problem, measure improvements • No prejudice - no sacred cows • Automated tests are not one-off investment, it's a continuous journey • Performance is a damn important feature
  • 60. Revised XP Promise CostofChange Time Waterfall XP Sad Reality
  • 61. Interested in such stuff? Talk to me at the conference or visit http://www.spartez.com/careers We are hiring in Gdańsk
  • 62. • Turtle - by Jonathan Zander, CC-BY-SA-3.0 • Loading - by MatthewJ13, CC-SA-3.0 • Magic Potion - by Koolmann1, CC-BY-SA-2.0 • Merlin Tool - by By L. Mahin, CC-BY-SA-3.0 • Choose Pills - by *rockysprings, CC-BY-SA-3.0 • Flashing Red Light - bt Chris Phan, CC BY 2.0 Images - Credits
  • 63. ThankYou! Tweet your feedback at @wseliga

×