This document discusses challenges with test automation and provides recommendations to address them. It notes that test automation can become problematic due to issues like intermittent failures, large test suite size and long duration runs. It recommends breaking tests into multiple test suites and running them in parallel to help address these issues. The document also emphasizes the importance of understanding why tests are failing, disabling failing tests one at a time, and combining techniques like waiting for explicit events. It argues that test automation issues are not related to management, money or external factors, but rather are a problem of team culture, and suggests removing those who don't value testing and hiring people experienced with open-source testing.

1. Taming the Testing Beast
Tuesday, October 23rd 2012
Daniel Doubrovkine (dB.)
@dblockdotorg
#agiledc 2012

2. State of Testing: Scary!
http://art.sy/artwork/francisco-de-goya-saturn-devouring-one-of-his-sons-from-the-series-of-black-paintings-dot

3. Test Automation @
3×
Independent Teams
Hardware Platforms
Vendors
QA
automated voting
user acceptance tests

4. Test Automation @
1124× Test Code vs. Code
http://www.sqlite.org/testing.html

5. Test Automation @ Your Company
1:1 Dev : Test
some automation
QA some people test
some software often
doesn’t install

6. Test Automation @ My Company
0
http://art.sy/artwork/conrad-shawcross-chord
Testers
Formalities
100% automation
QA continuous everything
8 developers
8 hours of sleep / day

7. It’s [Relatively] Big
4678 Automated Tests
100% user scenario
coverage ~ 800 tests
1h to Run http://art.sy/artwork/liu-di-animal-regulation-no-2

8. Technology and Tools
 RSpec BDD : it “does X”
 Mongo Data Models : ODM with “real” data
 Ruby API : HTTP request – response
 Integration : Capybara

9. Technology and Tools : Mongo ODM + Fabricators
context “spline” do
let(:spline) { Fabricate :spline }
it “reticulate! updates reticulated count” do
expect {
spline.reticulate!
}.to change(Spline.reticulated, :count).by(1)
end
end

10. Technology and Tools : Ruby API
context “spline” do
it “create” do
expect {
post “/api/spline?name=test”
response.status.should == 201
}.to change(Spline, :count).by(1)
end
end

11. Technology and Tools : Integration
context “spline” do
let(:spline) { Fabricate :spline }
it “displays a spline” do
visit “/spline/#{spline.id}”
page.should have_css “.spline[id=#{spline.id}]”
end
end

12. Mixed Bag
 High Confidence
 Size
 Duration
http://art.sy/artwork/liu-di-animal-regulation-no-4

13. It Quickly Became a Big Problem
 Intermittent Failures
 Size
 Duration
http://art.sy/artwork/liu-di-animal-regulation-no-6

14. Break Things Up
 Multiple Applications

15. Run Tests in Parallel
 Multiple Test Suites
http://artsy.github.com/blog/2012/10/09/how-to-run-rspec-test-suites-in-parallel-with-jenkins-ci-build-flow/

16. http://artsy.github.com/blog/2012/02/03/reliably-testing-asynchronous-ui-w-slash-rspec-and-capybara/
Understand
 Disable Failing Tests
 One Example at a Time
 No Hacks Allowed
 Wait for Explicit Events
 Combine Techniques
 Track Failures
http://www.friendshipismagic.org/forum/members/captain-falcon/albums/miscellaneous/9452-wat/

17. Test Anxiety
 High Expectations
 Fear of Disappointment
 Lack of Control
 Inability to Change
http://www.sodahead.com/fun/is-your-school-strict/question-3158781/

18. Test Automation is Not a ______ Problem
 Not a Management Problem
 Not a Money Problem
 Not an External Problem
 Not a Consultant’s Problem

19. Test Automation is a Team Culture Problem
 Remove Those that Talk About Testing
 Hire From Open-Source
 Remove the Safety Net

20. Daniel Doubrovkine (dB.)
@dblockdotorg