Your SlideShare is downloading. ×
Automated Acceptance Test Practices and Pitfalls
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Automated Acceptance Test Practices and Pitfalls


Published on

Key impediments and pitfalls people run into when trying to use AATs, and how to get over them

Key impediments and pitfalls people run into when trying to use AATs, and how to get over them

Published in: Technology, Education

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • IntroMy experience with AATsWhy – seen benefits and pitfalls, ways to mitigate the cons
  • Ask Q’s
  • Bad legacy code in mission critical system, not unit testable…Useful in general in a test suite for many reasons…
  • Acceptance TestsFits their requirements. The story is complete. Automated, pipeline, run
  • Readable by whole spectrum
  • What kind of tests do you run with your specs?UI – smoke, sanity, UI goo Functionality – Integration testsRun close to production; like a user would.
  • Mostly unit testsHARD to maintain, slow SO….
  • Unit test everything thru JavaScriptMost RELEVANT test cases for AATs (can’t hit all alt paths)Shouldn’t manually regression test
  • Tests are written before developer starts Whoever writes user storiesDev/Tester/BA BEFORE iteration starts
  • All know definition of User Story? Should be turned into specs before iteration starts (bus & )
  • -Communication with business stakeholders, BAs, QA
  • -Unit tests don’t account for everything, seams-Gaps in coverage, unit tests not hitting right test cases,
  • -Full regression doesn’t always happen-Manual testing lapseMistakes in testing, non-testing, non-regressingAllow testers to do more exploratory and higher level things, less boring
  • One of the things that takes the longest to get software delivered, is making sure everything works functionally and nothing that worked broke
  • Bugs will be caught earlier
  • Not doing, btw
  • Some have even said it’s not worth the cost
  • Some have even said it’s not worth the cost
  • Some have even said it’s not worth the cost
  • Transcript

    • 2. Agenda • Part I: Overview • Part II: Pros • Why even talking about them • Part III: Pitfalls & Solutions • Impediments • Resistance/Bias/Value • Bandwidth/Velocity • Learning curve/Experience • Getting specs • Pitfalls • Speed • Maintainability • Poor adoption
    • 3. Part I AAT Overview
    • 4. **Legacy Nightmare <div class="pdl13 pdr15"> <div class="fll w744"> <div class="wp100 bdr02 bgc03"> <div class="pd08 mostDownload01_l vat"> <div class="bgc02 vat"> <div class="mostDownload_m bgc02 vat"> <div class="wp100"> <div class="fll bgc05 pdr03 h20">
    • 5. Automated Acceptance Tests (AATs)
    • 6. Given – When – Then Given I have no food When I buy 2 pickles on Amazon Then I have some food
    • 7. End-to-end UI, Integration tests
    • 8. Acceptance Component Unit Black box White box End-to-end Business facing Localized Technology facing
    • 9. Manual Checking End-to-end UI Component Unit SOURCE:
    • 10. Manual Checking Inverting the Testing Pyramid UI Acceptance, Integration Unit -Exploratory testing
    • 11. Acceptance Test Strategy • Happy paths • Major unhappy paths • Legacy • Regression
    • 12. Who writes acceptance tests?
    • 13. What can you write them in? Vim
    • 14. Part II How they work
    • 15. Start with a story User story As an internet user, I want to search for “ALT.NET” And get valid results • Acceptance criteria 1: I get at least 100k results • Acceptance criteria 2: Results ordered by relevance … Scenarios Feature
    • 16. Tracking AATs
    • 17. Tracking AATs
    • 18. Various Frameworks Browser driver Browser automation Test runner/harness BDD Framework .NET Selenium, Phantom.js Selenium, Waitn NUnit, xUnit, MSTest SpecFlow, Fitness (.NET runner), Cuke4Nuke, MSpec JavaScript Selemium, Phantom.js Phantom.js/Casp er.js Mocha, Jasmine Chai, Jasmine Java/Ruby Selenium, Phantom.js Selenium, Waitr JUnit, test-unit JBehave, Fitness, Cucumber/RSpec/ Capybara
    • 19. Why Have AATs? (Pros)
    • 20. Communication • Helps specify behavior of the system in plain text • Provides a medium for non-tech and devs to agree on Are we talking about the right system?
    • 21. Seams, unit test mistakes
    • 22. Automation “There’s no place for human beings to be doing regression testing manually.” -Jez Humble
    • 23. Speedier deployments
    • 24. Save resources
    • 25. More testing during development
    • 26. Why NOT Have AATs?
    • 27. Why don‟t more people use them?
    • 28. Impediments • Poor adoption • Bandwidth/Velocity • Learning Curve/Experience • Business users won‟t write specifications
    • 29. Why NOT have AATs? • Maintenance • Speed • High false negatives, non-determinism
    • 30. Questions?
    • 31. Part IV Overcoming the Impediments
    • 32. Minimize # of end-to-end tests • AATs should be a small portion of your test suite • Is your new feature entirely new? • Balance high # of unit tests + selected end-to-end & acceptance
    • 33. Concise Specs • Declarative • Reuse steps • Concise, many methods, fat fixtures Bad: • Click on Log In button • Click username box and type „myUsername‟ and click „password‟ box and type „myPassword‟ • Click on link for Transfer Payment • Click box for amount and type 400 • Click „Transfer‟ button • Assert success message Good: • Login as „MyAccountName‟ • Navigate to Transfer Payment page • Transfer 400 dollars • Assert success message
    • 34. Use a headless browser
    • 35. Don‟t go through the UI • Controller down, service down • Use mocks, stubs
    • 36. When Acceptances Tests catches a bug • See why bug got through unit/integration tests • Add unit, integration tests • Prune AAT? Unit
    • 37. Organize your tests… Use Page Objects Keeps each page‟s tests • Encapsulated • Readable • Reusable
    • 38. public class GoogleHomepage: PageBase, IGoogleHomepage { // Flow for this page } Domain Service UI GoogleHomepage +search(query) +clickSearch() +clickGettingLucky Data Access GoogleHomepageService +search(query) +clickSearch() +clickGettingLucky IGoogleHomepage +search(query) +clickSearch() +clickGettingLucky Page Objects
    • 39. Page Objects Faster test creation – like Legos • Keeps step definitions very readable and thin
    • 40. Hides details • Abstracts SpecFlow from Selenium • Keeps infrastructure at a lower level
    • 41. Keeps changes minimal • Changes in the application should require minimal changes in the tests, and not in the specs
    • 42. Easier to write new tests • Constructing page objects • Can be used to create a DSL
    • 43. One way [When(@"I log in with the teacher account '(.*)'")] public void WhenILogInWithTheTeacherAccount(string orgCode) { var driver = new FirefixDriver(); driver.Navigate().GoToUrl(""); var userTb = Driver.FindElement(By.Name("username”)); var passwordTb = Driver.FindElement(By.Id("password")); userTb.SendKeys(username); passwordTb.SendKeys(password); var loggedInLink = Driver.FindElement(By.LinkText("Log Out")); Assert.That(loggedInLink != null); }
    • 44. Better way – page objects public class LogInPage : PageBase { public PageBase LogIn(string username, string password) { Driver.FindElement(By.Id("UserName")).SendKeys(username); Driver.FindElement(By.Id("Password")).SendKeys(password); Driver.FindElement(By.CssSelector("input[type="submit"]")).Click(); return GetInstance<AccountLandingPage>(Driver); } public void HasExpectedTab(string tabName) { var link = Driver.FindElement(By.LinkText(tabName)); Assert.AreEqual(tabName, link.Text); } }
    • 45. Other things • Managed test data • Distributed execution
    • 46. Tips
    • 47. Developer tips • Don't run in same assembly as other tests, so these can be run separately since they're very slow (i.e. nightly) • Navigation • Create different levels of suites depending on depth/level of feedback desired: • Smoke, Current iteration/sprint, Regression • Run AATs as close to the real environment as possible
    • 48. Gherkin Specs shouldn‟t have much setup code Given I am on the homepage And I click login And I enter username of „username‟ and password of „password‟ When clicking „Log In‟ Then take me to my landing page Given I am on the homepage When logging in with „username‟ and „password‟ Then take me to my landing page
    • 49. UI tests • Sometimes tests will fail if the page doesn‟t have enough time to load • Capture screen shots when tests fail • Run nightly • Run across multiple machines
    • 50. Anti-patterns • Developers writing acceptance tests by themselves, for themselves • Tying AATs to an implementation - unit tests are implementation specific, AATs are NOT • Too much repetition in the gherkin • Too many UI tests • UI tests are failing too easily when changing the UI
    • 51. Conclusion
    • 52. Where we can be • Reliable tests • Reusable test code (even a DSL) • Separation of concerns • Short and clear tests • Fast test creation • Quick regression feedback • Tests that are understood by non-technical people • Fewer bugs
    • 53. Resources: Tools used here: • SpecFlow – • PhantomJS – • Selenium WebDriver – Books: • Specification By Example, Gojko Adzic • Continuous Delivery, Jez Humble, David Farley • Growing Object-Oriented Software, Guided By Tests, Steve Freeman, Nat Pryce Articles: • Automated Acceptance Tests, • BDD with SpecFlow and ASP.NET MVC, development-bdd-with-specflow-and-aspnet-mvc/ • Using Gherkin Language in SpecFlow, • BDD with SpecFlow, MSDN, • Using SpecFlow with the Page Object, management/using-specflow-to • Problems with Acceptance Tests, • Top 10 reasons why teams fail with Acceptance Testing, fail-with-acceptance-testing/ • Maintaining Automated Acceptance Tests (ThoughtWorks), • Creating Maintainable Automated Acceptance Test Suites, Jez Humble, •
    • 54. Happy Three Amigos Slides: tests-in-net Code: @wynv