AUTOMATED
ACCEPTANCE TESTING
GOOD PRACTICES AND PITFALLS
Wyn Van Devanter
wyn.vandevanter@excella.com
@wynv
Part I
AAT Overview
Automated Acceptance Tests (AATs)
Given – When – Then
Given I have no food
When I buy 2 pickles on
Amazon
Then I have some food
End-to-end UI, Integration tests
Acceptance
Component
Unit
Black box
White box
End-to-end
Business facing
Localized
Technology
facing
Manual Checking
End-to-end
UI
Component
Unit
SOURCE: http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid
Manual
Checking
Inverting the Testing Pyramid
UI
Acceptance,
Integration
Unit
-Exploratory testing
Acceptance Test Strategy
• Happy paths
• Major unhappy paths
• Legacy
• Regression
Minimize # of end-to-end tests
• AATs should be a small portion of your test suite
• Is your new feature entirely new?
• B...
Create them for journeys, not stories
• Too heavy, slow and expensive to create for each story
• Must be tracked outside o...
Who writes acceptance tests?
Those closest to the work should
inplement the tests
• More efficient
• Most correct
A framework is key
• Code reuse
• Rapid development
• Readable tests
• More maintainable tests
Don’t go through the UI
• Controller down, service down
• Use mocks, stubs
What can you write them in?
Vim
Part II
How they work
Start with a story
User story
As an internet user,
I want to search for “ALT.NET”
And get valid results
• Acceptance crite...
Tracking AATs
Tracking AATs
Various Frameworks
Browser driver
Browser automation
Test runner/harness
BDD Framework
.NET
Selenium,
Phantom.js
Selenium,...
Why Have AATs? (Pros)
Communication
• Helps specify behavior of the system in plain text
• Provides a medium for non-tech and devs to agree on
A...
Seams, unit test mistakes
Automation
“There’s no place for
human beings to be
doing regression testing
manually.”
-Jez Humble
Speedier deployments
Save resources
More testing during development
Why NOT Have AATs?
Why don’t more people use them?
Impediments
• Poor adoption
• Bandwidth/Velocity
• Learning Curve/Experience
• Business users won’t write specifications
Why NOT have AATs?
• Maintenance
• Speed
• High false negatives, non-determinism
Questions?
Part IV
Overcoming the
Impediments
Use a headless browser
When Acceptances Tests catches a bug
• See why bug got through unit/integration tests
• Add unit, integration tests
• Prun...
Concise Specs
• Declarative
• Reuse steps
• Concise, many methods, fat fixtures
Bad:
• Click on Log In button
• Click user...
Organize your tests…
Use Page Objects
Keeps each page’s tests
• Encapsulated
• Readable
• Reusable
public class GoogleHomepage: PageBase, IGoogleHomepage {
// Flow for this page
}
Domain
Service
UI
GoogleHomepage
+search(...
Page Objects
Faster test creation – like Legos
• Keeps step definitions very readable and thin
Hides details
• Abstracts SpecFlow from Selenium
• Keeps infrastructure at a lower level
Keeps changes minimal
• Changes in the application should require minimal
changes in the tests, and not in the specs
Easier to write new tests
• Constructing page objects
• Can be used to create a DSL
One way
[When(@"I log in with the teacher account '(.*)'")]
public void WhenILogInWithTheTeacherAccount(string orgCode)
{
...
Better way – page objects
public class LogInPage : PageBase
{
public PageBase LogIn(string username, string password)
{
Dr...
Other things
• Managed test data
• Distributed execution
Tips
Developer tips
• Don't run in same assembly as other tests, so these can
be run separately since they're very slow (i.e. n...
Gherkin
Specs shouldn’t have much setup code
Given I am on the homepage
And I click login
And I enter username of ‘usernam...
UI tests
• Sometimes tests will fail if the page doesn’t have enough
time to load
• Capture screen shots when tests fail
•...
Anti-patterns
• Developers writing acceptance tests by themselves, for
themselves
• Tying AATs to an implementation - unit...
Conclusion
Where we can be
• Reliable tests
• Reusable test code (even a DSL)
• Separation of concerns
• Short and clear tests
• Fast...
Resources:
Tools used here:
• SpecFlow – specflow.org
• PhantomJS – phantomjs.org
• Selenium WebDriver – seleniumhq.org
Bo...
Happy Three Amigos
Slides: http://www.slideshare.net/wynvandevanter/automated-acceptance-
tests-in-net
Code:
https://githu...
Upcoming SlideShare
Loading in …5
×

Automated Acceptance Test Practices and Pitfalls

2,221 views

Published on

Key impediments and pitfalls people run into when trying to use AATs, and how to get over them

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,221
On SlideShare
0
From Embeds
0
Number of Embeds
219
Actions
Shares
0
Downloads
66
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Intro

    My experience with AATs

    Why – seen benefits and pitfalls, ways to mitigate the cons
  • Bad legacy code in mission critical system, not unit testable



    Useful in general in a test suite for many reasons…

  • Ask Q’s
  • Acceptance Tests

    Fits their requirements. The story is complete.

    Automated, pipeline, run
  • Readable by whole spectrum
  • What kind of tests do you run with your specs?

    UI – smoke, sanity, UI goo
    Functionality – Integration tests

    Run close to production; like a user would.
  • Mostly unit tests

    HARD to maintain, slow SO….


  • Unit test everything thru JavaScript

    Most RELEVANT test cases for AATs (can’t hit all alt paths)

    Shouldn’t manually regression test
  • Tests are written before developer starts

    Whoever writes user stories
    Dev/Tester/BA

    BEFORE iteration starts
  • All know definition of User Story?

    Should be turned into specs before iteration starts (bus & )
  • -Communication with business stakeholders, BAs, QA
  • -Unit tests don’t account for everything, seams
    -Gaps in coverage, unit tests not hitting right test cases,
  • -Full regression doesn’t always happen
    -Manual testing lapse

    Mistakes in testing, non-testing, non-regressing

    Allow testers to do more exploratory and higher level things, less boring
  • One of the things that takes the longest to get software delivered, is making sure everything works functionally and nothing that worked broke
  • Bugs will be caught earlier
  • Not doing, btw
  • Some have even said it’s not worth the cost

  • Some have even said it’s not worth the cost

  • Some have even said it’s not worth the cost

  • Automated Acceptance Test Practices and Pitfalls

    1. 1. AUTOMATED ACCEPTANCE TESTING GOOD PRACTICES AND PITFALLS Wyn Van Devanter wyn.vandevanter@excella.com @wynv
    2. 2. Part I AAT Overview
    3. 3. Automated Acceptance Tests (AATs)
    4. 4. Given – When – Then Given I have no food When I buy 2 pickles on Amazon Then I have some food
    5. 5. End-to-end UI, Integration tests
    6. 6. Acceptance Component Unit Black box White box End-to-end Business facing Localized Technology facing
    7. 7. Manual Checking End-to-end UI Component Unit SOURCE: http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid
    8. 8. Manual Checking Inverting the Testing Pyramid UI Acceptance, Integration Unit -Exploratory testing
    9. 9. Acceptance Test Strategy • Happy paths • Major unhappy paths • Legacy • Regression
    10. 10. Minimize # of end-to-end tests • AATs should be a small portion of your test suite • Is your new feature entirely new? • Balance high # of unit tests + selected end-to-end & acceptance
    11. 11. Create them for journeys, not stories • Too heavy, slow and expensive to create for each story • Must be tracked outside of stories
    12. 12. Who writes acceptance tests?
    13. 13. Those closest to the work should inplement the tests • More efficient • Most correct
    14. 14. A framework is key • Code reuse • Rapid development • Readable tests • More maintainable tests
    15. 15. Don’t go through the UI • Controller down, service down • Use mocks, stubs
    16. 16. What can you write them in? Vim
    17. 17. Part II How they work
    18. 18. Start with a story User story As an internet user, I want to search for “ALT.NET” And get valid results • Acceptance criteria 1: I get at least 100k results • Acceptance criteria 2: Results ordered by relevance … Scenarios Feature
    19. 19. Tracking AATs
    20. 20. Tracking AATs
    21. 21. Various Frameworks Browser driver Browser automation Test runner/harness BDD Framework .NET Selenium, Phantom.js Selenium, Waitn NUnit, xUnit, MSTest SpecFlow, Fitness (.NET runner), Cuke4Nuke, MSpec JavaScript Selemium, Phantom.js Phantom.js/Casp er.js Mocha, Jasmine Chai, Jasmine Java/Ruby Selenium, Phantom.js Selenium, Waitr JUnit, test-unit JBehave, Fitness, Cucumber/RSpec/ Capybara
    22. 22. Why Have AATs? (Pros)
    23. 23. Communication • Helps specify behavior of the system in plain text • Provides a medium for non-tech and devs to agree on Are we talking about the right system?
    24. 24. Seams, unit test mistakes
    25. 25. Automation “There’s no place for human beings to be doing regression testing manually.” -Jez Humble
    26. 26. Speedier deployments
    27. 27. Save resources
    28. 28. More testing during development
    29. 29. Why NOT Have AATs?
    30. 30. Why don’t more people use them?
    31. 31. Impediments • Poor adoption • Bandwidth/Velocity • Learning Curve/Experience • Business users won’t write specifications
    32. 32. Why NOT have AATs? • Maintenance • Speed • High false negatives, non-determinism
    33. 33. Questions?
    34. 34. Part IV Overcoming the Impediments
    35. 35. Use a headless browser
    36. 36. When Acceptances Tests catches a bug • See why bug got through unit/integration tests • Add unit, integration tests • Prune AAT? Unit
    37. 37. Concise Specs • Declarative • Reuse steps • Concise, many methods, fat fixtures Bad: • Click on Log In button • Click username box and type ‘myUsername’ and click ‘password’ box and type ‘myPassword’ • Click on link for Transfer Payment • Click box for amount and type 400 • Click ‘Transfer’ button • Assert success message Good: • Login as ‘MyAccountName’ • Navigate to Transfer Payment page • Transfer 400 dollars • Assert success message
    38. 38. Organize your tests… Use Page Objects Keeps each page’s tests • Encapsulated • Readable • Reusable
    39. 39. public class GoogleHomepage: PageBase, IGoogleHomepage { // Flow for this page } Domain Service UI GoogleHomepage +search(query) +clickSearch() +clickGettingLucky Data Access GoogleHomepageService +search(query) +clickSearch() +clickGettingLucky IGoogleHomepage +search(query) +clickSearch() +clickGettingLucky Page Objects
    40. 40. Page Objects Faster test creation – like Legos • Keeps step definitions very readable and thin
    41. 41. Hides details • Abstracts SpecFlow from Selenium • Keeps infrastructure at a lower level
    42. 42. Keeps changes minimal • Changes in the application should require minimal changes in the tests, and not in the specs
    43. 43. Easier to write new tests • Constructing page objects • Can be used to create a DSL
    44. 44. One way [When(@"I log in with the teacher account '(.*)'")] public void WhenILogInWithTheTeacherAccount(string orgCode) { var driver = new FirefixDriver(); driver.Navigate().GoToUrl("http://www.google.com"); var userTb = Driver.FindElement(By.Name("username”)); var passwordTb = Driver.FindElement(By.Id("password")); userTb.SendKeys(username); passwordTb.SendKeys(password); var loggedInLink = Driver.FindElement(By.LinkText("Log Out")); Assert.That(loggedInLink != null); }
    45. 45. Better way – page objects public class LogInPage : PageBase { public PageBase LogIn(string username, string password) { Driver.FindElement(By.Id("UserName")).SendKeys(username); Driver.FindElement(By.Id("Password")).SendKeys(password); Driver.FindElement(By.CssSelector("input[type="submit"]")).Click(); return GetInstance<AccountLandingPage>(Driver); } public void HasExpectedTab(string tabName) { var link = Driver.FindElement(By.LinkText(tabName)); Assert.AreEqual(tabName, link.Text); } }
    46. 46. Other things • Managed test data • Distributed execution
    47. 47. Tips
    48. 48. Developer tips • Don't run in same assembly as other tests, so these can be run separately since they're very slow (i.e. nightly) • Navigation • Create different levels of suites depending on depth/level of feedback desired: • Smoke, Current iteration/sprint, Regression • Run AATs as close to the real environment as possible
    49. 49. Gherkin Specs shouldn’t have much setup code Given I am on the homepage And I click login And I enter username of ‘username’ and password of ‘password’ When clicking ‘Log In’ Then take me to my landing page Given I am on the homepage When logging in with ‘username’ and ‘password’ Then take me to my landing page
    50. 50. UI tests • Sometimes tests will fail if the page doesn’t have enough time to load • Capture screen shots when tests fail • Run nightly • Run across multiple machines
    51. 51. Anti-patterns • Developers writing acceptance tests by themselves, for themselves • Tying AATs to an implementation - unit tests are implementation specific, AATs are NOT • Too much repetition in the gherkin • Too many UI tests • UI tests are failing too easily when changing the UI
    52. 52. Conclusion
    53. 53. Where we can be • Reliable tests • Reusable test code (even a DSL) • Separation of concerns • Short and clear tests • Fast test creation • Quick regression feedback • Tests that are understood by non-technical people • Fewer bugs
    54. 54. Resources: Tools used here: • SpecFlow – specflow.org • PhantomJS – phantomjs.org • Selenium WebDriver – seleniumhq.org Books: • Specification By Example, Gojko Adzic • Continuous Delivery, Jez Humble, David Farley • Growing Object-Oriented Software, Guided By Tests, Steve Freeman, Nat Pryce Articles: • Automated Acceptance Tests, http://www.thoughtworks.com/insights/articles/automated-acceptance-tests • BDD with SpecFlow and ASP.NET MVC, http://blog.stevensanderson.com/2010/03/03/behavior-driven- development-bdd-with-specflow-and-aspnet-mvc/ • Using Gherkin Language in SpecFlow, https://github.com/techtalk/SpecFlow/wiki/Using-Gherkin-Language-in-SpecFlow • BDD with SpecFlow, MSDN, http://msdn.microsoft.com/en-us/magazine/gg490346.aspx • Using SpecFlow with the Page Object, http://blogs.lessthandot.com/index.php/EnterpriseDev/application-lifecycle- management/using-specflow-to • Problems with Acceptance Tests, http://xprogramming.com/xpmag/problems-with-acceptance-testing • Top 10 reasons why teams fail with Acceptance Testing, http://gojko.net/2009/09/24/top-10-reasons-why-teams- fail-with-acceptance-testing/ • Maintaining Automated Acceptance Tests (ThoughtWorks), http://www.youtube.com/watch?v=uf0EVbH5hdA • Creating Maintainable Automated Acceptance Test Suites, Jez Humble, • http://www.youtube.com/watch?v=v-L_2y6g5DI
    55. 55. Happy Three Amigos Slides: http://www.slideshare.net/wynvandevanter/automated-acceptance- tests-in-net Code: https://github.com/Wyntuition/AATsInSpecFlowDemos wyn.vandevanter@excella.com @wynv

    ×