Automated Acceptance Tests in .NET

Uploaded on

Automated Acceptance Tests in .NET

Automated Acceptance Tests in .NET

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • Ask Q’s
  • Bad legacy code in mission critical system, not unit testable…Useful in general in a test suite for many reasons…
  • AcceptanceTestsFits their requirements. The story is complete. Automated
  • Readable by whole spectrum
  • What kind of tests do you run with your specs?UI – smoke, sanity, UI goo Functionality – Integration testsRun close to production; like a user would.
  • Mostly unit testsHARD to maintain, slow SO….
  • Unit test everything thru JavaScriptMost RELEVANT test cases for AATs (can’t hit all alt paths)Shouldn’t manually regression test
  • Whoever writes user storiesDev/Tester/BA BEFORE iteration starts
  • -Communication with business stakeholders, BAs, QA
  • -Unit tests don’t account for everything, seams-Gaps in coverage, unit tests not hitting right test cases,
  • -Full regression doesn’t always happen-Manual testing lapseMistakes in testing, non-testing, non-regressingAllow testers to do more exploratory and higher level things, less boring
  • Bugs will be caught earlier
  • Not doing, btw
  • Some have even said it’s not worth the cost
  • Create a spec in gherkinGerkin is a line-oriented language (line  step)In SpecFlow, Generate step definitionsIn Selenium & NUnit, write UI/integration tests to power the AATs
  • All know definition of User Story? Should be turned into specs before iteration starts (bus & )
  • Given – set up the context of your testWhen – action performed by userThen - asserts
  • Represents flow of pages in app
  • ----- Meeting Notes (7/2/13 12:08) ------spelling


  • 2. Agenda • Part I: AAT Overview • Part II: Pros & Cons of AAT • Part III: AAT Tools and Syntax in .NET • Part IV: Overcoming the Cons of AAT • Part V: Demo
  • 3. Part I AAT Overview
  • 4. Legacy Nightmare <div class="pdl13 pdr15"> <div class="fll w744"> <div class="wp100 bdr02 bgc03"> <div class="pd08 mostDownload01_l vat"> <div class="bgc02 vat"> <div class="mostDownload_m bgc02 vat"> <div class="wp100"> <div class="fll bgc05 pdr03 h20">
  • 5. Automated Acceptance Tests (AATs)
  • 6. Given – When – Then Given I am hungry When I eat 2 pickles Then I am less hungry
  • 7. End-to-end UI, Integration tests
  • 8. UI Service Unit Black box White box End-to-end Business facing Localized Technology facing
  • 9. Manual Checking End-to-end UI Integration Unit SOURCE:
  • 10. Manual Checking Inverting the Testing Pyramid UI Acceptance, Integration Unit
  • 11. Acceptance Test Strategy Happy paths, major unhappy paths, legacy
  • 12. Who writes acceptance tests?
  • 13. What can you write them in? Vim
  • 14. Why Have AATs?
  • 15. Communication • Helps specify behavior of the system in plain text • Provides a medium for non-tech and devs to agree on Are we talking about the right system?
  • 16. Seams, unit test mistakes
  • 17. Automation “There’s no place for human beings to be doing regression testing manually.” -Jez Humble
  • 18. Save resources
  • 19. Acceptance Test Driven Development (ATDD)
  • 20. Why NOT have ATTs? • Business users won‟t write specifications • Brittle and slow • High false negatives, non-determinism “Certainly it is not necessary to run every example to be sure that the code works. Probably it is necessary to run some.” – Ron Jeffries
  • 21. Questions?
  • 22. Part II Tools and Syntax
  • 23. Various Frameworks Browser driver Browser automation Test runner/harness BDD Framework .NET Selenium, Phanto m.js Selenium, Waitn NUnit, xUnit, MST est SpecFlow, Fitness (.NET runner), Cuke4Nu ke, MSpec JavaScript Selemium, Phant om.js Phantom.js/Casp er.js Mocha, Jasmine Chai, Jasmine Java/Ruby Selenium, Phantom .js Selenium, Waitr JUnit, test-unit JBehave, Fitness, Cucumber/RSpec/ Capybara
  • 24. Various Frameworks Browser driver Browser automation Test runner/harness BDD Framework .NET Selenium, Phant om.js Selenium, Waitn NUnit, xUnit, MS Test SpecFlow, Fitness (.NET runner), Cuke4Nu ke, MSpec JavaScript Selemium, Phant om.js Phantom.js/Casp er.js Mocha, Jasmine Chai, Jasmine Java/Ruby Selenium, Phantom.js Selenium, Waitr JUnit, test-unit JBehave, Fitness, Cucumber/RSpec/ Capybara
  • 25. App UI/Integration Tests Page Objects Acceptance Tests Architecture with SpecFlow
  • 26. Getting set up
  • 27. Getting set up
  • 28. Getting set up
  • 29. SpecFlow Workflow
  • 30. Start with a story User story As an internet user, I want to search for “ALT.NET” And get valid results • Acceptance criteria 1: I get at least 100k results • Acceptance criteria 2: Results ordered by relevance … Scenarios Feature
  • 31. Gherkin in a SpecFlow feature Feature: Google Search As an internet user I want to search for “ALT.NET” So that I can be knowledgeable about the organzation Background: I am logged into my Google account Scenario: Search for „ALT.NET‟ Given I am on the Google Home Page When I search for ”ALT.NET" Then I should see at least 100,000 results And the results should be ordered by relevance Step
  • 32. Generate Step Definitions [When(@"I search for ""(.+)""")] public void WhenISearchForSomething(string searchTerm) { GoogleSearchResultsPage = GoogleHomePage.Search(searchTerm); } Step Definition
  • 33. Running tests
  • 34. Demo
  • 35. Test Data • Populate a database with expected data for tests • Separately, or on startup of all/single test(s) & remove in tear down
  • 36. Tracking AATs
  • 37. Living documentation • Pickles - Html, Word, DITA and json
  • 38. Questions?
  • 39. Part IV Overcoming the Cons of AATs
  • 40. Minimize # of end-to-end tests • AATs for journeys, not stories • Is your new story entirely new? • Balance high # of unit tests + selected end-to-end & acceptance
  • 41. Page Objects public class GoogleHomepage: PageBase, IGoogleHomepage { // Flow for this page } GoogleHomepage +search(query) +clickSearch() +clickGettingLucky
  • 42. SpecFlow Workflow with Page Object
  • 43. public class GoogleHomepage: PageBase, IGoogleHomepage { // Flow for this page } Domain Service UI GoogleHomepage +search(query) +clickSearch() +clickGettingLucky Data Access GoogleHomepageService +search(query) +clickSearch() +clickGettingLucky IGoogleHomepage +search(query) +clickSearch() +clickGettingLucky
  • 44. Concise Specs • Declarative • Reuse steps • Concise, many methods, fat fixtures Bad: • Click on Log In button • Click username box and type „myUsername‟ and click „password‟ box and type „myPassword‟ • Click on link for Transfer Payment • Click box for amount and type 400 • Click „Transfer‟ button • Assert success message Good: • Login as „MyAccountName‟ • Navigate to Transfer Payment page • Transfer 400 dollars • Assert success message
  • 45. Use a headless browser
  • 46. Don‟t go through the UI • Controller down, service down • Use mocks, stubs
  • 47. When Acceptances Tests catches a bug • See why bug got through unit/integration tests • Add unit, integration tests • Prune AAT? Unit
  • 48. Tips
  • 49. Developer tips • Don't run in same assembly as other tests, so these can be run separately since they're very slow (i.e. nightly) • Navigation • Haywire! Sometimes when moving step definitions around, the gherkin gets confused and can‟t find the target. Restart VS.
  • 50. Developer tips [Given(@"I am not logged in")] [When(@"I am not logged in")] public void IAmNotLoggedIn() { … }
  • 51. Gherkin • Describe what, not how (Don‟t: „Click the Log In button‟. Do: „Navigate to the log in screen‟) • Specs shouldn‟t have much setup code Given I am on the homepage And I click login And I enter username of „username‟ and password of „password‟ When clicking „Log In‟ Then take me to my landing page Given I am on the homepage When logging in with „username‟ and „password‟ Then take me to my landing page
  • 52. Scenario Outline Parameterize! Scenario: eat 5 out of 12 Given there are 12 cucumbers When I eat 5 cucumbers Then I should have 7 cucumbers Scenario: eat 5 out of 20 Given there are 20 cucumbers When I eat 5 cucumbers Then I should have 15 cucumbers
  • 53. Scenario Outline Parameterize! Scenario Outline: eating Given there are <start> cucumbers When I eat <eat> cucumbers Then I should have <left> cucumbers Examples: | start | eat | left | | 12 | 5 | 7 | | 20 | 5 | 15 |
  • 54. Parameters Given I entered the following data into the new account form: | Field | Value | | Name | John Galt | | Birthdate | 2/2/1902 | | HeightInInches | 72 | | BankAccountBalance | 1234.56 | or in a horizontal table like this: | Name | Birthdate | HeightInInches | BankAccountBalance | | John Galt | 2/2/1902 | 72 | 1234.56 | | John Doe | 1/1/1980 | 68 | 1111.99 |
  • 55. UI tests • Sometimes tests will fail if the page doesn‟t have enough time to load. Use implicit waits, and explicit when really needed • Capture screen shots when tests fail • Selenium IDE
  • 56. Anti-patterns • Developers writing acceptance tests by themselves, for themselves • Tying AATs to an implementation - unit tests are implementation specific, AATs are NOT • Too much repetition in the gherkin • Too many UI tests
  • 57. Demo
  • 58. Conclusion
  • 59. Resources: Tools used here: • SpecFlow – • PhantomJS – • Selenium WebDriver – Books: • Specification By Example, Gojko Adzic • Continuous Delivery, Jez Humble, David Farley • Growing Object-Oriented Software, Guided By Tests, Steve Freeman, Nat Pryce Articles: • Automated Acceptance Tests, • BDD with SpecFlow and ASP.NET MVC, development-bdd-with-specflow-and-aspnet-mvc/ • Using Gherkin Language in SpecFlow, • BDD with SpecFlow, MSDN, • Using SpecFlow with the Page Object, management/using-specflow-to • Problems with Acceptance Tests, • Top 10 reasons why teams fail with Acceptance Testing, fail-with-acceptance-testing/ • Maintaining Automated Acceptance Tests (ThoughtWorks), • Creating Maintainable Automated Acceptance Test Suites, Jez Humble, •
  • 60. Happy Three Amigos Slides: tests-in-net Code: @wynv