AUTOMATED
ACCEPTANCE TESTING IN
.NET
Wyn Van Devanter
wyn.vandevanter@excella.com
@wynv
Agenda
• Part I: AAT Overview
• Part II: Pros & Cons of AAT
• Part III: AAT Tools and Syntax in .NET
• Part IV: Overcoming...
Part I
AAT Overview
Legacy Nightmare
<div class="pdl13 pdr15">
<div class="fll w744">
<div class="wp100 bdr02 bgc03">
<div class="pd08 mostDow...
Automated Acceptance Tests (AATs)
Given – When – Then
Given I am hungry
When I eat 2 pickles
Then I am less hungry
End-to-end UI, Integration tests
UI
Service
Unit
Black box
White box
End-to-end
Business facing
Localized
Technology
facing
Manual Checking
End-to-end
UI
Integration
Unit
SOURCE: http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid
Manual
Checking
Inverting the Testing Pyramid
UI
Acceptance, Integration
Unit
Acceptance Test Strategy
Happy paths, major unhappy paths, legacy
Who writes acceptance tests?
What can you write them in?
Vim
Why Have AATs?
Communication
• Helps specify behavior of the system in plain text
• Provides a medium for non-tech and devs to agree on
A...
Seams, unit test mistakes
Automation
“There’s no place for
human beings to be
doing regression testing
manually.”
-Jez Humble
Save resources
Acceptance
Test Driven Development (ATDD)
Why NOT have ATTs?
• Business users won‟t write specifications
• Brittle and slow
• High false negatives, non-determinism
...
Questions?
Part II
Tools and Syntax
Various Frameworks
Browser driver
Browser automation
Test runner/harness
BDD Framework
.NET
Selenium, Phanto
m.js
Selenium...
Various Frameworks
Browser driver
Browser automation
Test runner/harness
BDD Framework
.NET
Selenium, Phant
om.js
Selenium...
App
UI/Integration Tests
Page Objects
Acceptance Tests
Architecture with SpecFlow
Getting set up
Getting set up
Getting set up
SpecFlow Workflow
Start with a story
User story
As an internet user,
I want to search for “ALT.NET”
And get valid results
• Acceptance crite...
Gherkin in a SpecFlow feature
Feature: Google Search
As an internet user
I want to search for “ALT.NET”
So that I can be k...
Generate Step Definitions
[When(@"I search for ""(.+)""")]
public void WhenISearchForSomething(string searchTerm)
{
Google...
Running tests
Demo
Test Data
• Populate a database with expected data for tests
• Separately, or on startup of all/single test(s) & remove in...
Tracking AATs
Living documentation
• Pickles - Html, Word, DITA and json
Questions?
Part IV
Overcoming the Cons of
AATs
Minimize # of end-to-end tests
• AATs for journeys, not stories
• Is your new story entirely new?
• Balance high # of unit...
Page Objects
public class GoogleHomepage: PageBase, IGoogleHomepage {
// Flow for this page
}
GoogleHomepage
+search(query...
SpecFlow Workflow with Page Object
public class GoogleHomepage: PageBase, IGoogleHomepage {
// Flow for this page
}
Domain
Service
UI
GoogleHomepage
+search(...
Concise Specs
• Declarative
• Reuse steps
• Concise, many methods, fat fixtures
Bad:
• Click on Log In button
• Click user...
Use a headless browser
Don‟t go through the UI
• Controller down, service down
• Use mocks, stubs
When Acceptances Tests catches a bug
• See why bug got through unit/integration tests
• Add unit, integration tests
• Prun...
Tips
Developer tips
• Don't run in same assembly as other tests, so these can
be run separately since they're very slow (i.e. n...
Developer tips
[Given(@"I am not logged in")]
[When(@"I am not logged in")]
public void IAmNotLoggedIn()
{
…
}
Gherkin
• Describe what, not how (Don‟t: „Click the Log In button‟. Do:
„Navigate to the log in screen‟)
• Specs shouldn‟t...
Scenario Outline
Parameterize!
Scenario: eat 5 out of 12
Given there are 12 cucumbers
When I eat 5 cucumbers
Then I should...
Scenario Outline
Parameterize!
Scenario Outline: eating
Given there are <start> cucumbers
When I eat <eat> cucumbers
Then ...
Parameters
Given I entered the following data into the new account form:
| Field | Value |
| Name | John Galt |
| Birthdat...
UI tests
• Sometimes tests will fail if the page doesn‟t have enough
time to load. Use implicit waits, and explicit when r...
Anti-patterns
• Developers writing acceptance tests by themselves, for
themselves
• Tying AATs to an implementation - unit...
Demo
Conclusion
Resources:
Tools used here:
• SpecFlow – specflow.org
• PhantomJS – phantomjs.org
• Selenium WebDriver – seleniumhq.org
Bo...
Happy Three Amigos
Slides: http://www.slideshare.net/wynvandevanter/automated-acceptance-
tests-in-net
Code:
https://githu...
Upcoming SlideShare
Loading in …5
×

Automated Acceptance Tests in .NET

4,241 views

Published on

Automated Acceptance Tests in .NET

1 Comment
5 Likes
Statistics
Notes
No Downloads
Views
Total views
4,241
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
0
Comments
1
Likes
5
Embeds 0
No embeds

No notes for slide
  • Ask Q’s
  • Bad legacy code in mission critical system, not unit testable…Useful in general in a test suite for many reasons…
  • AcceptanceTestsFits their requirements. The story is complete. Automated
  • Readable by whole spectrum
  • What kind of tests do you run with your specs?UI – smoke, sanity, UI goo Functionality – Integration testsRun close to production; like a user would.
  • Mostly unit testsHARD to maintain, slow SO….
  • Unit test everything thru JavaScriptMost RELEVANT test cases for AATs (can’t hit all alt paths)Shouldn’t manually regression test
  • Whoever writes user storiesDev/Tester/BA BEFORE iteration starts
  • -Communication with business stakeholders, BAs, QA
  • -Unit tests don’t account for everything, seams-Gaps in coverage, unit tests not hitting right test cases,
  • -Full regression doesn’t always happen-Manual testing lapseMistakes in testing, non-testing, non-regressingAllow testers to do more exploratory and higher level things, less boring
  • Bugs will be caught earlier
  • Not doing, btw
  • Some have even said it’s not worth the cost
  • Create a spec in gherkinGerkin is a line-oriented language (line  step)In SpecFlow, Generate step definitionsIn Selenium &amp; NUnit, write UI/integration tests to power the AATs
  • All know definition of User Story? Should be turned into specs before iteration starts (bus &amp; )
  • Given – set up the context of your testWhen – action performed by userThen - asserts
  • Represents flow of pages in app
  • ----- Meeting Notes (7/2/13 12:08) ------spelling
  • Automated Acceptance Tests in .NET

    1. 1. AUTOMATED ACCEPTANCE TESTING IN .NET Wyn Van Devanter wyn.vandevanter@excella.com @wynv
    2. 2. Agenda • Part I: AAT Overview • Part II: Pros & Cons of AAT • Part III: AAT Tools and Syntax in .NET • Part IV: Overcoming the Cons of AAT • Part V: Demo
    3. 3. Part I AAT Overview
    4. 4. Legacy Nightmare <div class="pdl13 pdr15"> <div class="fll w744"> <div class="wp100 bdr02 bgc03"> <div class="pd08 mostDownload01_l vat"> <div class="bgc02 vat"> <div class="mostDownload_m bgc02 vat"> <div class="wp100"> <div class="fll bgc05 pdr03 h20">
    5. 5. Automated Acceptance Tests (AATs)
    6. 6. Given – When – Then Given I am hungry When I eat 2 pickles Then I am less hungry
    7. 7. End-to-end UI, Integration tests
    8. 8. UI Service Unit Black box White box End-to-end Business facing Localized Technology facing
    9. 9. Manual Checking End-to-end UI Integration Unit SOURCE: http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid
    10. 10. Manual Checking Inverting the Testing Pyramid UI Acceptance, Integration Unit
    11. 11. Acceptance Test Strategy Happy paths, major unhappy paths, legacy
    12. 12. Who writes acceptance tests?
    13. 13. What can you write them in? Vim
    14. 14. Why Have AATs?
    15. 15. Communication • Helps specify behavior of the system in plain text • Provides a medium for non-tech and devs to agree on Are we talking about the right system?
    16. 16. Seams, unit test mistakes
    17. 17. Automation “There’s no place for human beings to be doing regression testing manually.” -Jez Humble
    18. 18. Save resources
    19. 19. Acceptance Test Driven Development (ATDD)
    20. 20. Why NOT have ATTs? • Business users won‟t write specifications • Brittle and slow • High false negatives, non-determinism “Certainly it is not necessary to run every example to be sure that the code works. Probably it is necessary to run some.” – Ron Jeffries
    21. 21. Questions?
    22. 22. Part II Tools and Syntax
    23. 23. Various Frameworks Browser driver Browser automation Test runner/harness BDD Framework .NET Selenium, Phanto m.js Selenium, Waitn NUnit, xUnit, MST est SpecFlow, Fitness (.NET runner), Cuke4Nu ke, MSpec JavaScript Selemium, Phant om.js Phantom.js/Casp er.js Mocha, Jasmine Chai, Jasmine Java/Ruby Selenium, Phantom .js Selenium, Waitr JUnit, test-unit JBehave, Fitness, Cucumber/RSpec/ Capybara
    24. 24. Various Frameworks Browser driver Browser automation Test runner/harness BDD Framework .NET Selenium, Phant om.js Selenium, Waitn NUnit, xUnit, MS Test SpecFlow, Fitness (.NET runner), Cuke4Nu ke, MSpec JavaScript Selemium, Phant om.js Phantom.js/Casp er.js Mocha, Jasmine Chai, Jasmine Java/Ruby Selenium, Phantom.js Selenium, Waitr JUnit, test-unit JBehave, Fitness, Cucumber/RSpec/ Capybara
    25. 25. App UI/Integration Tests Page Objects Acceptance Tests Architecture with SpecFlow
    26. 26. Getting set up
    27. 27. Getting set up
    28. 28. Getting set up
    29. 29. SpecFlow Workflow
    30. 30. Start with a story User story As an internet user, I want to search for “ALT.NET” And get valid results • Acceptance criteria 1: I get at least 100k results • Acceptance criteria 2: Results ordered by relevance … Scenarios Feature
    31. 31. Gherkin in a SpecFlow feature Feature: Google Search As an internet user I want to search for “ALT.NET” So that I can be knowledgeable about the organzation Background: I am logged into my Google account Scenario: Search for „ALT.NET‟ Given I am on the Google Home Page When I search for ”ALT.NET" Then I should see at least 100,000 results And the results should be ordered by relevance Step
    32. 32. Generate Step Definitions [When(@"I search for ""(.+)""")] public void WhenISearchForSomething(string searchTerm) { GoogleSearchResultsPage = GoogleHomePage.Search(searchTerm); } Step Definition
    33. 33. Running tests
    34. 34. Demo
    35. 35. Test Data • Populate a database with expected data for tests • Separately, or on startup of all/single test(s) & remove in tear down
    36. 36. Tracking AATs
    37. 37. Living documentation • Pickles - Html, Word, DITA and json
    38. 38. Questions?
    39. 39. Part IV Overcoming the Cons of AATs
    40. 40. Minimize # of end-to-end tests • AATs for journeys, not stories • Is your new story entirely new? • Balance high # of unit tests + selected end-to-end & acceptance
    41. 41. Page Objects public class GoogleHomepage: PageBase, IGoogleHomepage { // Flow for this page } GoogleHomepage +search(query) +clickSearch() +clickGettingLucky
    42. 42. SpecFlow Workflow with Page Object
    43. 43. public class GoogleHomepage: PageBase, IGoogleHomepage { // Flow for this page } Domain Service UI GoogleHomepage +search(query) +clickSearch() +clickGettingLucky Data Access GoogleHomepageService +search(query) +clickSearch() +clickGettingLucky IGoogleHomepage +search(query) +clickSearch() +clickGettingLucky
    44. 44. Concise Specs • Declarative • Reuse steps • Concise, many methods, fat fixtures Bad: • Click on Log In button • Click username box and type „myUsername‟ and click „password‟ box and type „myPassword‟ • Click on link for Transfer Payment • Click box for amount and type 400 • Click „Transfer‟ button • Assert success message Good: • Login as „MyAccountName‟ • Navigate to Transfer Payment page • Transfer 400 dollars • Assert success message
    45. 45. Use a headless browser
    46. 46. Don‟t go through the UI • Controller down, service down • Use mocks, stubs
    47. 47. When Acceptances Tests catches a bug • See why bug got through unit/integration tests • Add unit, integration tests • Prune AAT? Unit
    48. 48. Tips
    49. 49. Developer tips • Don't run in same assembly as other tests, so these can be run separately since they're very slow (i.e. nightly) • Navigation • Haywire! Sometimes when moving step definitions around, the gherkin gets confused and can‟t find the target. Restart VS.
    50. 50. Developer tips [Given(@"I am not logged in")] [When(@"I am not logged in")] public void IAmNotLoggedIn() { … }
    51. 51. Gherkin • Describe what, not how (Don‟t: „Click the Log In button‟. Do: „Navigate to the log in screen‟) • Specs shouldn‟t have much setup code Given I am on the homepage And I click login And I enter username of „username‟ and password of „password‟ When clicking „Log In‟ Then take me to my landing page Given I am on the homepage When logging in with „username‟ and „password‟ Then take me to my landing page
    52. 52. Scenario Outline Parameterize! Scenario: eat 5 out of 12 Given there are 12 cucumbers When I eat 5 cucumbers Then I should have 7 cucumbers Scenario: eat 5 out of 20 Given there are 20 cucumbers When I eat 5 cucumbers Then I should have 15 cucumbers
    53. 53. Scenario Outline Parameterize! Scenario Outline: eating Given there are <start> cucumbers When I eat <eat> cucumbers Then I should have <left> cucumbers Examples: | start | eat | left | | 12 | 5 | 7 | | 20 | 5 | 15 |
    54. 54. Parameters Given I entered the following data into the new account form: | Field | Value | | Name | John Galt | | Birthdate | 2/2/1902 | | HeightInInches | 72 | | BankAccountBalance | 1234.56 | or in a horizontal table like this: | Name | Birthdate | HeightInInches | BankAccountBalance | | John Galt | 2/2/1902 | 72 | 1234.56 | | John Doe | 1/1/1980 | 68 | 1111.99 |
    55. 55. UI tests • Sometimes tests will fail if the page doesn‟t have enough time to load. Use implicit waits, and explicit when really needed • Capture screen shots when tests fail • Selenium IDE
    56. 56. Anti-patterns • Developers writing acceptance tests by themselves, for themselves • Tying AATs to an implementation - unit tests are implementation specific, AATs are NOT • Too much repetition in the gherkin • Too many UI tests
    57. 57. Demo
    58. 58. Conclusion
    59. 59. Resources: Tools used here: • SpecFlow – specflow.org • PhantomJS – phantomjs.org • Selenium WebDriver – seleniumhq.org Books: • Specification By Example, Gojko Adzic • Continuous Delivery, Jez Humble, David Farley • Growing Object-Oriented Software, Guided By Tests, Steve Freeman, Nat Pryce Articles: • Automated Acceptance Tests, http://www.thoughtworks.com/insights/articles/automated-acceptance-tests • BDD with SpecFlow and ASP.NET MVC, http://blog.stevensanderson.com/2010/03/03/behavior-driven- development-bdd-with-specflow-and-aspnet-mvc/ • Using Gherkin Language in SpecFlow, https://github.com/techtalk/SpecFlow/wiki/Using-Gherkin-Language-in-SpecFlow • BDD with SpecFlow, MSDN, http://msdn.microsoft.com/en-us/magazine/gg490346.aspx • Using SpecFlow with the Page Object, http://blogs.lessthandot.com/index.php/EnterpriseDev/application-lifecycle- management/using-specflow-to • Problems with Acceptance Tests, http://xprogramming.com/xpmag/problems-with-acceptance-testing • Top 10 reasons why teams fail with Acceptance Testing, http://gojko.net/2009/09/24/top-10-reasons-why-teams- fail-with-acceptance-testing/ • Maintaining Automated Acceptance Tests (ThoughtWorks), http://www.youtube.com/watch?v=uf0EVbH5hdA • Creating Maintainable Automated Acceptance Test Suites, Jez Humble, • http://www.youtube.com/watch?v=v-L_2y6g5DI
    60. 60. Happy Three Amigos Slides: http://www.slideshare.net/wynvandevanter/automated-acceptance- tests-in-net Code: https://github.com/Wyntuition/AATsInSpecFlowDemos wyn.vandevanter@excella.com @wynv

    ×