Automated Acceptance Tests in .NET

1. AUTOMATED
ACCEPTANCE TESTING IN
.NET
Wyn Van Devanter
wyn.vandevanter@excella.com
@wynv

2. Agenda
• Part I: AAT Overview
• Part II: Pros & Cons of AAT
• Part III: AAT Tools and Syntax in .NET
• Part IV: Overcoming the Cons of AAT
• Part V: Demo

3. Part I
AAT Overview

4. Legacy Nightmare
<div class="pdl13 pdr15">
<div class="fll w744">
<div class="wp100 bdr02 bgc03">
<div class="pd08 mostDownload01_l vat">
<div class="bgc02 vat">
<div class="mostDownload_m bgc02 vat">
<div class="wp100">
<div class="fll bgc05 pdr03 h20">

5. Automated Acceptance Tests (AATs)

6. Given – When – Then
Given I am hungry
When I eat 2 pickles
Then I am less hungry

7. End-to-end UI, Integration tests

8. UI
Service
Unit
Black box
White box
End-to-end
Business facing
Localized
Technology
facing

9. Manual Checking
End-to-end
UI
Integration
Unit
SOURCE: http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid

10. Manual
Checking
Inverting the Testing Pyramid
UI
Acceptance, Integration
Unit

11. Acceptance Test Strategy
Happy paths, major unhappy paths, legacy

12. Who writes acceptance tests?

13. What can you write them in?
Vim

14. Why Have AATs?

15. Communication
• Helps specify behavior of the system in plain text
• Provides a medium for non-tech and devs to agree on
Are we
talking
about
the right
system?

16. Seams, unit test mistakes

17. Automation
“There’s no place for
human beings to be
doing regression testing
manually.”
-Jez Humble

18. Save resources

19. Acceptance
Test Driven Development (ATDD)

20. Why NOT have ATTs?
• Business users won‟t write specifications
• Brittle and slow
• High false negatives, non-determinism
“Certainly it is not necessary to run every
example to be sure that the code works.
Probably it is necessary to run some.”
– Ron Jeffries

21. Questions?

22. Part II
Tools and Syntax

23. Various Frameworks
Browser driver
Browser automation
Test runner/harness
BDD Framework
.NET
Selenium, Phanto
m.js
Selenium,
Waitn
NUnit, xUnit, MST
est
SpecFlow,
Fitness (.NET
runner), Cuke4Nu
ke, MSpec
JavaScript
Selemium, Phant
om.js
Phantom.js/Casp
er.js
Mocha, Jasmine
Chai, Jasmine
Java/Ruby
Selenium, Phantom
.js
Selenium,
Waitr
JUnit, test-unit
JBehave, Fitness,
Cucumber/RSpec/
Capybara

24. Various Frameworks
Browser driver
Browser automation
Test runner/harness
BDD Framework
.NET
Selenium, Phant
om.js
Selenium,
Waitn
NUnit, xUnit, MS
Test
SpecFlow,
Fitness (.NET
runner), Cuke4Nu
ke,
MSpec
JavaScript
Selemium, Phant
om.js
Phantom.js/Casp
er.js
Mocha, Jasmine
Chai, Jasmine
Java/Ruby
Selenium,
Phantom.js
Selenium,
Waitr
JUnit, test-unit
JBehave, Fitness,
Cucumber/RSpec/
Capybara

25. App
UI/Integration Tests
Page Objects
Acceptance Tests
Architecture with SpecFlow

26. Getting set up

27. Getting set up

28. Getting set up

29. SpecFlow Workflow

30. Start with a story
User story
As an internet user,
I want to search for “ALT.NET”
And get valid results
• Acceptance criteria 1: I get at least 100k results
• Acceptance criteria 2: Results ordered by relevance
…
Scenarios
Feature

31. Gherkin in a SpecFlow feature
Feature: Google Search
As an internet user
I want to search for “ALT.NET”
So that I can be knowledgeable about the organzation
Background: I am logged into my Google account
Scenario: Search for „ALT.NET‟
Given I am on the Google Home Page
When I search for ”ALT.NET"
Then I should see at least 100,000 results
And the results should be ordered by relevance
Step

32. Generate Step Definitions
[When(@"I search for ""(.+)""")]
public void WhenISearchForSomething(string searchTerm)
{
GoogleSearchResultsPage = GoogleHomePage.Search(searchTerm);
}
Step Definition

33. Running tests

34. Demo

35. Test Data
• Populate a database with expected data for tests
• Separately, or on startup of all/single test(s) & remove in
tear down

36. Tracking AATs

37. Living documentation
• Pickles - Html, Word, DITA and json

38. Questions?

39. Part IV
Overcoming the Cons of
AATs

40. Minimize # of end-to-end tests
• AATs for journeys, not stories
• Is your new story entirely new?
• Balance high # of unit tests + selected end-to-end &
acceptance

41. Page Objects
public class GoogleHomepage: PageBase, IGoogleHomepage {
// Flow for this page
}
GoogleHomepage
+search(query)
+clickSearch()
+clickGettingLucky

42. SpecFlow Workflow with Page Object

43. public class GoogleHomepage: PageBase, IGoogleHomepage {
// Flow for this page
}
Domain
Service
UI
GoogleHomepage
+search(query)
+clickSearch()
+clickGettingLucky Data Access
GoogleHomepageService
+search(query)
+clickSearch()
+clickGettingLucky
IGoogleHomepage
+search(query)
+clickSearch()
+clickGettingLucky

44. Concise Specs
• Declarative
• Reuse steps
• Concise, many methods, fat fixtures
Bad:
• Click on Log In button
• Click username box and type „myUsername‟ and click „password‟ box and type
„myPassword‟
• Click on link for Transfer Payment
• Click box for amount and type 400
• Click „Transfer‟ button
• Assert success message
Good:
• Login as „MyAccountName‟
• Navigate to Transfer Payment page
• Transfer 400 dollars
• Assert success message

45. Use a headless browser

46. Don‟t go through the UI
• Controller down, service down
• Use mocks, stubs

47. When Acceptances Tests catches a bug
• See why bug got through unit/integration tests
• Add unit, integration tests
• Prune AAT?
Unit

48. Tips

49. Developer tips
• Don't run in same assembly as other tests, so these can
be run separately since they're very slow (i.e. nightly)
• Navigation
• Haywire! Sometimes when moving step definitions
around, the gherkin gets confused and can‟t find the
target. Restart VS.

50. Developer tips
[Given(@"I am not logged in")]
[When(@"I am not logged in")]
public void IAmNotLoggedIn()
{
…
}

51. Gherkin
• Describe what, not how (Don‟t: „Click the Log In button‟. Do:
„Navigate to the log in screen‟)
• Specs shouldn‟t have much setup code
Given I am on the homepage
And I click login
And I enter username of „username‟ and password of „password‟
When clicking „Log In‟
Then take me to my landing page
Given I am on the homepage
When logging in with „username‟ and „password‟
Then take me to my landing page

52. Scenario Outline
Parameterize!
Scenario: eat 5 out of 12
Given there are 12 cucumbers
When I eat 5 cucumbers
Then I should have 7 cucumbers
Scenario: eat 5 out of 20
Given there are 20 cucumbers
When I eat 5 cucumbers
Then I should have 15 cucumbers

53. Scenario Outline
Parameterize!
Scenario Outline: eating
Given there are <start> cucumbers
When I eat <eat> cucumbers
Then I should have <left> cucumbers
Examples:
| start | eat | left |
| 12 | 5 | 7 |
| 20 | 5 | 15 |

54. Parameters
Given I entered the following data into the new account form:
| Field | Value |
| Name | John Galt |
| Birthdate | 2/2/1902 |
| HeightInInches | 72 |
| BankAccountBalance | 1234.56 |
or in a horizontal table like this:
| Name | Birthdate | HeightInInches | BankAccountBalance |
| John Galt | 2/2/1902 | 72 | 1234.56 |
| John Doe | 1/1/1980 | 68 | 1111.99 |

55. UI tests
• Sometimes tests will fail if the page doesn‟t have enough
time to load. Use implicit waits, and explicit when really
needed
• Capture screen shots when tests fail
• Selenium IDE

56. Anti-patterns
• Developers writing acceptance tests by themselves, for
themselves
• Tying AATs to an implementation - unit tests are
implementation specific, AATs are NOT
• Too much repetition in the gherkin
• Too many UI tests

57. Demo

58. Conclusion

59. Resources:
Tools used here:
• SpecFlow – specflow.org
• PhantomJS – phantomjs.org
• Selenium WebDriver – seleniumhq.org
Books:
• Specification By Example, Gojko Adzic
• Continuous Delivery, Jez Humble, David Farley
• Growing Object-Oriented Software, Guided By Tests, Steve Freeman, Nat Pryce
Articles:
• Automated Acceptance Tests, http://www.thoughtworks.com/insights/articles/automated-acceptance-tests
• BDD with SpecFlow and ASP.NET MVC, http://blog.stevensanderson.com/2010/03/03/behavior-driven-
development-bdd-with-specflow-and-aspnet-mvc/
• Using Gherkin Language in SpecFlow,
https://github.com/techtalk/SpecFlow/wiki/Using-Gherkin-Language-in-SpecFlow
• BDD with SpecFlow, MSDN, http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
• Using SpecFlow with the Page Object, http://blogs.lessthandot.com/index.php/EnterpriseDev/application-lifecycle-
management/using-specflow-to
• Problems with Acceptance Tests, http://xprogramming.com/xpmag/problems-with-acceptance-testing
• Top 10 reasons why teams fail with Acceptance Testing, http://gojko.net/2009/09/24/top-10-reasons-why-teams-
fail-with-acceptance-testing/
• Maintaining Automated Acceptance Tests (ThoughtWorks), http://www.youtube.com/watch?v=uf0EVbH5hdA
• Creating Maintainable Automated Acceptance Test Suites, Jez Humble,
• http://www.youtube.com/watch?v=v-L_2y6g5DI

60. Happy Three Amigos
Slides: http://www.slideshare.net/wynvandevanter/automated-acceptance-
tests-in-net
Code:
https://github.com/Wyntuition/AATsInSpecFlowDemos
wyn.vandevanter@excella.com
@wynv