The document discusses automated acceptance testing (AAT) in .NET. It provides an overview of AAT, including the benefits of communication, catching mistakes, and automation. It then discusses various AAT tools and frameworks for .NET, with a focus on SpecFlow for writing tests in Gherkin. The document demonstrates SpecFlow's workflow and syntax. It concludes by addressing challenges with AAT and providing tips for overcoming issues like brittle tests through techniques like page objects and headless browsers.
Automated Acceptance Testing (and tool choice)
Automated acceptance testing has many names: acceptance-test driven development (ATDD), story-test driven development (STDD), agile acceptance testing and, most recently, specification by example. At the heart of all these approaches is to produce business-facing tests which are system tests running end-to-end, picking up regression issues and improving confidence that the code works as required.
In this talk, I will contextualise how each of these approaches share in common a three-tier layering strategy: acceptance criteria, test implementation layer and application driver layer. This is important because applying this approach requires a tool choice and each tool tends to have its own sweet (and blind) spot that is best understood through these layers.
I will first deep dive into sample code across a few tools (Cucumber, Fitnesse, Concordion) to illustrate this layering. I use an example that shows how to decouple the GUI from tests (window driver pattern).
Finally, I will look at some typical client scenarios to examine which tools might best suited because tool choice is not simply a host operating system question (.Net, Java, Ruby).
BDD (Behavior-driven development), also known as ATDD (Acceptance test–driven development) is a software development process that helps Agile teams design, develop, test and deliver software efficiently, it's about how to implement agile acceptance testing and binding business requirements to code. BDD helps to bridge the communication gap between stakeholders and implementation teams, build quality into software from the start, design, develop and deliver systems fit for purpose. SpecFlow is the official tool to implement BDD on the .NET platform.
Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016Steven Smith
Unit testing software can be difficult, especially when the software wasn't designed to be testable. Dependencies on infrastructure concerns and software we don't control are one of the biggest contributors to testing difficulty. In this session, you'll learn the difference between unit tests and other kinds of tests, how to recognize and invert dependencies, and how to unit test your code's interactions with these dependencies without testing the infrastructure itself.
Improving the Quality of Existing SoftwareSteven Smith
Given at DogFoodCon 2016 in Columbus, Ohio
As developers, most of our time is spent working on existing software – even if it’s just the software we wrote ourselves, yesterday. And over time, software rots. If were not diligent, our beautiful code can degrade into a worthless mess. Keeping our code in working condition is no different than changing the oil in our car “ its preventive maintenance. In this session, Steve will cover some common places to look for signs of degradation in existing applications, and describe the steps we can take to improve our code. Examples will use C# and primarily ASP.NET.
Automated Acceptance Testing (and tool choice)
Automated acceptance testing has many names: acceptance-test driven development (ATDD), story-test driven development (STDD), agile acceptance testing and, most recently, specification by example. At the heart of all these approaches is to produce business-facing tests which are system tests running end-to-end, picking up regression issues and improving confidence that the code works as required.
In this talk, I will contextualise how each of these approaches share in common a three-tier layering strategy: acceptance criteria, test implementation layer and application driver layer. This is important because applying this approach requires a tool choice and each tool tends to have its own sweet (and blind) spot that is best understood through these layers.
I will first deep dive into sample code across a few tools (Cucumber, Fitnesse, Concordion) to illustrate this layering. I use an example that shows how to decouple the GUI from tests (window driver pattern).
Finally, I will look at some typical client scenarios to examine which tools might best suited because tool choice is not simply a host operating system question (.Net, Java, Ruby).
BDD (Behavior-driven development), also known as ATDD (Acceptance test–driven development) is a software development process that helps Agile teams design, develop, test and deliver software efficiently, it's about how to implement agile acceptance testing and binding business requirements to code. BDD helps to bridge the communication gap between stakeholders and implementation teams, build quality into software from the start, design, develop and deliver systems fit for purpose. SpecFlow is the official tool to implement BDD on the .NET platform.
Breaking Dependencies to Allow Unit Testing - DevIntersection Spring 2016Steven Smith
Unit testing software can be difficult, especially when the software wasn't designed to be testable. Dependencies on infrastructure concerns and software we don't control are one of the biggest contributors to testing difficulty. In this session, you'll learn the difference between unit tests and other kinds of tests, how to recognize and invert dependencies, and how to unit test your code's interactions with these dependencies without testing the infrastructure itself.
Improving the Quality of Existing SoftwareSteven Smith
Given at DogFoodCon 2016 in Columbus, Ohio
As developers, most of our time is spent working on existing software – even if it’s just the software we wrote ourselves, yesterday. And over time, software rots. If were not diligent, our beautiful code can degrade into a worthless mess. Keeping our code in working condition is no different than changing the oil in our car “ its preventive maintenance. In this session, Steve will cover some common places to look for signs of degradation in existing applications, and describe the steps we can take to improve our code. Examples will use C# and primarily ASP.NET.
Improving The Quality of Existing SoftwareSteven Smith
Presented at DevReach 2013.
As developers, most of our time is spent working on existing software. Sure, occasionally we get the chance to fire up a new Solution in Visual Studio, and that can be exciting. But after the first day, we find ourselves once more having to deal with the work we did yesterday. And today, we know more than we did yesterday, so there are things we’d do differently, if we had it to do over.
Over time, software rots. If we’re not diligent, our beautiful code can degrade into a worthless mess. Keeping our code in working condition is no different than changing the oil in our car – it’s preventive maintenance. In this session, Steve will look at some common places to look for signs of degradation in existing applications, and steps to take to improve the code. Examples will use C# and primarily ASP.NET.
Improving the Quality of Existing Software - DevIntersection April 2016Steven Smith
How do you improve the quality of your existing software, while continuing to add value for your customers? What are some heuristics and code smells you can look for, and principles and patterns you can use to guide you, as you make your software better over time instead of worse? How can we improve our skills and techniques so that writing high quality software becomes our default, fastest way of working?
Improving the Quality of Existing SoftwareSteven Smith
How do you improve the quality of your existing software, while continuing to add value for your customers? What are some heuristics and code smells you can look for, and principles and patterns you can use to guide you, as you make your software better over time instead of worse?
How BDD with tools like Cucumber can create a stronger team, a better quality product, and ultimately a more useable API. Given at the #apistrat SF conference 10/24/2013
Continuous Testing Meets the Classroom at Code.orgSauce Labs
Code.org's Brian Jordan, a Software Engineer, takes the audience for a fun tour of Code.org’s continuous, automated testing suite. Brian discusses how Code.org approaches testing throughout the product development cycle, given their unique testing challenges—developing interactive, game-like curriculum for just the types of browsers you’d expect to find in school computer labs—from Internet Explorer 9 to iPads across 40+ languages.
What makes up an acceptance testing framework? Especially one that will help you use Selenium successfully? How about a list of what ones currently exist? We've got you covered.
This talk was prepared for the DC Selenium Meetup in April 2013.
DevQA: make your testers happier with Groovy, Spock and Geb (Greach 2014)Alvaro Sanchez-Mariscal
Writing functional tests using Geb in a Grails application is fine for a development team. But when you have QA automation engineers, giving them access to the Grails app might not be the best solution (specially when they belong to a different team).
So the same way DevOps allow developers and sysadmins collaborate together, let’s talk about DevQA, and make them happy using a framework stack powered by Groovy.
Besides above considerations, in this talk I will show a live example on how to setup an independent project for functional tests using Gradle, Groovy, Spock and Geb.
Rspec and Capybara Intro Tutorial at RailsConf 2013Brian Sam-Bodden
Behavior-Driven Development and Acceptance Testing are heavily intertwined and in many aspects are one and the same. Both focus on starting at the outer layers of your application by concentrating on what matter to users; behavior. In this session/workshop we'll talk about how testing can be used both for specifying your application yet to be develop expected behavior and as accurate, running documentation that can be used to validate your stakeholder's acceptance criteria. We'll talk about the different types of testing and do a few hands-on exercises to flesh out a Rails application with RSpec and Capybara.
Want to learn how to use Selenium from the ground up? This presentation will show you how to start from nothing and build out a well factored, maintainable, resilient, and parallelized set of tests that will run locally, on a Continuous Integration server, and in the cloud. These tests will not only work well, but exercise relevant functionality that matters to the business.
Test and Behaviour Driven Development (TDD/BDD)Lars Thorup
In this introduction to Test Driven Development (TDD) or Behaviour Driven Development (BDD) we give a high level description of what it is and why it is useful for developers. Then we go into some details on stubs and mocks, test data, UI testing, SQL testing, JavaScript testing, web services testing and how to start doing TDD/BDD on an existing code base.
5 Considerations When Adopting Automated TestingBhupesh Dahal
Most organizations have realized the benefits of and need for test automation—but is your investment being wisely utilized? Are you unknowingly building a test automation suite that will end up costing more than your actual product? Are you building a legacy test automation framework that may be ready to retire before you reap the benefits?
This presentation will discuss five points of consideration that will help your organization answer these questions and mitigate risks that they bring into light.
This presentation provides an overview of a Test Automation Framework with BDD and Cucumber. It also includes several open-source initiatives that Rhoynar Software Consulting (www.rhoynar.com) has been working on in the fields of QA Automation and DevOps. Lastly, it also includes links to some of the open-source projects that you can use right now for your work.
- Continuous Integration Infra a la OpenStack - https://github.com/Rhoynar/ci-infra
- An Email Verification Library in Java:
https://github.com/Rhoynar/EmailVerify
- Automatic Test Generation using Selenium WebDriver, Java and TestNG
https://github.com/Rhoynar/AutoTestR
- Barebones BDD and Cucumber Framework integrated with Java Maven and TestNG:
https://github.com/Rhoynar/qa-automation
Improving The Quality of Existing SoftwareSteven Smith
Presented at DevReach 2013.
As developers, most of our time is spent working on existing software. Sure, occasionally we get the chance to fire up a new Solution in Visual Studio, and that can be exciting. But after the first day, we find ourselves once more having to deal with the work we did yesterday. And today, we know more than we did yesterday, so there are things we’d do differently, if we had it to do over.
Over time, software rots. If we’re not diligent, our beautiful code can degrade into a worthless mess. Keeping our code in working condition is no different than changing the oil in our car – it’s preventive maintenance. In this session, Steve will look at some common places to look for signs of degradation in existing applications, and steps to take to improve the code. Examples will use C# and primarily ASP.NET.
Improving the Quality of Existing Software - DevIntersection April 2016Steven Smith
How do you improve the quality of your existing software, while continuing to add value for your customers? What are some heuristics and code smells you can look for, and principles and patterns you can use to guide you, as you make your software better over time instead of worse? How can we improve our skills and techniques so that writing high quality software becomes our default, fastest way of working?
Improving the Quality of Existing SoftwareSteven Smith
How do you improve the quality of your existing software, while continuing to add value for your customers? What are some heuristics and code smells you can look for, and principles and patterns you can use to guide you, as you make your software better over time instead of worse?
How BDD with tools like Cucumber can create a stronger team, a better quality product, and ultimately a more useable API. Given at the #apistrat SF conference 10/24/2013
Continuous Testing Meets the Classroom at Code.orgSauce Labs
Code.org's Brian Jordan, a Software Engineer, takes the audience for a fun tour of Code.org’s continuous, automated testing suite. Brian discusses how Code.org approaches testing throughout the product development cycle, given their unique testing challenges—developing interactive, game-like curriculum for just the types of browsers you’d expect to find in school computer labs—from Internet Explorer 9 to iPads across 40+ languages.
What makes up an acceptance testing framework? Especially one that will help you use Selenium successfully? How about a list of what ones currently exist? We've got you covered.
This talk was prepared for the DC Selenium Meetup in April 2013.
DevQA: make your testers happier with Groovy, Spock and Geb (Greach 2014)Alvaro Sanchez-Mariscal
Writing functional tests using Geb in a Grails application is fine for a development team. But when you have QA automation engineers, giving them access to the Grails app might not be the best solution (specially when they belong to a different team).
So the same way DevOps allow developers and sysadmins collaborate together, let’s talk about DevQA, and make them happy using a framework stack powered by Groovy.
Besides above considerations, in this talk I will show a live example on how to setup an independent project for functional tests using Gradle, Groovy, Spock and Geb.
Rspec and Capybara Intro Tutorial at RailsConf 2013Brian Sam-Bodden
Behavior-Driven Development and Acceptance Testing are heavily intertwined and in many aspects are one and the same. Both focus on starting at the outer layers of your application by concentrating on what matter to users; behavior. In this session/workshop we'll talk about how testing can be used both for specifying your application yet to be develop expected behavior and as accurate, running documentation that can be used to validate your stakeholder's acceptance criteria. We'll talk about the different types of testing and do a few hands-on exercises to flesh out a Rails application with RSpec and Capybara.
Want to learn how to use Selenium from the ground up? This presentation will show you how to start from nothing and build out a well factored, maintainable, resilient, and parallelized set of tests that will run locally, on a Continuous Integration server, and in the cloud. These tests will not only work well, but exercise relevant functionality that matters to the business.
Test and Behaviour Driven Development (TDD/BDD)Lars Thorup
In this introduction to Test Driven Development (TDD) or Behaviour Driven Development (BDD) we give a high level description of what it is and why it is useful for developers. Then we go into some details on stubs and mocks, test data, UI testing, SQL testing, JavaScript testing, web services testing and how to start doing TDD/BDD on an existing code base.
5 Considerations When Adopting Automated TestingBhupesh Dahal
Most organizations have realized the benefits of and need for test automation—but is your investment being wisely utilized? Are you unknowingly building a test automation suite that will end up costing more than your actual product? Are you building a legacy test automation framework that may be ready to retire before you reap the benefits?
This presentation will discuss five points of consideration that will help your organization answer these questions and mitigate risks that they bring into light.
This presentation provides an overview of a Test Automation Framework with BDD and Cucumber. It also includes several open-source initiatives that Rhoynar Software Consulting (www.rhoynar.com) has been working on in the fields of QA Automation and DevOps. Lastly, it also includes links to some of the open-source projects that you can use right now for your work.
- Continuous Integration Infra a la OpenStack - https://github.com/Rhoynar/ci-infra
- An Email Verification Library in Java:
https://github.com/Rhoynar/EmailVerify
- Automatic Test Generation using Selenium WebDriver, Java and TestNG
https://github.com/Rhoynar/AutoTestR
- Barebones BDD and Cucumber Framework integrated with Java Maven and TestNG:
https://github.com/Rhoynar/qa-automation
Being Epic: Best Practices for Android DevelopmentReto Meier
Turn good ideas into great apps by following some essential Android development best practices. Starting with an overview of the 5 deadly sins and 5 golden rules, you will learn how to build apps that users love and that are good citizens of the mobile environment. You will also learn specific best practices for background apps and location based services.
Как совместить, казалось бы, несовместимое: Пирамиду Тестирования Майка Коэна и ROI калькулятор? Как добиться синергетического эффекта от использования обоих инструментов. Как эффективно использовать «парочку» от pre-sales до post-release активностей? Как адаптировать полученное решение для нужд вашего конкретного проекта. Давайте попробуем ответить на эти и многие другие вопросы в процессе нашей беседы.
Mock what? What Mock?Learn What is Mocking, and how to use Mocking with ColdFusion testing, development, and continuous integration. Look at Mocking and Stubbing with a touch of Theory and a lot of Examples, including what you could test, and what you should test… and what you shouldn't test (but might be fun).
Similar to Automated Acceptance Tests in .NET (20)
In deploying apps that have been containerized, you have a lot to think about regarding what to use in production. There are a lot of things to manage, so orchestrators become a huge help. providing many services together such as scheduling, container communication, scaling, health, and more. There are major platforms to consider from Kubernetes, Swarm to ECS. In this talk we'll go through the overview of orchestrators and some of the differences between the big players. You should come out of the talk knowing where to go next in determining your orchestrator needs.
AATs can be expensive and not valuable if not done right, and doing them right is not easy. They provide enormous benefit though, and are critical as software takes over the world and manual regression testing becomes infeasible. This goes through key benefits.
ORMs have become a standard tool, though they can shield us from thinking about what's really happening with our database. It is easy and common to inadvertently have the ORM inefficiently interact with the database, if you're not aware of how key concepts should be applied in different scenarios. Some of the most common performance problems in business applications today stem from misuse of the ORM. We'll go over key concepts to be aware of, that apply to the Entity Framework and Nhibernate, such as table mapping strategies and lazy/eager loading scenarios. We'll also cover simple profiling to get at the heart of ORM performance issues on a .NET project.
2. Agenda
• Part I: AAT Overview
• Part II: Pros & Cons of AAT
• Part III: AAT Tools and Syntax in .NET
• Part IV: Overcoming the Cons of AAT
• Part V: Demo
15. Communication
• Helps specify behavior of the system in plain text
• Provides a medium for non-tech and devs to agree on
Are we
talking
about
the right
system?
20. Why NOT have ATTs?
• Business users won‟t write specifications
• Brittle and slow
• High false negatives, non-determinism
“Certainly it is not necessary to run every
example to be sure that the code works.
Probably it is necessary to run some.”
– Ron Jeffries
30. Start with a story
User story
As an internet user,
I want to search for “ALT.NET”
And get valid results
• Acceptance criteria 1: I get at least 100k results
• Acceptance criteria 2: Results ordered by relevance
…
Scenarios
Feature
31. Gherkin in a SpecFlow feature
Feature: Google Search
As an internet user
I want to search for “ALT.NET”
So that I can be knowledgeable about the organzation
Background: I am logged into my Google account
Scenario: Search for „ALT.NET‟
Given I am on the Google Home Page
When I search for ”ALT.NET"
Then I should see at least 100,000 results
And the results should be ordered by relevance
Step
32. Generate Step Definitions
[When(@"I search for ""(.+)""")]
public void WhenISearchForSomething(string searchTerm)
{
GoogleSearchResultsPage = GoogleHomePage.Search(searchTerm);
}
Step Definition
40. Minimize # of end-to-end tests
• AATs for journeys, not stories
• Is your new story entirely new?
• Balance high # of unit tests + selected end-to-end &
acceptance
41. Page Objects
public class GoogleHomepage: PageBase, IGoogleHomepage {
// Flow for this page
}
GoogleHomepage
+search(query)
+clickSearch()
+clickGettingLucky
43. public class GoogleHomepage: PageBase, IGoogleHomepage {
// Flow for this page
}
Domain
Service
UI
GoogleHomepage
+search(query)
+clickSearch()
+clickGettingLucky Data Access
GoogleHomepageService
+search(query)
+clickSearch()
+clickGettingLucky
IGoogleHomepage
+search(query)
+clickSearch()
+clickGettingLucky
44. Concise Specs
• Declarative
• Reuse steps
• Concise, many methods, fat fixtures
Bad:
• Click on Log In button
• Click username box and type „myUsername‟ and click „password‟ box and type
„myPassword‟
• Click on link for Transfer Payment
• Click box for amount and type 400
• Click „Transfer‟ button
• Assert success message
Good:
• Login as „MyAccountName‟
• Navigate to Transfer Payment page
• Transfer 400 dollars
• Assert success message
49. Developer tips
• Don't run in same assembly as other tests, so these can
be run separately since they're very slow (i.e. nightly)
• Navigation
• Haywire! Sometimes when moving step definitions
around, the gherkin gets confused and can‟t find the
target. Restart VS.
51. Gherkin
• Describe what, not how (Don‟t: „Click the Log In button‟. Do:
„Navigate to the log in screen‟)
• Specs shouldn‟t have much setup code
Given I am on the homepage
And I click login
And I enter username of „username‟ and password of „password‟
When clicking „Log In‟
Then take me to my landing page
Given I am on the homepage
When logging in with „username‟ and „password‟
Then take me to my landing page
52. Scenario Outline
Parameterize!
Scenario: eat 5 out of 12
Given there are 12 cucumbers
When I eat 5 cucumbers
Then I should have 7 cucumbers
Scenario: eat 5 out of 20
Given there are 20 cucumbers
When I eat 5 cucumbers
Then I should have 15 cucumbers
53. Scenario Outline
Parameterize!
Scenario Outline: eating
Given there are <start> cucumbers
When I eat <eat> cucumbers
Then I should have <left> cucumbers
Examples:
| start | eat | left |
| 12 | 5 | 7 |
| 20 | 5 | 15 |
54. Parameters
Given I entered the following data into the new account form:
| Field | Value |
| Name | John Galt |
| Birthdate | 2/2/1902 |
| HeightInInches | 72 |
| BankAccountBalance | 1234.56 |
or in a horizontal table like this:
| Name | Birthdate | HeightInInches | BankAccountBalance |
| John Galt | 2/2/1902 | 72 | 1234.56 |
| John Doe | 1/1/1980 | 68 | 1111.99 |
55. UI tests
• Sometimes tests will fail if the page doesn‟t have enough
time to load. Use implicit waits, and explicit when really
needed
• Capture screen shots when tests fail
• Selenium IDE
56. Anti-patterns
• Developers writing acceptance tests by themselves, for
themselves
• Tying AATs to an implementation - unit tests are
implementation specific, AATs are NOT
• Too much repetition in the gherkin
• Too many UI tests
Bad legacy code in mission critical system, not unit testable…Useful in general in a test suite for many reasons…
AcceptanceTestsFits their requirements. The story is complete. Automated
Readable by whole spectrum
What kind of tests do you run with your specs?UI – smoke, sanity, UI goo Functionality – Integration testsRun close to production; like a user would.
Mostly unit testsHARD to maintain, slow SO….
Unit test everything thru JavaScriptMost RELEVANT test cases for AATs (can’t hit all alt paths)Shouldn’t manually regression test
Whoever writes user storiesDev/Tester/BA BEFORE iteration starts
-Communication with business stakeholders, BAs, QA
-Unit tests don’t account for everything, seams-Gaps in coverage, unit tests not hitting right test cases,
-Full regression doesn’t always happen-Manual testing lapseMistakes in testing, non-testing, non-regressingAllow testers to do more exploratory and higher level things, less boring
Bugs will be caught earlier
Not doing, btw
Some have even said it’s not worth the cost
Create a spec in gherkinGerkin is a line-oriented language (line step)In SpecFlow, Generate step definitionsIn Selenium & NUnit, write UI/integration tests to power the AATs
All know definition of User Story? Should be turned into specs before iteration starts (bus & )
Given – set up the context of your testWhen – action performed by userThen - asserts