Automating test scenarios in software

1. Requirements Delivery –
Using Examples

2. About Us
Bharti Rupani
@bhartir24
brupani@enterprise-knowledge.com
https://www.linkedin.com/in/bhartir
Wyn Van Devanter
@wynv
wyn.vandevanter@excella.com
https://www.linkedin.com/in/wyntuition
http://www.slideshare.net/wynvandevanter

3. Terminology
• Specification by Example
• Executable Specifications
• Automated Acceptance Tests (AAT)
• Behavior Driven Development (BDD)
• Acceptance Test Driven Development
(ATDD)
• Testable Requirements

4. “Specifications
by examples
are acceptance
tests”
- Gajko Adzic

5. Testable Requirements
• User Journeys
• Deliver business value
• Clear examples
• Executable

6. Why should you care?
• Clearly communicate the specifications
• No functional gaps
• Verify business rules
• Build the right thing
6

7. Building Quality
Plan Collaborate Deliver
Build the
right thing
Build the thing
right

8. Developing Examples
Collaboration
Prioritized
Backlog
Business Goals User
Journey
User Story
2
Acceptance
Criteria 1
Example 1
User Story
1
Acceptance
Criteria 1
Example 1
Acceptance
Criteria 2
Example 1

9. Collaborate How do I
code this
feature?What are
the details
of this
feature?
How do I
validate
that this
work is
done?

10. Three Amigos
• Business Analyst, Developers and Testers
• Collaborate
• Share knowledge
• Think about the desired outcome
• Shared understanding

11. User Journey
You are a Business Analyst that is looking for a
conference to attend.

12. Example of using an Example
Attend BA
Conference
View Results
Region Date
View
DCBADD
Details
View
information
about IIBADC
View speaker
profiles
View Cost
Select
DCBADD
Register via
Eventbrite
Determine
Discounts

13. User Story
As a business analyst, I want to review the various
IIBA discounts available so that I can pay a
reduced ticket price for DCBADD.

14. What is Gherkin?
• Business readable, domain specific language
• Connects human concept of cause and effect to software
concept of input/process/output
• Can be interpreted by automation tools to drive
acceptance tests

15. Gherkin Syntax
Source: Jeffrey Davidson

16. Scenarios
Scenario: Standard Rate
Given I am purchasing a
ticket to DCBADD
And I am not an IIBA
member
When I view my ticket price
Then I see the full rate of
$200
Scenario: IIBA Rate
Given I am purchasing a
ticket to DCBADD
And I am an IIBA member
When I view my ticket price
Then I see the discounted
rate of $150

17. Example
Scenario Outline: Determine DCBADD ticket price
Given I am purchasing a ticket to DCBADD
And I am a “<membership type>”
When I view my ticket price
Then I see the discounted “<ticket price>”
Examples:
membership price
nonmember $200
IIBA national $150
IIBA DC $125

18. Example
Using tables as arguments as input to a Given or as expected output
from a Then:
Scenario: Validate Order Total
Given I am a business analyst
And I am purchasing multiple tickets with varying “<membership type>”
to DCBADD:
When I view my total
Then I see my “<total>”
ticket number & type price
2 nonmember $400
1 IIBA national $150

19. Guidelines
• Be precise
• Make sure it’s testable
• Focus on business functionality
• Avoid specific details
• Use active voice, not passive
• Have a single action trigger the expected behavior
(WHEN condition)
• Use parameters for concise expression of examples (i.e.
scenario outlines)

20. Automating the scenarios

21. Automated Acceptance Tests
(AATs)
• Code automates them
• They run constantly

22. Why Have AATs? (Pros)

23. Communication
• Helps specify behavior of the system in plain text
• Provides a medium for non-tech and devs to agree on
Are we
talking
about
the right
system?

24. Seams, unit test mistakes

25. Automation
“There’s no place for
human beings to be
doing regression testing
manually.”
-Jez Humble

26. Speedier deployments

27. Save resources

28. More testing during development

29. Why NOT Have AATs?
(Cons)

30. Why don’t more people use them?

31. Why NOT have AATs?
• Maintenance
• Speed
• High false negatives, non-determinism

32. Impediments
• Poor adoption
• Bandwidth/Velocity
• Learning Curve/Experience
• Business users won’t write specifications
• Developers won’t automate them
• Brittle if recorded

33. Solutions

34. Acceptance Test Strategy
• Happy paths
• Major unhappy paths
• Legacy
• Regression

35. Minimize # of end-to-end tests
• AATs for journeys, not stories
• Is your new story entirely new?
• Balance high # of unit tests + selected end-to-end &
acceptance

36. Gherkin
Specs shouldn’t have much setup code
Given my friend ‘Bob’ typed ‘I’m excited’ into the text box
And clicks the Post button on my timeline
When I press the Like link on the post
Then the screen should say ‘Like’
And should include my name as liking it
Given my friend ‘Bob’ has posted ‘I’m excited’
And I am on my timeline
When I like on the post
Then the post should be liked by me

37. General Tips
• Create different levels of suites depending on depth/level
of feedback desired:
• Smoke, Current iteration/sprint, Regression
• Run AATs as close to the real environment as possible

38. When Acceptances Tests catches a
bug
• See why bug got through unit/integration tests
• Add unit, integration tests
• Prune AATs?

39. UI tests
• Sometimes tests will fail if the page doesn’t have enough
time to load. Use implicit waits, and explicit when really
needed
• Capture screen shots when tests fail
• Selenium IDE

40. AAT Workflow

41. AAT workflow
• Start with a scenario(s)
Given my friend ‘Bob’ has posted ‘I’m excited’
And I am on my timeline
When I like on the post
Then the post should be liked by me

42. What can you write them in?
Vim
Sublime
Text

43. Tracking AATs

44. Tracking AATs

45. Conclusion
• Worth the investment if done properly
• Very expensive and wasteful if not
• Requires collaboration between analysts, testers,
developers
• Developers automate the gherkin
• Focus on journey and scenarios as opposed to stories

46. Resources:
Books:
• Continuous Delivery, Jez Humble, David Farley
• Growing Object-Oriented Software, Guided By Tests, Steve
Freeman, Nat Pryce
• Specification by Example, Gajko Adzic
Articles:
• Automated Acceptance Tests,
http://www.thoughtworks.com/insights/articles/automated-
acceptance-tests
• Using SpecFlow with the Page Object,
http://blogs.lessthandot.com/index.php/EnterpriseDev/application-
lifecycle-management/using-specflow-to
• Maintaining Automated Acceptance Tests (ThoughtWorks),
http://www.youtube.com/watch?v=uf0EVbH5hdA
• Creating Maintainable Automated Acceptance Test Suites, Jez
Humble,
• http://www.youtube.com/watch?v=v-L_2y6g5DI

47. Thanks!
Bharti Rupani
@bhartir24
brupani@enterprise-knowledge.com
https://www.linkedin.com/in/bhartir
Wyn Van Devanter
@wynv
wyn.vandevanter@excella.com
https://www.linkedin.com/in/wyntuition
http://www.slideshare.net/wynvandevanter