Software teams mostly find themselves working with three broad categories of tests - unit, integration and functional (excluding technology verification test categories like performance, load, stress etc.). Unit tests indicate whether the code is doing things right. Functional tests are complementary to - but quite different from unit tests. Functional tests tell whether the completed application is working correctly and providing the proper functionality. Simply put, unit tests are written from the code developer's perspective, while functional tests are written from the end user's perspective. When they work reliably, functional tests give users, stakeholders and developers confidence that the software meets agreed upon requirements. In reality, a lot of teams find themselves grappling with perennially failing, hard to understand, slow running tests, which take herculean efforts to maintain while inspiring low confidence in the reliability of the end product. This article examines recipes on how to create and maintain a smoothly running suite of functional/acceptance tests that can be reliably used to verify that the software is ready for release.

1. Functional
Testing Pa0erns
Premanand Chandrasekaran
ThoughtWorks, Inc.
premanand@thoughtworks.com

2. Types of Tests
Business Facing
Support Programming
Functional Tests
Showcases
Acceptance Tests
Exploratory Tests
Critique Product
Usability Tests
Unit Tests
Performance Tests
Component Tests
Stress Tests
System Tests
Security Tests
Technology Facing
h0p://www.exampler.com/old-­‐‑blog/2003/08/21/#agile-­‐‑testing-­‐‑project-­‐‑1
© 2012 ThoughtWorks, Inc. 2

3. Functional Tests -­‐‑ The Pitch
• What Are Functional Tests?
o Verification that business requirements are
met
o Black Box
o Automated
• Why Functional Tests?
o Maintain (high) external quality
o Allow team to be bolder
o Allow team to go faster
© 2012 ThoughtWorks, Inc. 3

4. Functional Tests – The Reality
• Flaky
• Take long time to run
• Cumbersome to maintain
• Don’t catch many bugs
• Some things are hard to automate
• Don’t quite give you the feedback
you were hoping for
© 2012 ThoughtWorks, Inc. 4

5. Consequences
• Low confidence in automation
• Over-reliance on manual testing
• Slipped Deadlines AND/OR
• Lesser Testing AND/OR
• Testing in Production :-)
© 2012 ThoughtWorks, Inc. 5

6. How To Get Be0er?
Technology
Process
People
© 2012 ThoughtWorks, Inc. 6

7. Process
© 2012 ThoughtWorks, Inc. 7

8. Functional Test – Example
© 2012 ThoughtWorks, Inc. 8

9. Example -­‐‑ Specification
© 2012 ThoughtWorks, Inc. 9

10. Example – Implementation
© 2012 ThoughtWorks, Inc. 10

11. Implementation Problems?
• No correlation between specification
and implementation
• Intent lost in translation
• Easy to write, hard to read, maintain
• Too monolithic, lacks abstraction
• Causes non-technical domain experts
to tune out of the process
© 2012 ThoughtWorks, Inc. 11

12. Executable Specifications
Specification Step
Tight Correlation
Implementation Step
© 2012 ThoughtWorks, Inc. 12

13. Functional Test -­‐‑ Specification
© 2012 ThoughtWorks, Inc. 13

14. More useful information
in failure reports
© 2012 ThoughtWorks, Inc. 14

15. Acceptance Criteria DSL
• Given some initial context
• When an event occurs
• Then ensure outcome
• Example –
© 2012 ThoughtWorks, Inc. 15

16. Functional Test – New Implementation
© 2012 ThoughtWorks, Inc. 16

17. Good Acceptance Criteria
Scenario: Advanced Search for a hat!
!
Given I am searching on "http://www.etsy.com"!
When I click on the select box with css class "select.handmade"!
And I select "Knitting"!
And I click on the text box with id "#search_query"!
And type in "hat"!
Then there are search results!
Focus on WHAT, not HOW
Scenario: Advanced Search for a hat!
!
Given I am searching on Etsy.com!
And I am not logged in!
And I specify the Knitting sub category!
And I search for hat! Be@er
Then there are search results!
!
© 2012 ThoughtWorks, Inc. 17

18. How Many Tests?
Manual Testing
10%
Functional
Tests
Integration Tests
20%
Unit Tests
70%
h0p://blog.mountaingoatsoftware.com/tag/agile-­‐‑testing
© 2012 ThoughtWorks, Inc. 18

19. What About Legacy Systems?
• Start with functional tests
• Cover with unit tests for new
functionality
• Be on the lookout to replace
functional tests more fine-grained tests
at every given opportunity
© 2012 ThoughtWorks, Inc. 19

20. What To Automate?
• The MOST important scenarios
o End-To-End system level scenarios
o Happy paths
• Avoid automating individual feature-
level acceptance criteria
• Make a distinction between
“ephemeral” and “long-standing”
tests
© 2012 ThoughtWorks, Inc. 20

21. When do these tests run?
• After every check-in
• As part of CI build pipeline
o Unit à Smoke à Regression à Deploy
• Incubator Test Suites
© 2012 ThoughtWorks, Inc. 21

22. Workflow
• Sprint [N -1]
o Feature elaboration with entry level acceptance criteria
• Sprint [N]
o Feature kick-off meeting
o Developers implement feature
o Testers (optionally) add additional acceptance criteria
• Sprint[N, N + 1]
o Testers verify all acceptance criteria to be satisfied
o Testers refactor functional test suite
o Feature marked complete
© 2012 ThoughtWorks, Inc. 22

23. Technology
© 2012 ThoughtWorks, Inc. 23

24. Test Characteristics
• Be idempotent
o At least re-runnable
o Keep tests isolated from one another
• Be runnable in parallel
o No inter-dependencies between tests
• Be deterministic
o Quarantine/eradicate non-deterministic tests
o Don’t “sleep”
• Prefer callbacks or at least poll for asynchronous responses
o Consider mocking remote third-party services
• Have separate tests to verify interaction with remote services
o Consider using relative dates/times or mock the system clock
o http://martinfowler.com/articles/nonDeterminism.html
© 2012 ThoughtWorks, Inc. 24

25. Test Sources
• Don’t have any dependencies on
production sources
• Treat as first-class citizens
o Keep DRY
o Run static analysis metrics
• Leverage design patterns like
o Page Objects
o Compound Steps
• Consider the use of terser languages
o Groovy, Ruby etc.
© 2012 ThoughtWorks, Inc. 25

26. Test Data
• Use application to create test data
• Use test data creation steps
o Frameworks like DBUnit
• Use anonymized production data subsets
o Requires lot of time and effort
• Use anonymized production data copies
o Huge data volumes may make it prohibitive
• Avoid the shared database anti-pattern
• Consider mocking remote third-party
services
© 2012 ThoughtWorks, Inc. 26

27. Test Infrastructure
• Invest in fast build hardware
o CI servers support multiple remote agents
• Use same software configuration as
production in all environments
• Consider running tests on the cloud
o http://www.saucelabs.com
© 2012 ThoughtWorks, Inc. 27

28. People
© 2012 ThoughtWorks, Inc. 28

29. Who Writes/Owns The Tests?
• The specifications – acceptance
criteria?
o Domain Experts
• Step implementations?
o Developers & Testers
• The functional suite?
o The Team
o Day-to-Day – Testers
© 2012 ThoughtWorks, Inc. 29

30. Team Dynamics
• Fix broken builds and tests immediately
o OK to occasionally break the build
o Not OK to leave the build broken for long
• Do not allow check-ins over a broken build
o Unless check-in is being made to fix the build
• Co-locate teams as much as possible
o At least create fully formed remote teams
© 2012 ThoughtWorks, Inc. 30

31. Questions?
premanand@thoughtworks.com
Thank You!
© 2012 ThoughtWorks, Inc. 31