Functional testing patterns

  • 1,862 views
Uploaded on

Software teams mostly find themselves working with three broad categories of tests - unit, integration and functional (excluding technology verification test categories like performance, load, stress …

Software teams mostly find themselves working with three broad categories of tests - unit, integration and functional (excluding technology verification test categories like performance, load, stress etc.). Unit tests indicate whether the code is doing things right. Functional tests are complementary to - but quite different from unit tests. Functional tests tell whether the completed application is working correctly and providing the proper functionality. Simply put, unit tests are written from the code developer's perspective, while functional tests are written from the end user's perspective. When they work reliably, functional tests give users, stakeholders and developers confidence that the software meets agreed upon requirements.

In reality, a lot of teams find themselves grappling with perennially failing, hard to understand, slow running tests, which take herculean efforts to maintain while inspiring low confidence in the reliability of the end product.

This article examines recipes on how to create and maintain a smoothly running suite of functional/acceptance tests that can be reliably used to verify that the software is ready for release.

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Nice Presentation!
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
1,862
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
22
Comments
1
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Functional  Testing  Pa0erns Premanand Chandrasekaran ThoughtWorks, Inc. premanand@thoughtworks.com
  • 2. Types  of  Tests Business  Facing Support  Programming Functional  Tests Showcases Acceptance  Tests Exploratory  Tests Critique  Product Usability  Tests Unit  Tests Performance  Tests Component  Tests Stress  Tests System  Tests Security  Tests Technology  Facing h0p://www.exampler.com/old-­‐‑blog/2003/08/21/#agile-­‐‑testing-­‐‑project-­‐‑1 © 2012 ThoughtWorks, Inc. 2
  • 3. Functional  Tests  -­‐‑  The  Pitch •  What Are Functional Tests? o Verification that business requirements are met o Black Box o Automated•  Why Functional Tests? o Maintain (high) external quality o Allow team to be bolder o Allow team to go faster© 2012 ThoughtWorks, Inc. 3
  • 4. Functional  Tests  –  The  Reality •  Flaky•  Take long time to run•  Cumbersome to maintain•  Don’t catch many bugs•  Some things are hard to automate•  Don’t quite give you the feedback you were hoping for© 2012 ThoughtWorks, Inc. 4
  • 5. Consequences •  Low confidence in automation•  Over-reliance on manual testing•  Slipped Deadlines AND/OR•  Lesser Testing AND/OR•  Testing in Production :-)© 2012 ThoughtWorks, Inc. 5
  • 6. How  To  Get  Be0er? Technology Process People © 2012 ThoughtWorks, Inc. 6
  • 7. Process © 2012 ThoughtWorks, Inc. 7
  • 8. Functional  Test  –  Example © 2012 ThoughtWorks, Inc. 8
  • 9. Example  -­‐‑  Specification © 2012 ThoughtWorks, Inc. 9
  • 10. Example  –  Implementation © 2012 ThoughtWorks, Inc. 10
  • 11. Implementation  Problems? •  No correlation between specification and implementation•  Intent lost in translation•  Easy to write, hard to read, maintain•  Too monolithic, lacks abstraction•  Causes non-technical domain experts to tune out of the process© 2012 ThoughtWorks, Inc. 11
  • 12. Executable  Specifications Specification  Step                                                       Tight  Correlation Implementation  Step                                     © 2012 ThoughtWorks, Inc. 12
  • 13. Functional  Test  -­‐‑  Specification © 2012 ThoughtWorks, Inc. 13
  • 14. More  useful  information in  failure  reports © 2012 ThoughtWorks, Inc. 14
  • 15. Acceptance  Criteria  DSL •  Given some initial context•  When an event occurs•  Then ensure outcome•  Example – © 2012 ThoughtWorks, Inc. 15
  • 16. Functional  Test  –  New  Implementation © 2012 ThoughtWorks, Inc. 16
  • 17. Good  Acceptance  Criteria Scenario: Advanced Search for a hat!!Given I am searching on "http://www.etsy.com"!When I click on the select box with css class "select.handmade"!And I select "Knitting"!And I click on the text box with id "#search_query"!And type in "hat"!Then there are search results! Focus  on  WHAT,  not  HOW Scenario: Advanced Search for a hat! ! Given I am searching on Etsy.com! And I am not logged in! And I specify the Knitting sub category! And I search for hat! Be@er Then there are search results! ! © 2012 ThoughtWorks, Inc. 17
  • 18. How  Many  Tests? Manual  Testing 10% Functional Tests Integration  Tests 20% Unit  Tests 70% h0p://blog.mountaingoatsoftware.com/tag/agile-­‐‑testing © 2012 ThoughtWorks, Inc. 18
  • 19. What  About  Legacy  Systems? •  Start with functional tests•  Cover with unit tests for new functionality•  Be on the lookout to replace functional tests more fine-grained tests at every given opportunity© 2012 ThoughtWorks, Inc. 19
  • 20. What  To  Automate? •  The MOST important scenarios o  End-To-End system level scenarios o  Happy paths•  Avoid automating individual feature- level acceptance criteria•  Make a distinction between “ephemeral” and “long-standing” tests© 2012 ThoughtWorks, Inc. 20
  • 21. When  do  these  tests  run? •  After every check-in•  As part of CI build pipeline o Unit à Smoke à Regression à Deploy•  Incubator Test Suites© 2012 ThoughtWorks, Inc. 21
  • 22. Workflow •  Sprint [N -1] o  Feature elaboration with entry level acceptance criteria•  Sprint [N] o  Feature kick-off meeting o  Developers implement feature o  Testers (optionally) add additional acceptance criteria•  Sprint[N, N + 1] o  Testers verify all acceptance criteria to be satisfied o  Testers refactor functional test suite o  Feature marked complete © 2012 ThoughtWorks, Inc. 22
  • 23. Technology © 2012 ThoughtWorks, Inc. 23
  • 24. Test  Characteristics •  Be idempotent o  At least re-runnable o  Keep tests isolated from one another•  Be runnable in parallel o  No inter-dependencies between tests•  Be deterministic o  Quarantine/eradicate non-deterministic tests o  Don’t “sleep” •  Prefer callbacks or at least poll for asynchronous responses o  Consider mocking remote third-party services •  Have separate tests to verify interaction with remote services o  Consider using relative dates/times or mock the system clock o  http://martinfowler.com/articles/nonDeterminism.html © 2012 ThoughtWorks, Inc. 24
  • 25. Test  Sources •  Don’t have any dependencies on production sources•  Treat as first-class citizens o  Keep DRY o  Run static analysis metrics•  Leverage design patterns like o  Page Objects o  Compound Steps•  Consider the use of terser languages o  Groovy, Ruby etc.© 2012 ThoughtWorks, Inc. 25
  • 26. Test  Data •  Use application to create test data•  Use test data creation steps o  Frameworks like DBUnit•  Use anonymized production data subsets o  Requires lot of time and effort•  Use anonymized production data copies o  Huge data volumes may make it prohibitive•  Avoid the shared database anti-pattern•  Consider mocking remote third-party services© 2012 ThoughtWorks, Inc. 26
  • 27. Test  Infrastructure •  Invest in fast build hardware o  CI servers support multiple remote agents•  Use same software configuration as production in all environments•  Consider running tests on the cloud o  http://www.saucelabs.com© 2012 ThoughtWorks, Inc. 27
  • 28. People © 2012 ThoughtWorks, Inc. 28
  • 29. Who  Writes/Owns  The  Tests? •  The specifications – acceptance criteria? o  Domain Experts•  Step implementations? o  Developers & Testers•  The functional suite? o  The Team o  Day-to-Day – Testers© 2012 ThoughtWorks, Inc. 29
  • 30. Team  Dynamics •  Fix broken builds and tests immediately o  OK to occasionally break the build o  Not OK to leave the build broken for long•  Do not allow check-ins over a broken build o  Unless check-in is being made to fix the build•  Co-locate teams as much as possible o  At least create fully formed remote teams© 2012 ThoughtWorks, Inc. 30
  • 31. Questions? premanand@thoughtworks.com Thank You!© 2012 ThoughtWorks, Inc. 31