Plone Testing Tools And Techniques


Published on

Plone Conferrence 2009 talk, an overview of Testing Tools and Techniques with Plone

Published in: Technology, Education
1 Comment
  • I always forget about handleErrors. Nice to have reference like this for some important details
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Plone Testing Tools And Techniques

  1. 1. Plone Testing Tools and Techniques An Overview Presented By: Jordan Baker, Scryent Plone Conference 2009, Budapest Image CC Encyclopedia Britannica Wednesday, October 28, 2009
  2. 2. Goals of this Talk • practical and relevant usage of testing in Plone, not exhaustive • give you the confidence to start testing, or incorporate it more fully into your development process • give you some hands-on examples of using testing • provide a good review of the topic for more experienced developers • provide some quality resources as reference Wednesday, October 28, 2009
  3. 3. Wednesday, October 28, 2009
  4. 4. Wednesday, October 28, 2009
  5. 5. About Me (Jordan Baker) • Using Plone since version 1.0 • Senior Consultant/Founder of Scryent, a Plone service provider based in Toronto, Canada • Over the years I have gone from writing no tests, to writing lots of tests, to writing tests first, to getting 100% code coverage and beyond • Quality code and enhanced developer productivity are some of my passions Wednesday, October 28, 2009
  6. 6. Slides and examples • These slides are available on Slideshare so you can follow along: • Also the Plone example used in this talk is available from bitbucket. To checkout a copy and build: $ hg clone $ cd pttt $ python2.4 && bin/buildout Wednesday, October 28, 2009
  7. 7. When to test? • When writing production code. Testing reduces defects as measured by reported bugs. • Prototyping doesn't always require tests, as there is indeed some overhead to writing and maintaining tests. • Safe as long as you remember to throw away your prototypes. Wednesday, October 28, 2009
  8. 8. Testing Results • improves code quality, reduced defects • provides more maintainable developer documentation • creates more loosely coupled code which can offer maintenance and re-use improvements • “Integrating Unit Testing into a Software Development Team's Process”: Wednesday, October 28, 2009
  9. 9. Why Test • produce more robust code faster • tests passing is proof that you delivered what you were contracted to produce • acceptance tests can map directly to your software requirements • business measurement help prevent scope creep • finding bugs early makes them cheaper to fix • refactor with confidence Wednesday, October 28, 2009
  10. 10. Test First! • writing tests first forces you to get clear about requirements up front • facilitates a conversation with your product owners • generally results in more modular and reusable code • manual testing is slow and error-prone! The longer you put off writing tests the less efficient you are. Wednesday, October 28, 2009
  11. 11. Test Levels in the Testing Pyramid functional, system integration unit Wednesday, October 28, 2009
  12. 12. Test-Code Cycle Wednesday, October 28, 2009
  13. 13. More tests is not always better • Tests are software too and must be maintained. • Too many tests makes running your entire test suite slow. • Remember the pyramid. Just like the food pyramid you want to focus on having lots of tests from the lower levels which run quickly, and fewer from the top of the pyramid. • Some people are opportunistic testers. They will write a minimal test for the main path of their code and then only write additional tests if regressions are found. Wednesday, October 28, 2009
  14. 14. Unit Tests • Answer the question: Is the code doing the right thing? Written from developer’s perspective • Tests implementation of a sliver of behaviour • Fast • Doesn't rely on any external resources (so use mocks or fakes for external resources) • Have lots of these, for every important variation of behaviour • Python’s unittest module - Wednesday, October 28, 2009
  15. 15. When to Unit Test • as often as possible when able to test components in isolation. • if you depend on a few outside components, think about using a Mock object to represent them • if you rely on having a full Plone site configured, then you are a better off with an integration test Wednesday, October 28, 2009
  16. 16. Unit Testing Example class TestFizzBuzz(unittest.TestCase): """Unit test for the FizzBuzzWriter """ def setUp(self): self.fizzbuzz = FizzBuzzWriter() def test_numbers_divisable_by_three(self): self.failUnlessEqual(self.fizzbuzz.write_number(3), 'FIZZ') • setUp called before each test method • tearDown called after each test method • methods beginning with test* are automatically added Wednesday, October 28, 2009
  17. 17. Unit Testing Boilerplate • Tests are defined in a package named tests. Typically this should be created a subdirectory tests in your package. Don’t forget an empty! • Create a seperate test file for each component you are testing, eg. • Only files starting with test_ are picked up by Zope’s test runner • Required to find all tests and make them available to the test runner def test_suite(): return unittest.defaultTestLoader.loadTestsFromName(__name__) Wednesday, October 28, 2009
  18. 18. Test methods • Base class unittest.TestCase defines several assertion methods to use: • failIf(expr[, msg]) • failUnless(expr[, msg]) • failUnlessEqual(first, second[, msg]) • failUnlessRaises(exception, callable, ...) • there’s also equivalent methods that start with assert...() • and more, see docs Wednesday, October 28, 2009
  19. 19. FizzBuzz implementation class FizzBuzzWriter(object): def write_number(self, number): if number % 3 == 0 and number % 5 == 0: return 'FIZZ BUZZ' if number % 3 == 0: return 'FIZZ' if number % 5 == 0: return 'BUZZ' return str(number) def write_numbers(self, count): return ( self.write_number(number) for number in range(1, count + 1) ) Wednesday, October 28, 2009
  20. 20. Using Zope's testrunner • TIP: run only the test you need to run: $ bin/instance test -s your.module -t test_something • By using -s you can specify the module to look for tests. • If you want to run only one particular test method then use -t to specify which one. • Also to run only unit tests (as opposed to integration and functional tests) use the -u option: $ bin/instance test -s your.module -u Wednesday, October 28, 2009
  21. 21. Running the Tests $ bin/instance test -s pttt.fizzbuzz Running tests at level 1 Running unit tests: Running: ..... Wednesday, October 28, 2009
  22. 22. Sample Applications of Unit Testing in Plone • View Classes: instantiate your view class, using mock objects as necessary for request and context • Test validators for all corner cases, eg. phone number validator • Simple Event handlers Wednesday, October 28, 2009
  23. 23. Additional Unit Testing Resources • official unittest docs: • • presentation • Wednesday, October 28, 2009
  24. 24. Integration Tests • Plone is a complex system with many moving parts • integration tests will be necessary to confirm that components are properly registered and work together • When to use • when testing integration of various components and the flow in and out of your code. eg. cataloging, relationships, object creation, security, workflow • Verifying that view was registered correctly and works in context of the Plone site Wednesday, October 28, 2009
  25. 25. Integration Testing with PloneTestCase • Builds on top of ZopeTestCase and PortalTestCase. • All tests happen in a sandbox. Zope cleans up after you and it doesn’t affect any of your on-disk ZODBs. • They are by their very nature quite slow. So write them as unit tests if possible. Essentially a full Plone site is created to run your tests in. Wednesday, October 28, 2009
  26. 26. Plone Integration Test Example • Because users love to run FizzBuzz so often they want to generate a Page of Fizz Buzz results. • Let’s introduce a new view (using five.grok) which when given an appropriate number and a document id will generate a page with the results. Wednesday, October 28, 2009
  27. 27. Writing an Integration Test class FizzBuzzIntegration(PloneTestCase): layer = ptc_layer def test_fizzbuzz_contents(self): view = self.folder.restrictedTraverse('@@fizzbuzz') view.generate_document(5) # document with id fizzbuzz exists self.failUnless('fizzbuzz' in self.folder) text = self.folder.fizzbuzz.getText() # verify contents are correct (Document converted to HTML for us) self.failUnlessEqual(text, '<p>1<br />2<br />FIZZ<br />4<br />BUZZ</p>') def test_fizzbuzz_anonymous_cant_generate(self): self.logout() view = self.folder.restrictedTraverse('@@fizzbuzz') self.assertRaises(Unauthorized, view.generate_document, 5) Wednesday, October 28, 2009
  28. 28. Quick review of PloneTestCase’s API • self.loginAsPortalOwner() • .login(user) • .logout() • .setRoles([roles]) • .portal • .folder • PloneTestCase.default_user Wednesday, October 28, 2009
  29. 29. collective.testcaselayer and PloneTestCase • collective.testcaselayer, developed by Ross Patterson, provides layer base classes for Zope, CMF and Plone development. • Layers allow expensive setup of fixtures, such as a Plone site, to be shared by many tests. • Simplifies repetitive Plone testing boilerplate code while providing a clean way to extend. • Also of interest is the ability to create ZODB sandboxes for each layer so that sibling layers are isolated from each other. Wednesday, October 28, 2009
  30. 30. Installing collective.testcaselayer using testing extras • When to use: when adding dependencies that are only required for testing • examples: collective.testcaselayer, mocker library, zc.testbrowser • dependencies are registered as setuptools extras so they can be optionally included • you should have a separate development buildout configuration that specifies the extras Wednesday, October 28, 2009
  31. 31. Adding collective.testcaselayer • Use the extra_requires functionality of setuptools so that production usage won’t require collective.testcaselayer • from setuptools import setup, find_packages ... tests_require = ['collective.testcaselayer'] ... setup(name='pttt.fizzbuzz', ... install_requires=[ 'setuptools', # -*- Extra requirements: -*- ], tests_require=tests_require, extras_require={'tests': tests_require}, ... entry_points=""" Wednesday, October 28, 2009
  32. 32. Enabling tests extras in your buildout • Find the eggs section in your buildout part and add: ... eggs += pttt.fizzbuzz [tests] ... Wednesday, October 28, 2009
  33. 33. Autoinclude your ZCML • Plone 3.3 includes support via z3c.autoinclude to automatically load ZCML in dependent packages using a setuptools entry point. • For Plone 3.2 you have to add z3c.autoinclude to your buildout and something like this somewhere in your buildout ZCML: <includePlugins package="plone" file="configure.zcml" /> • Here’s how we’d register ours in our packages’ entry_points=""" [z3c.autoinclude.plugin] target = plone “"", Wednesday, October 28, 2009
  34. 34. Writing an Integration Test class FizzBuzzIntegration(PloneTestCase): layer = ptc_layer def test_fizzbuzz_contents(self): view = self.folder.restrictedTraverse('@@fizzbuzz') view.generate_document(5) # document with id fizzbuzz exists self.failUnless('fizzbuzz' in self.folder) text = self.folder.fizzbuzz.getText() # verify contents are correct (Document converted to HTML for us) self.failUnlessEqual(text, '<p>1<br />2<br />FIZZ<br />4<br />BUZZ</p>') def test_fizzbuzz_anonymous_cant_generate(self): self.logout() view = self.folder.restrictedTraverse('@@fizzbuzz') self.assertRaises(Unauthorized, view.generate_document, 5) Wednesday, October 28, 2009
  35. 35. Generating the Fizzbuzz document class FizzBuzz(grok.View): grok.context(ATFolder) grok.require('zope2.View') def generate_document(self, count): writer = FizzBuzzWriter() text = 'n'.join(writer.write_numbers(count)) self.context.invokeFactory('Document', id='fizzbuzz', title="Fizzbuzz results", text=text) Wednesday, October 28, 2009
  36. 36. Running the test $ bin/instance test -s pttt.fizzbuzz -t test_fizzbuzz_contents -t test_fizzbuzz_anonymous_cant_generate Wednesday, October 28, 2009
  37. 37. Make it faster with roadrunner • Roadrunner is a tool that preloads Plone to make running your integration tests much much faster. • Due to interactions with z3c.autoinclude it will currently only reload your tests without repeating the Plone site setup. • Its easy to use too. Just add a roadrunner part to your buildout.cfg like so: [roadrunner] recipe = roadrunner:plone packages-under-test = pttt.* • Roadrunner accepts the same command line arguments as Zope testrunner. So we can run our tests like this: $ bin/roadrunner -s pttt.fizzbuzz Wednesday, October 28, 2009
  38. 38. Plone Integration Testing Resources • Programmatically Manipulate Plone: index.html • PloneTestCase Tutorial - PloneConf 2004: shh/Tutorial • plonetestcase • PloneTestCase: Wednesday, October 28, 2009
  39. 39. Doctests • tests that look like interactive Python sessions • When to use • API Documentation, verify examples work • its a matter of taste, some people prefer unittests especially when there are a lot of variations • they are especially good for functional tests where the prose explains the acceptance test and the code verifies that it is working Wednesday, October 28, 2009
  40. 40. Doctest boilerplate import unittest import doctest from Testing import ZopeTestCase as ztc from example.tests.tests import base def test_suite(): """This sets up a test suite that actually runs the tests in the class above """ return unittest.TestSuite([ ztc.ZopeDocFileSuite( 'tests/functional.txt', package='example.tests', test_class=base.ExampleFunctionalTestCase, optionflags=doctest.REPORT_ONLY_FIRST_FAILURE | doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS), ]) Wednesday, October 28, 2009
  41. 41. A Simple Doctest First we'll import and instantiate our FizzBuzzWriter class >>> from pttt.fizzbuzz.fizzbuzz import FizzBuzzWriter >>> writer = FizzBuzzWriter() Let's write out single numbers >>> writer.write_number(1) '1' >>> writer.write_number(3) 'FIZZ' >>> writer.write_number(15) 'FIZZ BUZZ' Finally let's write a sequence of numbers >>> list(writer.write_numbers(15)) ['1', '2', 'FIZZ', '4', 'BUZZ', 'FIZZ', '7', '8', 'FIZZ', 'BUZZ', '11', 'FIZZ', '13', '14', 'FIZZ BUZZ'] Wednesday, October 28, 2009
  42. 42. Functional and System Tests • End to end tests • Typically black-box tests that act as an end-user would and are written from end user’s point of view. eg. click the button labeled “Login” vs. call the login method • inherently brittle as the UI often changes • Use zope.testbrowser, twill, Selenium • typically written as doctests but don’t need to be Wednesday, October 28, 2009
  43. 43. Functional Tests • When to use: • blackbox testing • automated acceptance testing • Smoke or face-saving tests - exercise most important paths of business value • When not to use: • testing every corner case. use unit tests for that Wednesday, October 28, 2009
  44. 44. About zope.testbrowser • testbrowser implements a Python scriptable browser that simulates accessing your Plone site. • Full docs for zope.testbrowser: • Note that using the testbrowser in Plone we should use the Five version which wraps it for use in a Zope 2 environment: from Products.Five.testbrowser import Browser • Not Javascript aware. If you need to test DHTML, Javascript, Ajax et al then you will need to use something like Selenium. Wednesday, October 28, 2009
  45. 45. Using zope.testbrowser • Don’t hammer ;) from zope.testbrowser.browser import Browser # this version of testbrowser works with external resources # for testing your own apps using internal Browser class: # from Products.Five.testbrowser import Browser browser = Browser('') browser.getControl(name='SearchableText').value = 'Testing' browser.getForm(id='search').submit() assert browser.isHtml assert 'Untested Code is Broken Code' in browser.contents Wednesday, October 28, 2009
  46. 46. Plone Functional testbrowser API basics • Open your test Plone site:'http://nohost/plone/') • Click a link: browser.getLink(link_text).click() • Click the first link when multiple have the same name: browser.getLink(link_text, index=0).click() • Get URL: browser.url • Reload: browser.reload() • Change form value: browser.getControl(input_name).value = ‘Whatever’ • Submit form: browser.getForm(id) Wednesday, October 28, 2009
  47. 47. Two ways to login • One is to use Plone’s login mechanism, arguably more black-box since it should simulate what your users will actually do. >>> >>> browser.getControl(name='__ac_name').value = portal_owner >>> browser.getControl(name='__ac_password').value = default_password >>> browser.getControl(name='submit').click() • Alternatively you could add an auth header: >>> browser.addHeader('Authorization', 'Basic root:secret') Wednesday, October 28, 2009
  48. 48. Functional Testing Example • Let’s take a look at an existing functional testing example from Plone’s source code from plone/app/workflow/tests/sharingpage.txt: • A couple things about this • browser.handleErrors = False means you’ll get proper tracebacks instead of cryptic 500 Errors • demonstrates pure black box testing, no assumptions of internal state Wednesday, October 28, 2009
  49. 49. Functional Testing Resources • • Face Saving Tests: testing Wednesday, October 28, 2009
  50. 50. System Tests using twill • You can write system tests using testbrowser as well • Twill provides a simple domain specific language (DSL) for performing web tests. It also provides a Python API. • When to use • when doing whole system testing that includes caching proxy, multiple web servers, backend and database • testing live systems • deployment testing Wednesday, October 28, 2009
  51. 51. Installing twill in your buildout • twill is on pypi, simply use zc.recipe.egg to install its scripts: [twill] recipe = zc.recipe.egg eggs = twill • Also make sure to add the part to your [buildout] parts section. Wednesday, October 28, 2009
  52. 52. Simple Twill Script go formvalue searchform SearchableText testing submit # assert that we found the following text find 'Untested Code is Broken Code' # assert that we didn't find this text: notfind 'Testing sucks' Wednesday, October 28, 2009
  53. 53. Running twill interactively $ bin/twill-sh -= Welcome to twill! =- current page: *empty page* >> go ==> at current page: >> formvalue searchform SearchableText testing current page: >> submit current page: >> current page: >> # assert that we found the following text current page: >> find 'Untested Code is Broken Code' current page: >> current page: >> # assert that we didn't find this text: current page: >> notfind 'Testing sucks' current page: Wednesday, October 28, 2009
  54. 54. Writing a System Test • Verify your SSL configuration: # TEST: join form redirects to https go follow "Join" # test that the resulting URL is https:// url "^https:" url "/join_form$" find "Registration Form" # fill in registration form and register fv 2 fullname "Test User" fv 2 email "" submit find "You have been registered." fv 2 form.submitted 1 submit find "You are now logged in" # home page is http url "^http://" Wednesday, October 28, 2009
  55. 55. Mock and Fake objects • When to use: • unittest code that has external dependencies such as databases or network connections • depends on external components Wednesday, October 28, 2009
  56. 56. Basic Mocks • Let’s say you have a test method that expects an object to be provided which provides an absolute_url method in order to return a calculated URL: • Without using an Integration test, how could you do this? Simply create a simple Python class that implements only the methods you need. • This can get tiresome if you have to create many of these mocks, or if you want to make some assertions about what is called on those mocks with what arguments. Wednesday, October 28, 2009
  57. 57. Basic Mocking Example def calc_url(context): return context.absolute_url() + "/calculated_url" class MockDocument: def absolute_url(self): return "http://plone/some_document" def test_calc_url(self): mock = MockDocument() self.failUnlessEqual(calc_url(mock), "http://plone/some_document/calculated_url") Wednesday, October 28, 2009
  58. 58. Mocker • Python library for creating Mock objects and managing expectations we have for those Mock objects. • Other more advanced features as well such as proxies, patching and proxy replacement. • Unittest integration • Martin Aspeli took this further and created plone.mocktestcase which provides some helper methods that are specific to Zope and Plone development. Wednesday, October 28, 2009
  59. 59. A Mocking Example • Writing an interface to an external network service • Using the Component Architecture to create an abstract interface for a remote network service and using MockTestCase’s mock_utility to make this available to application code under test: In the network_service = self.mocker.mock() self.mock_utility(self.video_provider, INetworkService) Wednesday, October 28, 2009
  60. 60. Mocking Resources • plone.mocktestcase Wednesday, October 28, 2009
  61. 61. Coverage • answers the question: are my tests covering all the code in my application? • for production code I aim for 100%. Why? anything less than 100% can hide defects. • 100% code coverage is considered minimum standard * • does NOT guarantee the code has no bugs • * - Wednesday, October 28, 2009
  62. 62. Coverage won’t catch this • This pseudo-code example would give 100% code coverage. What if I passed 0 to myfunc? def test_myfunc(): myfunc(100) def myfunc(n): return 100 / n Wednesday, October 28, 2009
  63. 63. Generating coverage report • Zope’s testrunner comes with a builtin coverage report tool, invoke like this: $ bin/instance test -s pttt.fizzbuzz --coverage=./coverage • When calculating coverage you should know that code runs much slower. So grab a coffee, tea or water and catch up on your xkcd. • The results are in: Ran 5 tests with 0 failures and 0 errors in 0.005 seconds. Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds. lines cov% module (path) 16 100% pttt.fizzbuzz.fizzbuzz (/Users/jbb/co/pttt/src/pttt.fizzbuzz/ pttt/fizzbuzz/ Wednesday, October 28, 2009
  64. 64. HTML Coverage report using z3c.coverage • Produces nice looking reports that you can discuss with your team: Wednesday, October 28, 2009
  65. 65. HTML Coverage report using z3c.coverage • Add this part to buildout: [z3c-coverage] recipe = zc.recipe.egg eggs = z3c.coverage extra-paths = ${zope2:location}/lib/python • Don’t forget to add z3c-coverage to [buildout] parts = • Generate reports: $ bin/coverage ./coverage ./coverage/reports Wednesday, October 28, 2009
  66. 66. Coverage Resources • Wednesday, October 28, 2009
  67. 67. Pdb-fu (debugging) • Getting test failures from Zope Testrunner? Use -D to drop you into Pdb using post-mortem debugging • Use pdb’s jump command to re-run certain lines of codes. Watch out for side-effects. • Special variable __return__ is set. Use “continue” to go to the end of the method and then you can easily examine the return result. • Breakpoint variables. Looping through 1000 items, set a breakpoint with an expression. • Missing source code? This could help, add to your buildout.cfg: [buildout] unzip = true Wednesday, October 28, 2009
  68. 68. General Testing Tips • Write at the lowest level possible. This will ensure that your test will run as quickly as possible which becomes important when you have hundreds or thousands of tests. • Take your time, write out what you’re trying to do in your commends first before writing actual code: def test_whatever(self): # verify document was created # ensure that notification was sent Wednesday, October 28, 2009
  69. 69. General Testing Tips • Broken tests that lie around for a long time because they are minor issues that no one can be bothered to fix can mask larger bugs due to too much noise and not enough signal. 0 failing tests (and no cheating!) • Hard to test something? Step back, try to break it up into easier to test pieces. Wednesday, October 28, 2009
  70. 70. Some Further Resources • Plone Testing tutorial: • plone.reload: • • example tests of all sorts, mostly still valid: collective/examples/example.tests/trunk/ Wednesday, October 28, 2009
  71. 71. Open Spaces and Sprinting • For all of those who want to have open discussions about Testing in Plone we’ll have an Open Space on Friday. • Also I’ll be sprinting on bringing roadrunner up to spec with Plone 4, Dexterity and Zope 2.12. Interested? Drop me a line. Wednesday, October 28, 2009
  72. 72. Thanks • Your Feedback is appreciated: • Find me on twitter: @hexsprite • Good luck and happy testing! Wednesday, October 28, 2009
  73. 73. Thanks • Photos used licensed under Creative Commons 2.0: • • Wednesday, October 28, 2009