Plone Testing Tools and Techniques
An Overview
Presented By: Jordan Baker, Scryent
Plone Conference 2009, Budapest
Image CC Encyclopedia Britannica
Wednesday, October 28, 2009

Goals of this Talk
• practical and relevant usage of testing in Plone, not exhaustive
• give you the confidence to start testing, or incorporate it more fully into your
development process
• give you some hands-on examples of using testing
• provide a good review of the topic for more experienced developers
• provide some quality resources as reference
Wednesday, October 28, 2009

Wednesday, October 28, 2009

Wednesday, October 28, 2009

About Me (Jordan Baker)
• Using Plone since version 1.0
• Senior Consultant/Founder of Scryent, a Plone service provider based in
Toronto, Canada
• Over the years I have gone from writing no tests, to writing lots of tests, to
writing tests first, to getting 100% code coverage and beyond
• Quality code and enhanced developer productivity are some of my passions
Wednesday, October 28, 2009

Slides and examples
• These slides are available on Slideshare so you can follow along:
http://tinyurl.com/pttt-slides
• Also the Plone example used in this talk is available from bitbucket. To
checkout a copy and build:
$ hg clone https://hexsprite@bitbucket.org/hexsprite/pttt/
$ cd pttt
$ python2.4 bootstrap.py && bin/buildout
Wednesday, October 28, 2009

When to test?
• When writing production code. Testing reduces defects as measured by
reported bugs.
• Prototyping doesn't always require tests, as there is indeed some overhead to
writing and maintaining tests.
• Safe as long as you remember to throw away your prototypes.
Wednesday, October 28, 2009

Testing Results
• improves code quality, reduced defects
• provides more maintainable developer documentation
• creates more loosely coupled code which can offer maintenance and re-use
improvements
• “Integrating Unit Testing into a Software Development Team's Process”:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.5798&rep=rep1&type=pdf
Wednesday, October 28, 2009

Why Test
• produce more robust code faster
• tests passing is proof that you delivered what you were contracted to
produce
• acceptance tests can map directly to your software requirements
• business measurement help prevent scope creep
• finding bugs early makes them cheaper to fix
• refactor with confidence
Wednesday, October 28, 2009

Test First!
• writing tests first forces you to get clear about requirements up front
• facilitates a conversation with your product owners
• generally results in more modular and reusable code
• manual testing is slow and error-prone! The longer you put off writing tests
the less efficient you are.
Wednesday, October 28, 2009

Test Levels in the Testing Pyramid
functional,
system
integration
unit
Wednesday, October 28, 2009

Test-Code Cycle
Wednesday, October 28, 2009

More tests is not always better
• Tests are software too and must be maintained.
• Too many tests makes running your entire test suite slow.
• Remember the pyramid. Just like the food pyramid you want to focus on
having lots of tests from the lower levels which run quickly, and fewer from
the top of the pyramid.
• Some people are opportunistic testers. They will write a minimal test for the
main path of their code and then only write additional tests if regressions are
found.
Wednesday, October 28, 2009

Unit Tests
• Answer the question: Is the code doing the right thing? Written from
developer’s perspective
• Tests implementation of a sliver of behaviour
• Fast
• Doesn't rely on any external resources (so use mocks or fakes for external
resources)
• Have lots of these, for every important variation of behaviour
• Python’s unittest module - http://docs.python.org/library/unittest.html
Wednesday, October 28, 2009

When to Unit Test
• as often as possible when able to test components in isolation.
• if you depend on a few outside components, think about using a Mock object
to represent them
• if you rely on having a full Plone site configured, then you are a better off with
an integration test
Wednesday, October 28, 2009

Unit Testing Example
class TestFizzBuzz(unittest.TestCase):
"""Unit test for the FizzBuzzWriter
"""
def setUp(self):
self.fizzbuzz = FizzBuzzWriter()
def test_numbers_divisable_by_three(self):
self.failUnlessEqual(self.fizzbuzz.write_number(3), 'FIZZ')
• setUp called before each test method
• tearDown called after each test method
• methods beginning with test* are automatically added
Wednesday, October 28, 2009

Unit Testing Boilerplate
• Tests are defined in a package named tests. Typically this should be created
a subdirectory tests in your package. Don’t forget an empty __init__.py!
• Create a seperate test file for each component you are testing, eg.
test_fizzbuzz.py
• Only files starting with test_ are picked up by Zope’s test runner
• Required to find all tests and make them available to the test runner
def test_suite():
return unittest.defaultTestLoader.loadTestsFromName(__name__)
Wednesday, October 28, 2009

Test methods
• Base class unittest.TestCase defines several assertion methods to use:
• failIf(expr[, msg])
• failUnless(expr[, msg])
• failUnlessEqual(first, second[, msg])
• failUnlessRaises(exception, callable, ...)
• there’s also equivalent methods that start with assert...()
• and more, see docs
Wednesday, October 28, 2009

FizzBuzz implementation
class FizzBuzzWriter(object):
def write_number(self, number):
if number % 3 == 0 and number % 5 == 0:
return 'FIZZ BUZZ'
if number % 3 == 0:
return 'FIZZ'
if number % 5 == 0:
return 'BUZZ'
return str(number)
def write_numbers(self, count):
return ( self.write_number(number) for number in range(1, count + 1) )
Wednesday, October 28, 2009

Using Zope's testrunner
• TIP: run only the test you need to run:
$ bin/instance test -s your.module -t test_something
• By using -s you can specify the module to look for tests.
• If you want to run only one particular test method then use -t to specify which
one.
• Also to run only unit tests (as opposed to integration and functional tests) use
the -u option:
$ bin/instance test -s your.module -u
Wednesday, October 28, 2009

Running the Tests
$ bin/instance test -s pttt.fizzbuzz
Running tests at level 1
Running unit tests:
Running:
.....
Wednesday, October 28, 2009

Sample Applications of Unit Testing in Plone
• View Classes: instantiate your view class, using mock objects as necessary
for request and context
• Test validators for all corner cases, eg. phone number validator
• Simple Event handlers
Wednesday, October 28, 2009

Additional Unit Testing Resources
• official unittest docs: http://docs.python.org/library/unittest.html
• http://plone.org/documentation/tutorial/best-practices/unit-testing
• http://www.slideshare.net/Quintagroup/intro-to-testing-in-zope-plone-
presentation
• http://trizpug.org/Members/tbryan/unittest_talk
Wednesday, October 28, 2009

Integration Tests
• Plone is a complex system with many moving parts
• integration tests will be necessary to confirm that components are properly
registered and work together
• When to use
• when testing integration of various components and the flow in and out of
your code. eg. cataloging, relationships, object creation, security, workflow
• Verifying that view was registered correctly and works in context of the
Plone site
Wednesday, October 28, 2009

Integration Testing with PloneTestCase
• Builds on top of ZopeTestCase and PortalTestCase.
• All tests happen in a sandbox. Zope cleans up after you and it doesn’t affect
any of your on-disk ZODBs.
• They are by their very nature quite slow. So write them as unit tests if
possible. Essentially a full Plone site is created to run your tests in.
Wednesday, October 28, 2009

Plone Integration Test Example
• Because users love to run FizzBuzz so often they want to generate a Page of
Fizz Buzz results.
• Let’s introduce a new view (using five.grok) which when given an appropriate
number and a document id will generate a page with the results.
Wednesday, October 28, 2009

Writing an Integration Test
class FizzBuzzIntegration(PloneTestCase):
layer = ptc_layer
def test_fizzbuzz_contents(self):
view = self.folder.restrictedTraverse('@@fizzbuzz')
view.generate_document(5)
# document with id fizzbuzz exists
self.failUnless('fizzbuzz' in self.folder)
text = self.folder.fizzbuzz.getText()
# verify contents are correct (Document converted to HTML for us)
self.failUnlessEqual(text, '<p>1<br />2<br />FIZZ<br />4<br />BUZZ</p>')
def test_fizzbuzz_anonymous_cant_generate(self):
self.logout()
view = self.folder.restrictedTraverse('@@fizzbuzz')
self.assertRaises(Unauthorized, view.generate_document, 5)
Wednesday, October 28, 2009

Quick review of PloneTestCase’s API
• self.loginAsPortalOwner()
• .login(user)
• .logout()
• .setRoles([roles])
• .portal
• .folder
• PloneTestCase.default_user
Wednesday, October 28, 2009

collective.testcaselayer and PloneTestCase
• collective.testcaselayer, developed by Ross Patterson, provides layer base
classes for Zope, CMF and Plone development.
• Layers allow expensive setup of fixtures, such as a Plone site, to be shared
by many tests.
• Simplifies repetitive Plone testing boilerplate code while providing a clean
way to extend.
• Also of interest is the ability to create ZODB sandboxes for each layer so that
sibling layers are isolated from each other.
Wednesday, October 28, 2009

Installing collective.testcaselayer using testing
extras
• When to use: when adding dependencies that are only required for testing
• examples: collective.testcaselayer, mocker library, zc.testbrowser
• dependencies are registered as setuptools extras so they can be optionally
included
• you should have a separate development buildout configuration that specifies
the extras
Wednesday, October 28, 2009

Adding collective.testcaselayer
• Use the extra_requires functionality of setuptools so that production usage
won’t require collective.testcaselayer
•
from setuptools import setup, find_packages
...
tests_require = ['collective.testcaselayer']
...
setup(name='pttt.fizzbuzz',
...
install_requires=[
'setuptools',
# -*- Extra requirements: -*-
],
tests_require=tests_require,
extras_require={'tests': tests_require},
...
entry_points="""
Wednesday, October 28, 2009

Enabling tests extras in your buildout
• Find the eggs section in your buildout part and add:
...
eggs +=
pttt.fizzbuzz [tests]
...
Wednesday, October 28, 2009

Autoinclude your ZCML
• Plone 3.3 includes support via z3c.autoinclude to automatically load ZCML in
dependent packages using a setuptools entry point.
• For Plone 3.2 you have to add z3c.autoinclude to your buildout and
something like this somewhere in your buildout ZCML:
<includePlugins package="plone" file="configure.zcml" />
• Here’s how we’d register ours in our packages’ setup.py:
entry_points="""
[z3c.autoinclude.plugin]
target = plone
“"",
Wednesday, October 28, 2009

Writing an Integration Test
class FizzBuzzIntegration(PloneTestCase):
layer = ptc_layer
def test_fizzbuzz_contents(self):
view = self.folder.restrictedTraverse('@@fizzbuzz')
view.generate_document(5)
# document with id fizzbuzz exists
self.failUnless('fizzbuzz' in self.folder)
text = self.folder.fizzbuzz.getText()
# verify contents are correct (Document converted to HTML for us)
self.failUnlessEqual(text, '<p>1<br />2<br />FIZZ<br />4<br />BUZZ</p>')
def test_fizzbuzz_anonymous_cant_generate(self):
self.logout()
view = self.folder.restrictedTraverse('@@fizzbuzz')
self.assertRaises(Unauthorized, view.generate_document, 5)
Wednesday, October 28, 2009

Generating the Fizzbuzz document
class FizzBuzz(grok.View):
grok.context(ATFolder)
grok.require('zope2.View')
def generate_document(self, count):
writer = FizzBuzzWriter()
text = 'n'.join(writer.write_numbers(count))
self.context.invokeFactory('Document', id='fizzbuzz', title="Fizzbuzz
results", text=text)
Wednesday, October 28, 2009

Running the test
$ bin/instance test -s pttt.fizzbuzz -t test_fizzbuzz_contents -t
test_fizzbuzz_anonymous_cant_generate
Wednesday, October 28, 2009

Make it faster with roadrunner
• Roadrunner is a tool that preloads Plone to make running your integration
tests much much faster. http://pypi.python.org/pypi/roadrunner
• Due to interactions with z3c.autoinclude it will currently only reload your tests
without repeating the Plone site setup.
• Its easy to use too. Just add a roadrunner part to your buildout.cfg like so:
[roadrunner]
recipe = roadrunner:plone
packages-under-test = pttt.*
• Roadrunner accepts the same command line arguments as Zope testrunner.
So we can run our tests like this:
$ bin/roadrunner -s pttt.fizzbuzz
Wednesday, October 28, 2009

Plone Integration Testing Resources
• Programmatically Manipulate Plone: http://plonemanual.twinapex.fi/content/
index.html
• PloneTestCase Tutorial - PloneConf 2004: http://www.zope.org/Members/
shh/Tutorial
• http://plone.org/documentation/tutorial/testing/integration-doctests-using-
plonetestcase
• PloneTestCase: http://plone.org/products/plonetestcase
Wednesday, October 28, 2009

Doctests
• tests that look like interactive Python sessions
• When to use
• API Documentation, verify examples work
• its a matter of taste, some people prefer unittests especially when there
are a lot of variations
• they are especially good for functional tests where the prose explains the
acceptance test and the code verifies that it is working
Wednesday, October 28, 2009

Doctest boilerplate
import unittest
import doctest
from Testing import ZopeTestCase as ztc
from example.tests.tests import base
def test_suite():
"""This sets up a test suite that actually runs the tests in the class
above
"""
return unittest.TestSuite([
ztc.ZopeDocFileSuite(
'tests/functional.txt', package='example.tests',
test_class=base.ExampleFunctionalTestCase,
optionflags=doctest.REPORT_ONLY_FIRST_FAILURE |
doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS),
])
Wednesday, October 28, 2009

A Simple Doctest
First we'll import and instantiate our FizzBuzzWriter class
>>> from pttt.fizzbuzz.fizzbuzz import FizzBuzzWriter
>>> writer = FizzBuzzWriter()
Let's write out single numbers
>>> writer.write_number(1)
'1'
>>> writer.write_number(3)
'FIZZ'
>>> writer.write_number(15)
'FIZZ BUZZ'
Finally let's write a sequence of numbers
>>> list(writer.write_numbers(15))
['1', '2', 'FIZZ', '4', 'BUZZ', 'FIZZ', '7', '8', 'FIZZ', 'BUZZ',
'11', 'FIZZ', '13', '14', 'FIZZ BUZZ']
Wednesday, October 28, 2009

Functional and System Tests
• End to end tests
• Typically black-box tests that act as an end-user would and are written from
end user’s point of view. eg. click the button labeled “Login” vs. call the login
method
• inherently brittle as the UI often changes
• Use zope.testbrowser, twill, Selenium
• typically written as doctests but don’t need to be
Wednesday, October 28, 2009

Functional Tests
• When to use:
• blackbox testing
• automated acceptance testing
• Smoke or face-saving tests - exercise most important paths of business
value
• When not to use:
• testing every corner case. use unit tests for that
Wednesday, October 28, 2009

About zope.testbrowser
• testbrowser implements a Python scriptable browser that simulates
accessing your Plone site.
• Full docs for zope.testbrowser: http://pypi.python.org/pypi/zope.testbrowser
• Note that using the testbrowser in Plone we should use the Five version
which wraps it for use in a Zope 2 environment:
from Products.Five.testbrowser import Browser
• Not Javascript aware. If you need to test DHTML, Javascript, Ajax et al then
you will need to use something like Selenium.
Wednesday, October 28, 2009

Using zope.testbrowser
• Don’t hammer Plone.org ;)
from zope.testbrowser.browser import Browser
# this version of testbrowser works with external resources
# for testing your own apps using internal Browser class:
# from Products.Five.testbrowser import Browser
browser = Browser('http://plone.org')
browser.getControl(name='SearchableText').value = 'Testing'
browser.getForm(id='search').submit()
assert browser.isHtml
assert 'Untested Code is Broken Code' in browser.contents
Wednesday, October 28, 2009

Plone Functional testbrowser API basics
• Open your test Plone site: browser.open('http://nohost/plone/')
• Click a link: browser.getLink(link_text).click()
• Click the first link when multiple have the same name:
browser.getLink(link_text, index=0).click()
• Get URL: browser.url
• Reload: browser.reload()
• Change form value: browser.getControl(input_name).value = ‘Whatever’
• Submit form: browser.getForm(id)
Wednesday, October 28, 2009

Two ways to login
• One is to use Plone’s login mechanism, arguably more black-box since it
should simulate what your users will actually do.
>>> browser.open(portal_url)
>>> browser.getControl(name='__ac_name').value = portal_owner
>>> browser.getControl(name='__ac_password').value =
default_password
>>> browser.getControl(name='submit').click()
• Alternatively you could add an auth header:
>>> browser.addHeader('Authorization', 'Basic root:secret')
Wednesday, October 28, 2009

Functional Testing Example
• Let’s take a look at an existing functional testing example from Plone’s source
code from plone/app/workflow/tests/sharingpage.txt:
• A couple things about this
• browser.handleErrors = False means you’ll get proper tracebacks
instead of cryptic 500 Errors
• demonstrates pure black box testing, no assumptions of internal state
Wednesday, October 28, 2009

Functional Testing Resources
• http://plone.org/documentation/tutorial/testing/functional-tests
• Face Saving Tests: http://gojko.net/2007/09/25/effective-user-interface-
testing
Wednesday, October 28, 2009

System Tests using twill
• You can write system tests using testbrowser as well
• Twill provides a simple domain specific language (DSL) for performing web
tests. It also provides a Python API.
• When to use
• when doing whole system testing that includes caching proxy, multiple
web servers, backend and database
• testing live systems
• deployment testing
Wednesday, October 28, 2009

Installing twill in your buildout
• twill is on pypi, simply use zc.recipe.egg to install its scripts:
[twill]
recipe = zc.recipe.egg
eggs =
twill
• Also make sure to add the part to your [buildout] parts section.
Wednesday, October 28, 2009

Simple Twill Script
go http://plone.org
formvalue searchform SearchableText testing
submit
# assert that we found the following text
find 'Untested Code is Broken Code'
# assert that we didn't find this text:
notfind 'Testing sucks'
Wednesday, October 28, 2009

Running twill interactively
$ bin/twill-sh
-= Welcome to twill! =-
current page: *empty page*
>> go http://plone.org
==> at http://plone.org
current page: http://plone.org
>> formvalue searchform SearchableText testing
current page: http://plone.org
>> submit
current page: http://plone.org/search?SearchableText=testing
>>
current page: http://plone.org/search?SearchableText=testing
>> # assert that we found the following text
current page: http://plone.org/search?SearchableText=testing
>> find 'Untested Code is Broken Code'
current page: http://plone.org/search?SearchableText=testing
>>
current page: http://plone.org/search?SearchableText=testing
>> # assert that we didn't find this text:
current page: http://plone.org/search?SearchableText=testing
>> notfind 'Testing sucks'
current page: http://plone.org/search?SearchableText=testing
Wednesday, October 28, 2009

Writing a System Test
• Verify your SSL configuration:
# TEST: join form redirects to https
go http://mysite.com
follow "Join"
# test that the resulting URL is https://
url "^https:"
url "/join_form$"
find "Registration Form"
# fill in registration form and register
fv 2 fullname "Test User"
fv 2 email "test@tester.com"
submit
find "You have been registered."
fv 2 form.submitted 1
submit
find "You are now logged in"
# home page is http
url "^http://"
Wednesday, October 28, 2009

Mock and Fake objects
• When to use:
• unittest code that has external dependencies such as databases or
network connections
• depends on external components
Wednesday, October 28, 2009

Basic Mocks
• Let’s say you have a test method that expects an object to be provided which
provides an absolute_url method in order to return a calculated URL:
• Without using an Integration test, how could you do this? Simply create a
simple Python class that implements only the methods you need.
• This can get tiresome if you have to create many of these mocks, or if you
want to make some assertions about what is called on those mocks with
what arguments.
Wednesday, October 28, 2009

Basic Mocking Example
def calc_url(context):
return context.absolute_url() + "/calculated_url"
class MockDocument:
def absolute_url(self):
return "http://plone/some_document"
def test_calc_url(self):
mock = MockDocument()
self.failUnlessEqual(calc_url(mock), "http://plone/some_document/calculated_url")
Wednesday, October 28, 2009

Mocker
• Python library for creating Mock objects and managing expectations we have
for those Mock objects.
• Other more advanced features as well such as proxies, patching and proxy
replacement.
• Unittest integration
• Martin Aspeli took this further and created plone.mocktestcase which
provides some helper methods that are specific to Zope and Plone
development.
http://pypi.python.org/pypi/plone.mocktestcase
Wednesday, October 28, 2009

A Mocking Example
• Writing an interface to an external network service
• Using the Component Architecture to create an abstract interface for a
remote network service and using MockTestCase’s mock_utility to make this
available to application code under test:
In the
network_service = self.mocker.mock()
self.mock_utility(self.video_provider, INetworkService)
Wednesday, October 28, 2009

Mocking Resources
• http://www.martinaspeli.net/articles/mock-testing-with-mocker-and-
plone.mocktestcase
Wednesday, October 28, 2009

Coverage
• answers the question: are my tests covering all the code in my application?
• for production code I aim for 100%. Why? anything less than 100% can hide
defects.
• 100% code coverage is considered minimum standard *
• does NOT guarantee the code has no bugs
• * - http://www.plope.com/Members/chrism/a_hundred_percent
Wednesday, October 28, 2009

Coverage won’t catch this
• This pseudo-code example would give 100% code coverage. What if I
passed 0 to myfunc?
def test_myfunc():
myfunc(100)
def myfunc(n):
return 100 / n
Wednesday, October 28, 2009

Generating coverage report
• Zope’s testrunner comes with a builtin coverage report tool, invoke like this:
$ bin/instance test -s pttt.fizzbuzz --coverage=./coverage
• When calculating coverage you should know that code runs much slower. So
grab a coffee, tea or water and catch up on your xkcd.
• The results are in:
Ran 5 tests with 0 failures and 0 errors in 0.005 seconds.
Tearing down left over layers:
Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
lines cov% module (path)
16 100% pttt.fizzbuzz.fizzbuzz (/Users/jbb/co/pttt/src/pttt.fizzbuzz/
pttt/fizzbuzz/fizzbuzz.py)
Wednesday, October 28, 2009

HTML Coverage report using z3c.coverage
• Produces nice looking reports that you can discuss with your team:
Wednesday, October 28, 2009

HTML Coverage report using z3c.coverage
• Add this part to buildout:
[z3c-coverage]
recipe = zc.recipe.egg
eggs = z3c.coverage
extra-paths = ${zope2:location}/lib/python
• Don’t forget to add z3c-coverage to [buildout] parts =
• Generate reports:
$ bin/coverage ./coverage ./coverage/reports
Wednesday, October 28, 2009

Coverage Resources
• http://plone.org/documentation/tutorial/testing/code-coverage
Wednesday, October 28, 2009

Pdb-fu (debugging)
• Getting test failures from Zope Testrunner? Use -D to drop you into Pdb using
post-mortem debugging
• Use pdb’s jump command to re-run certain lines of codes. Watch out for
side-effects.
• Special variable __return__ is set. Use “continue” to go to the end of the
method and then you can easily examine the return result.
• Breakpoint variables. Looping through 1000 items, set a breakpoint with an
expression.
• Missing source code? This could help, add to your buildout.cfg:
[buildout]
unzip = true
Wednesday, October 28, 2009

General Testing Tips
• Write at the lowest level possible. This will ensure that your test will run as
quickly as possible which becomes important when you have hundreds or
thousands of tests.
• Take your time, write out what you’re trying to do in your commends first
before writing actual code:
def test_whatever(self):
# verify document was created
# ensure that notification was sent
Wednesday, October 28, 2009

General Testing Tips
• Broken tests that lie around for a long time because they are minor issues
that no one can be bothered to fix can mask larger bugs due to too much
noise and not enough signal. 0 failing tests (and no cheating!)
• Hard to test something? Step back, try to break it up into easier to test
pieces.
Wednesday, October 28, 2009

Some Further Resources
• Plone Testing tutorial: http://plone.org/documentation/tutorial/testing/
• plone.reload: http://pypi.python.org/pypi/plone.reload
• http://diveintopython.org/unit_testing/index.html
• example tests of all sorts, mostly still valid: http://svn.plone.org/svn/
collective/examples/example.tests/trunk/
Wednesday, October 28, 2009

Open Spaces and Sprinting
• For all of those who want to have open discussions about Testing in Plone
we’ll have an Open Space on Friday.
• Also I’ll be sprinting on bringing roadrunner up to spec with Plone 4, Dexterity
and Zope 2.12. Interested? Drop me a line.
Wednesday, October 28, 2009

Thanks
• Your Feedback is appreciated: jbb@scryent.com
• Find me on twitter: @hexsprite
• Good luck and happy testing!
Wednesday, October 28, 2009

Thanks
• Photos used licensed under Creative Commons 2.0:
• http://www.flickr.com/photos/rupertuk/2102548824/
• http://www.flickr.com/photos/kacey/2626513873
Wednesday, October 28, 2009