Plone Testing Tools And Techniques

Plone Testing Tools and Techniques An Overview Presented By: Jordan Baker, Scryent Plone Conference 2009, Budapest Image CC Encyclopedia Britannica Wednesday, October 28, 2009

Goals of this Talk • practical and relevant usage of testing in Plone, not exhaustive • give you the conﬁdence to start testing, or incorporate it more fully into your development process • give you some hands-on examples of using testing • provide a good review of the topic for more experienced developers • provide some quality resources as reference Wednesday, October 28, 2009

Wednesday, October 28, 2009

About Me (Jordan Baker) • Using Plone since version 1.0 • Senior Consultant/Founder of Scryent, a Plone service provider based in Toronto, Canada • Over the years I have gone from writing no tests, to writing lots of tests, to writing tests ﬁrst, to getting 100% code coverage and beyond • Quality code and enhanced developer productivity are some of my passions Wednesday, October 28, 2009

Slides and examples • These slides are available on Slideshare so you can follow along: http://tinyurl.com/pttt-slides • Also the Plone example used in this talk is available from bitbucket. To checkout a copy and build: $ hg clone https://hexsprite@bitbucket.org/hexsprite/pttt/ $ cd pttt $ python2.4 bootstrap.py && bin/buildout Wednesday, October 28, 2009

When to test? • When writing production code. Testing reduces defects as measured by reported bugs. • Prototyping doesn't always require tests, as there is indeed some overhead to writing and maintaining tests. • Safe as long as you remember to throw away your prototypes. Wednesday, October 28, 2009

Testing Results • improves code quality, reduced defects • provides more maintainable developer documentation • creates more loosely coupled code which can offer maintenance and re-use improvements • “Integrating Unit Testing into a Software Development Team's Process”: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.5798&rep=rep1&type=pdf Wednesday, October 28, 2009

Why Test • produce more robust code faster • tests passing is proof that you delivered what you were contracted to produce • acceptance tests can map directly to your software requirements • business measurement help prevent scope creep • finding bugs early makes them cheaper to fix • refactor with confidence Wednesday, October 28, 2009

Test First! • writing tests ﬁrst forces you to get clear about requirements up front • facilitates a conversation with your product owners • generally results in more modular and reusable code • manual testing is slow and error-prone! The longer you put off writing tests the less efﬁcient you are. Wednesday, October 28, 2009

Test Levels in the Testing Pyramid functional, system integration unit Wednesday, October 28, 2009

Test-Code Cycle Wednesday, October 28, 2009

More tests is not always better • Tests are software too and must be maintained. • Too many tests makes running your entire test suite slow. • Remember the pyramid. Just like the food pyramid you want to focus on having lots of tests from the lower levels which run quickly, and fewer from the top of the pyramid. • Some people are opportunistic testers. They will write a minimal test for the main path of their code and then only write additional tests if regressions are found. Wednesday, October 28, 2009

Unit Tests • Answer the question: Is the code doing the right thing? Written from developer’s perspective • Tests implementation of a sliver of behaviour • Fast • Doesn't rely on any external resources (so use mocks or fakes for external resources) • Have lots of these, for every important variation of behaviour • Python’s unittest module - http://docs.python.org/library/unittest.html Wednesday, October 28, 2009

When to Unit Test • as often as possible when able to test components in isolation. • if you depend on a few outside components, think about using a Mock object to represent them • if you rely on having a full Plone site conﬁgured, then you are a better off with an integration test Wednesday, October 28, 2009

Unit Testing Example class TestFizzBuzz(unittest.TestCase): """Unit test for the FizzBuzzWriter """ def setUp(self): self.fizzbuzz = FizzBuzzWriter() def test_numbers_divisable_by_three(self): self.failUnlessEqual(self.fizzbuzz.write_number(3), 'FIZZ') • setUp called before each test method • tearDown called after each test method • methods beginning with test* are automatically added Wednesday, October 28, 2009

Unit Testing Boilerplate • Tests are defined in a package named tests. Typically this should be created a subdirectory tests in your package. Don’t forget an empty __init__.py! • Create a seperate test file for each component you are testing, eg. test_fizzbuzz.py • Only files starting with test_ are picked up by Zope’s test runner • Required to find all tests and make them available to the test runner def test_suite(): return unittest.defaultTestLoader.loadTestsFromName(__name__) Wednesday, October 28, 2009

Test methods • Base class unittest.TestCase deﬁnes several assertion methods to use: • failIf(expr[, msg]) • failUnless(expr[, msg]) • failUnlessEqual(ﬁrst, second[, msg]) • failUnlessRaises(exception, callable, ...) • there’s also equivalent methods that start with assert...() • and more, see docs Wednesday, October 28, 2009

FizzBuzz implementation class FizzBuzzWriter(object): def write_number(self, number): if number % 3 == 0 and number % 5 == 0: return 'FIZZ BUZZ' if number % 3 == 0: return 'FIZZ' if number % 5 == 0: return 'BUZZ' return str(number) def write_numbers(self, count): return ( self.write_number(number) for number in range(1, count + 1) ) Wednesday, October 28, 2009

Using Zope's testrunner • TIP: run only the test you need to run: $ bin/instance test -s your.module -t test_something • By using -s you can specify the module to look for tests. • If you want to run only one particular test method then use -t to specify which one. • Also to run only unit tests (as opposed to integration and functional tests) use the -u option: $ bin/instance test -s your.module -u Wednesday, October 28, 2009

Running the Tests $ bin/instance test -s pttt.fizzbuzz Running tests at level 1 Running unit tests: Running: ..... Wednesday, October 28, 2009

Sample Applications of Unit Testing in Plone • View Classes: instantiate your view class, using mock objects as necessary for request and context • Test validators for all corner cases, eg. phone number validator • Simple Event handlers Wednesday, October 28, 2009

Additional Unit Testing Resources • ofﬁcial unittest docs: http://docs.python.org/library/unittest.html • http://plone.org/documentation/tutorial/best-practices/unit-testing • http://www.slideshare.net/Quintagroup/intro-to-testing-in-zope-plone- presentation • http://trizpug.org/Members/tbryan/unittest_talk Wednesday, October 28, 2009

Integration Tests • Plone is a complex system with many moving parts • integration tests will be necessary to confirm that components are properly registered and work together • When to use • when testing integration of various components and the flow in and out of your code. eg. cataloging, relationships, object creation, security, workflow • Verifying that view was registered correctly and works in context of the Plone site Wednesday, October 28, 2009

Integration Testing with PloneTestCase • Builds on top of ZopeTestCase and PortalTestCase. • All tests happen in a sandbox. Zope cleans up after you and it doesn’t affect any of your on-disk ZODBs. • They are by their very nature quite slow. So write them as unit tests if possible. Essentially a full Plone site is created to run your tests in. Wednesday, October 28, 2009

Plone Integration Test Example • Because users love to run FizzBuzz so often they want to generate a Page of Fizz Buzz results. • Let’s introduce a new view (using ﬁve.grok) which when given an appropriate number and a document id will generate a page with the results. Wednesday, October 28, 2009

Writing an Integration Test class FizzBuzzIntegration(PloneTestCase): layer = ptc_layer def test_fizzbuzz_contents(self): view = self.folder.restrictedTraverse('@@fizzbuzz') view.generate_document(5) # document with id fizzbuzz exists self.failUnless('fizzbuzz' in self.folder) text = self.folder.fizzbuzz.getText() # verify contents are correct (Document converted to HTML for us) self.failUnlessEqual(text, '1 2 FIZZ 4 BUZZ') def test_fizzbuzz_anonymous_cant_generate(self): self.logout() view = self.folder.restrictedTraverse('@@fizzbuzz') self.assertRaises(Unauthorized, view.generate_document, 5) Wednesday, October 28, 2009

Quick review of PloneTestCase’s API • self.loginAsPortalOwner() • .login(user) • .logout() • .setRoles([roles]) • .portal • .folder • PloneTestCase.default_user Wednesday, October 28, 2009

collective.testcaselayer and PloneTestCase • collective.testcaselayer, developed by Ross Patterson, provides layer base classes for Zope, CMF and Plone development. • Layers allow expensive setup of ﬁxtures, such as a Plone site, to be shared by many tests. • Simpliﬁes repetitive Plone testing boilerplate code while providing a clean way to extend. • Also of interest is the ability to create ZODB sandboxes for each layer so that sibling layers are isolated from each other. Wednesday, October 28, 2009

Installing collective.testcaselayer using testing extras • When to use: when adding dependencies that are only required for testing • examples: collective.testcaselayer, mocker library, zc.testbrowser • dependencies are registered as setuptools extras so they can be optionally included • you should have a separate development buildout conﬁguration that speciﬁes the extras Wednesday, October 28, 2009

Adding collective.testcaselayer • Use the extra_requires functionality of setuptools so that production usage won’t require collective.testcaselayer • from setuptools import setup, find_packages ... tests_require = ['collective.testcaselayer'] ... setup(name='pttt.fizzbuzz', ... install_requires=[ 'setuptools', # -*- Extra requirements: -*- ], tests_require=tests_require, extras_require={'tests': tests_require}, ... entry_points=""" Wednesday, October 28, 2009

Enabling tests extras in your buildout • Find the eggs section in your buildout part and add: ... eggs += pttt.fizzbuzz [tests] ... Wednesday, October 28, 2009

Autoinclude your ZCML • Plone 3.3 includes support via z3c.autoinclude to automatically load ZCML in dependent packages using a setuptools entry point. • For Plone 3.2 you have to add z3c.autoinclude to your buildout and something like this somewhere in your buildout ZCML: <includePlugins package="plone" file="configure.zcml" /> • Here’s how we’d register ours in our packages’ setup.py: entry_points=""" [z3c.autoinclude.plugin] target = plone “"", Wednesday, October 28, 2009

Writing an Integration Test class FizzBuzzIntegration(PloneTestCase): layer = ptc_layer def test_fizzbuzz_contents(self): view = self.folder.restrictedTraverse('@@fizzbuzz') view.generate_document(5) # document with id fizzbuzz exists self.failUnless('fizzbuzz' in self.folder) text = self.folder.fizzbuzz.getText() # verify contents are correct (Document converted to HTML for us) self.failUnlessEqual(text, '1 2 FIZZ 4 BUZZ') def test_fizzbuzz_anonymous_cant_generate(self): self.logout() view = self.folder.restrictedTraverse('@@fizzbuzz') self.assertRaises(Unauthorized, view.generate_document, 5) Wednesday, October 28, 2009

Generating the Fizzbuzz document class FizzBuzz(grok.View): grok.context(ATFolder) grok.require('zope2.View') def generate_document(self, count): writer = FizzBuzzWriter() text = 'n'.join(writer.write_numbers(count)) self.context.invokeFactory('Document', id='fizzbuzz', title="Fizzbuzz results", text=text) Wednesday, October 28, 2009

Running the test $ bin/instance test -s pttt.fizzbuzz -t test_fizzbuzz_contents -t test_fizzbuzz_anonymous_cant_generate Wednesday, October 28, 2009

Make it faster with roadrunner • Roadrunner is a tool that preloads Plone to make running your integration tests much much faster. http://pypi.python.org/pypi/roadrunner • Due to interactions with z3c.autoinclude it will currently only reload your tests without repeating the Plone site setup. • Its easy to use too. Just add a roadrunner part to your buildout.cfg like so: [roadrunner] recipe = roadrunner:plone packages-under-test = pttt.* • Roadrunner accepts the same command line arguments as Zope testrunner. So we can run our tests like this: $ bin/roadrunner -s pttt.ﬁzzbuzz Wednesday, October 28, 2009

Plone Integration Testing Resources • Programmatically Manipulate Plone: http://plonemanual.twinapex.ﬁ/content/ index.html • PloneTestCase Tutorial - PloneConf 2004: http://www.zope.org/Members/ shh/Tutorial • http://plone.org/documentation/tutorial/testing/integration-doctests-using- plonetestcase • PloneTestCase: http://plone.org/products/plonetestcase Wednesday, October 28, 2009

Doctests • tests that look like interactive Python sessions • When to use • API Documentation, verify examples work • its a matter of taste, some people prefer unittests especially when there are a lot of variations • they are especially good for functional tests where the prose explains the acceptance test and the code veriﬁes that it is working Wednesday, October 28, 2009

Doctest boilerplate import unittest import doctest from Testing import ZopeTestCase as ztc from example.tests.tests import base def test_suite(): """This sets up a test suite that actually runs the tests in the class above """ return unittest.TestSuite([ ztc.ZopeDocFileSuite( 'tests/functional.txt', package='example.tests', test_class=base.ExampleFunctionalTestCase, optionflags=doctest.REPORT_ONLY_FIRST_FAILURE | doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS), ]) Wednesday, October 28, 2009

A Simple Doctest First we'll import and instantiate our FizzBuzzWriter class >>> from pttt.fizzbuzz.fizzbuzz import FizzBuzzWriter >>> writer = FizzBuzzWriter() Let's write out single numbers >>> writer.write_number(1) '1' >>> writer.write_number(3) 'FIZZ' >>> writer.write_number(15) 'FIZZ BUZZ' Finally let's write a sequence of numbers >>> list(writer.write_numbers(15)) ['1', '2', 'FIZZ', '4', 'BUZZ', 'FIZZ', '7', '8', 'FIZZ', 'BUZZ', '11', 'FIZZ', '13', '14', 'FIZZ BUZZ'] Wednesday, October 28, 2009

Functional and System Tests • End to end tests • Typically black-box tests that act as an end-user would and are written from end user’s point of view. eg. click the button labeled “Login” vs. call the login method • inherently brittle as the UI often changes • Use zope.testbrowser, twill, Selenium • typically written as doctests but don’t need to be Wednesday, October 28, 2009

Functional Tests • When to use: • blackbox testing • automated acceptance testing • Smoke or face-saving tests - exercise most important paths of business value • When not to use: • testing every corner case. use unit tests for that Wednesday, October 28, 2009

About zope.testbrowser • testbrowser implements a Python scriptable browser that simulates accessing your Plone site. • Full docs for zope.testbrowser: http://pypi.python.org/pypi/zope.testbrowser • Note that using the testbrowser in Plone we should use the Five version which wraps it for use in a Zope 2 environment: from Products.Five.testbrowser import Browser • Not Javascript aware. If you need to test DHTML, Javascript, Ajax et al then you will need to use something like Selenium. Wednesday, October 28, 2009

Using zope.testbrowser • Don’t hammer Plone.org ;) from zope.testbrowser.browser import Browser # this version of testbrowser works with external resources # for testing your own apps using internal Browser class: # from Products.Five.testbrowser import Browser browser = Browser('http://plone.org') browser.getControl(name='SearchableText').value = 'Testing' browser.getForm(id='search').submit() assert browser.isHtml assert 'Untested Code is Broken Code' in browser.contents Wednesday, October 28, 2009

Plone Functional testbrowser API basics • Open your test Plone site: browser.open('http://nohost/plone/') • Click a link: browser.getLink(link_text).click() • Click the ﬁrst link when multiple have the same name: browser.getLink(link_text, index=0).click() • Get URL: browser.url • Reload: browser.reload() • Change form value: browser.getControl(input_name).value = ‘Whatever’ • Submit form: browser.getForm(id) Wednesday, October 28, 2009

Two ways to login • One is to use Plone’s login mechanism, arguably more black-box since it should simulate what your users will actually do. >>> browser.open(portal_url) >>> browser.getControl(name='__ac_name').value = portal_owner >>> browser.getControl(name='__ac_password').value = default_password >>> browser.getControl(name='submit').click() • Alternatively you could add an auth header: >>> browser.addHeader('Authorization', 'Basic root:secret') Wednesday, October 28, 2009

Functional Testing Example • Let’s take a look at an existing functional testing example from Plone’s source code from plone/app/workﬂow/tests/sharingpage.txt: • A couple things about this • browser.handleErrors = False means you’ll get proper tracebacks instead of cryptic 500 Errors • demonstrates pure black box testing, no assumptions of internal state Wednesday, October 28, 2009

Functional Testing Resources • http://plone.org/documentation/tutorial/testing/functional-tests • Face Saving Tests: http://gojko.net/2007/09/25/effective-user-interface- testing Wednesday, October 28, 2009

System Tests using twill • You can write system tests using testbrowser as well • Twill provides a simple domain speciﬁc language (DSL) for performing web tests. It also provides a Python API. • When to use • when doing whole system testing that includes caching proxy, multiple web servers, backend and database • testing live systems • deployment testing Wednesday, October 28, 2009

Installing twill in your buildout • twill is on pypi, simply use zc.recipe.egg to install its scripts: [twill] recipe = zc.recipe.egg eggs = twill • Also make sure to add the part to your [buildout] parts section. Wednesday, October 28, 2009

Simple Twill Script go http://plone.org formvalue searchform SearchableText testing submit # assert that we found the following text find 'Untested Code is Broken Code' # assert that we didn't find this text: notfind 'Testing sucks' Wednesday, October 28, 2009

Running twill interactively $ bin/twill-sh -= Welcome to twill! =- current page: *empty page* >> go http://plone.org ==> at http://plone.org current page: http://plone.org >> formvalue searchform SearchableText testing current page: http://plone.org >> submit current page: http://plone.org/search?SearchableText=testing >> current page: http://plone.org/search?SearchableText=testing >> # assert that we found the following text current page: http://plone.org/search?SearchableText=testing >> find 'Untested Code is Broken Code' current page: http://plone.org/search?SearchableText=testing >> current page: http://plone.org/search?SearchableText=testing >> # assert that we didn't find this text: current page: http://plone.org/search?SearchableText=testing >> notfind 'Testing sucks' current page: http://plone.org/search?SearchableText=testing Wednesday, October 28, 2009

Writing a System Test • Verify your SSL conﬁguration: # TEST: join form redirects to https go http://mysite.com follow "Join" # test that the resulting URL is https:// url "^https:" url "/join_form$" find "Registration Form" # fill in registration form and register fv 2 fullname "Test User" fv 2 email "test@tester.com" submit find "You have been registered." fv 2 form.submitted 1 submit find "You are now logged in" # home page is http url "^http://" Wednesday, October 28, 2009

Mock and Fake objects • When to use: • unittest code that has external dependencies such as databases or network connections • depends on external components Wednesday, October 28, 2009

Basic Mocks • Let’s say you have a test method that expects an object to be provided which provides an absolute_url method in order to return a calculated URL: • Without using an Integration test, how could you do this? Simply create a simple Python class that implements only the methods you need. • This can get tiresome if you have to create many of these mocks, or if you want to make some assertions about what is called on those mocks with what arguments. Wednesday, October 28, 2009

Basic Mocking Example def calc_url(context): return context.absolute_url() + "/calculated_url" class MockDocument: def absolute_url(self): return "http://plone/some_document" def test_calc_url(self): mock = MockDocument() self.failUnlessEqual(calc_url(mock), "http://plone/some_document/calculated_url") Wednesday, October 28, 2009

Mocker • Python library for creating Mock objects and managing expectations we have for those Mock objects. • Other more advanced features as well such as proxies, patching and proxy replacement. • Unittest integration • Martin Aspeli took this further and created plone.mocktestcase which provides some helper methods that are speciﬁc to Zope and Plone development. http://pypi.python.org/pypi/plone.mocktestcase Wednesday, October 28, 2009

A Mocking Example • Writing an interface to an external network service • Using the Component Architecture to create an abstract interface for a remote network service and using MockTestCase’s mock_utility to make this available to application code under test: In the network_service = self.mocker.mock() self.mock_utility(self.video_provider, INetworkService) Wednesday, October 28, 2009

Mocking Resources • http://www.martinaspeli.net/articles/mock-testing-with-mocker-and- plone.mocktestcase Wednesday, October 28, 2009

Coverage • answers the question: are my tests covering all the code in my application? • for production code I aim for 100%. Why? anything less than 100% can hide defects. • 100% code coverage is considered minimum standard * • does NOT guarantee the code has no bugs • * - http://www.plope.com/Members/chrism/a_hundred_percent Wednesday, October 28, 2009

Coverage won’t catch this • This pseudo-code example would give 100% code coverage. What if I passed 0 to myfunc? def test_myfunc(): myfunc(100) def myfunc(n): return 100 / n Wednesday, October 28, 2009

Generating coverage report • Zope’s testrunner comes with a builtin coverage report tool, invoke like this: $ bin/instance test -s pttt.ﬁzzbuzz --coverage=./coverage • When calculating coverage you should know that code runs much slower. So grab a coffee, tea or water and catch up on your xkcd. • The results are in: Ran 5 tests with 0 failures and 0 errors in 0.005 seconds. Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds. lines cov% module (path) 16 100% pttt.fizzbuzz.fizzbuzz (/Users/jbb/co/pttt/src/pttt.fizzbuzz/ pttt/fizzbuzz/fizzbuzz.py) Wednesday, October 28, 2009

HTML Coverage report using z3c.coverage • Produces nice looking reports that you can discuss with your team: Wednesday, October 28, 2009

HTML Coverage report using z3c.coverage • Add this part to buildout: [z3c-coverage] recipe = zc.recipe.egg eggs = z3c.coverage extra-paths = ${zope2:location}/lib/python • Don’t forget to add z3c-coverage to [buildout] parts = • Generate reports: $ bin/coverage ./coverage ./coverage/reports Wednesday, October 28, 2009

Coverage Resources • http://plone.org/documentation/tutorial/testing/code-coverage Wednesday, October 28, 2009

Pdb-fu (debugging) • Getting test failures from Zope Testrunner? Use -D to drop you into Pdb using post-mortem debugging • Use pdb’s jump command to re-run certain lines of codes. Watch out for side-effects. • Special variable __return__ is set. Use “continue” to go to the end of the method and then you can easily examine the return result. • Breakpoint variables. Looping through 1000 items, set a breakpoint with an expression. • Missing source code? This could help, add to your buildout.cfg: [buildout] unzip = true Wednesday, October 28, 2009

General Testing Tips • Write at the lowest level possible. This will ensure that your test will run as quickly as possible which becomes important when you have hundreds or thousands of tests. • Take your time, write out what you’re trying to do in your commends ﬁrst before writing actual code: def test_whatever(self): # verify document was created # ensure that notification was sent Wednesday, October 28, 2009

General Testing Tips • Broken tests that lie around for a long time because they are minor issues that no one can be bothered to ﬁx can mask larger bugs due to too much noise and not enough signal. 0 failing tests (and no cheating!) • Hard to test something? Step back, try to break it up into easier to test pieces. Wednesday, October 28, 2009

Some Further Resources • Plone Testing tutorial: http://plone.org/documentation/tutorial/testing/ • plone.reload: http://pypi.python.org/pypi/plone.reload • http://diveintopython.org/unit_testing/index.html • example tests of all sorts, mostly still valid: http://svn.plone.org/svn/ collective/examples/example.tests/trunk/ Wednesday, October 28, 2009

Open Spaces and Sprinting • For all of those who want to have open discussions about Testing in Plone we’ll have an Open Space on Friday. • Also I’ll be sprinting on bringing roadrunner up to spec with Plone 4, Dexterity and Zope 2.12. Interested? Drop me a line. Wednesday, October 28, 2009

Thanks • Your Feedback is appreciated: jbb@scryent.com • Find me on twitter: @hexsprite • Good luck and happy testing! Wednesday, October 28, 2009

Thanks • Photos used licensed under Creative Commons 2.0: • http://www.ﬂickr.com/photos/rupertuk/2102548824/ • http://www.ﬂickr.com/photos/kacey/2626513873 Wednesday, October 28, 2009

http://blog.plumi.org	92
http://www.slideshare.net	28
http://static.slidesharecdn.com	1
http://translate.googleusercontent.com	1

Plone Testing Tools And Techniques

by hexsprite on Oct 28, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 122

Statistics

Plone Testing Tools And Techniques — Presentation Transcript