Continuous Integration Testing for Plone Using Hudson

Continuous Integration Testing with Hudson Wednesday, October 27, 2010

Continuous Integration Wednesday, October 27, 2010

Continuous Integration • Extreme Programming principle • Martin Fowler http://www.martinfowler.com/articles/ continuousIntegration.html Wednesday, October 27, 2010

Continuous Integration • Maintain a Source • Test in a Clone of the Repository Production Environment • Automate the Build • Make the Build Self- • Make it Easy to Get the Latest Testing Deliverables • Everyone Commits • Everyone Can See Every Day the Results of the • Every Commit Latest Build Should Be Built • Automate • Keep the Build Fast Deployment Wednesday, October 27, 2010

Essentially • Commit your code • Test the crap out of it • Regressions are bad • Assign blame • Shame is a great motivator Wednesday, October 27, 2010

Wednesday, October 27, 2010

Hudson vs Buildbot? Wednesday, October 27, 2010

*shrug* Wednesday, October 27, 2010

Hudson (Pros) Wednesday, October 27, 2010

Easy to install/run %> java -jar hudson.war Wednesday, October 27, 2010

Easy to Conﬁgure • Create • Modify • Manage • All TTW Wednesday, October 27, 2010

Frequent Releases ~ 1 / Week Wednesday, October 27, 2010

Inline Documentation Wednesday, October 27, 2010

Plugins Wednesday, October 27, 2010

Hudson (Cons) Wednesday, October 27, 2010

It’s Java. Wednesday, October 27, 2010

It’s Java. 1. It eats resources 2. I don’t know how to write plugins Wednesday, October 27, 2010

Hudson Wednesday, October 27, 2010

Project Layout Wednesday, October 27, 2010

Project Layout • Project/Job Wednesday, October 27, 2010

Project Layout • Project/Job • Build Wednesday, October 27, 2010

Project Layout • Project • Build • Trigger Wednesday, October 27, 2010

Triggers • Other projects • Remote trigger • Periodically • SCM polling Wednesday, October 27, 2010

Project Layout • Project/Job • Build • Trigger • Build Step Wednesday, October 27, 2010

Build Steps • Shell script • Python script • Ant • Maven Wednesday, October 27, 2010

Project Layout • Project/Job • Build • Trigger • Build Step • Post-Build Action Wednesday, October 27, 2010

Post-Build Actions • Trigger builds • Archive results • Generate reports • Send email Wednesday, October 27, 2010

Build Status Wednesday, October 27, 2010

Build Status • Successful • Unstable • Failed Wednesday, October 27, 2010

Build Stability Wednesday, October 27, 2010

Filesystem Layout Wednesday, October 27, 2010

Filesystem Layout hudson / conﬁg.xml hudson.plugins.*.xml jobs / myjob / builds / 2010-10-21_11-56-40 2010-10-21_12-20-26 conﬁg.xml workspace / Wednesday, October 27, 2010

Plugins Wednesday, October 27, 2010

Plugins • Green Balls • Violations • Python • Cobertura • Subversion • Disk Usage • Trac • Chuck Norris • Warnings Wednesday, October 27, 2010

Plone Integration Wednesday, October 27, 2010

Buildout & mr.developer Wednesday, October 27, 2010

buildout.cfg [buildout] ... extensions = mr.developer Wednesday, October 27, 2010

buildout.cfg [buildout] ... extensions = mr.developer sources = sources [sources] fsd.core = svn https:.../weblion/fsd.core/trunk fsd.membrane = svn https:.../weblion/fsd.membrane/trunk Wednesday, October 27, 2010

mr.developer • bin/develop checkout • bin/develop status • bin/develop update • bin/develop activate (deactivate) • bin/develop reset • bin/develop purge Wednesday, October 27, 2010

buildout.cfg [buildout] ... extensions = mr.developer sources = sources auto-checkout = fsd.core fsd.membrane [sources] fsd.core = svn https:.../weblion/fsd.core/trunk fsd.membrane = svn https:.../weblion/fsd.membrane/trunk Wednesday, October 27, 2010

buildout bin/ buildout develop ... bootstrap.py buildout.cfg ... src/ fsd.core fsd.membrane Wednesday, October 27, 2010

Tests Wednesday, October 27, 2010

Tests • bin/test returns text • Hudson wants JUnit (XML) format • collective.xmltestreport Wednesday, October 27, 2010

hudson.cfg [buildout] extends = buildout.cfg parts += xmltestrunner [xmltestrunner] recipe = collective.xmltestreport eggs = ${test:eggs} defaults = ${test:defaults} + ['--xml'] Wednesday, October 27, 2010

Test Coverage Wednesday, October 27, 2010

Test Coverage [coverage] recipe = zc.recipe.egg eggs = coverage initialization = sys.argv = sys.argv[:] + ['run', 'bin/xmltestrunner', '-k', '-q', '--xml'] Wednesday, October 27, 2010

Test Coverage [report] recipe = zc.recipe.egg eggs = coverage scripts = coverage=report initialization = eggs = '${buildout:eggs-directory}' bin = '${buildout:directory}/bin' exclude = '--omit=' + ','.join([eggs, sys.prefix, bin]) sys.argv = sys.argv[:] + ['xml', '-i', exclude, include] Wednesday, October 27, 2010

Test Coverage Wednesday, October 27, 2010

Code Analysis Wednesday, October 27, 2010

Code Analysis • zptlint • pyﬂakes • pylint • jslint Wednesday, October 27, 2010

zptlint Wednesday, October 27, 2010

zptlint • http://pypi.python.org/pypi/zptlint • Scan page templates for parser and output errors Wednesday, October 27, 2010

zptlint [zptlint] recipe = zc.recipe.egg eggs = zptlint entry-points = zptlint=zptlint:run Wednesday, October 27, 2010

zptlint <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" xmlns:tal="http://xml.zope.org/namespaces/tal" xmlns:metal="http://xml.zope.org/namespaces/metal" lang="en" metal:use-macro="context/main_template/macros/master"> <body> <metal:fill fill-slot="content-core"> <div tal:content="structure view/context/text/output"> <h2>People</h2> <div tal:content="structure context/@@people" /> </metal:fill> </body> </html> Wednesday, October 27, 2010

zptlint %> bin/zptlint src/.../view.pt Wednesday, October 27, 2010

zptlint %> bin/zptlint src/.../view.pt *** Error in: src/.../view.pt <class 'zope.tal.taldefs.TALError'>: TAL attributes on <div> require explicit </div> , at line 12, column 9 Wednesday, October 27, 2010

Parsing Log Messages • Warnings plug-in • Regular expression • Groovy script (groovy.codehaus.org) Wednesday, October 27, 2010

Priorities • HIGH • NORMAL • LOW Wednesday, October 27, 2010

zptlint [zptlint-test] recipe = collective.recipe.template input = inline: #!/bin/sh if [ -e zptlint.log ]; then echo "Old zptlint.log file removed" rm zptlint.log fi echo "Running zptlint-test" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find $pkg -regex ".*.[c|z]?pt" | xargs -r bin/zptlint | perl -p -e 's/s+$/ /g;s/s+/ /g;s/*{3}s?/n/g' >> zptlint.log done output = ${buildout:directory}/bin/zptlint-test mode = 755 Wednesday, October 27, 2010

zptlint [buildout] ... # A list of package locations to be examined # by Hudson package-directories = src/fsd.core/fsd/core src/fsd.membrane/fsd/membrane Wednesday, October 27, 2010

zptlint [zptlint-test] recipe = collective.recipe.template input = inline: #!/bin/sh if [ -e zptlint.log ]; then echo "Old zptlint.log file removed" rm zptlint.log fi echo "Running zptlint-test" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find $pkg -regex ".*.[c|z]?pt" | xargs -r bin/zptlint | perl -p -e 's/s+$/ /g;s/s+/ /g;s/*{3}s?/n/g' >> zptlint.log done output = ${buildout:directory}/bin/zptlint-test mode = 755 Wednesday, October 27, 2010

pyﬂakes Wednesday, October 27, 2010

pyﬂakes • Python syntax checking • Find common errors quickly • Doesn’t attempt to run Python code Wednesday, October 27, 2010

pyﬂakes [pyflakes] recipe = zc.recipe.egg eggs = pyflakes entry-points = pyflakes=pyflakes.scripts.pyflakes:main Wednesday, October 27, 2010

pyﬂakes %> bin/pyflakes src/.../person.py Wednesday, October 27, 2010

pyﬂakes %> bin/pyflakes src/.../person.py src/.../person.py:25: 'ATTRIBUTE_NAME' imported but unused src/.../person.py:25: 'IAttributeUUID' imported but unused Wednesday, October 27, 2010

pyﬂakes [pyflakes-test] recipe = collective.recipe.template input = inline: #!/bin/sh if [ -s pyflakes.log ]; then rm pyflakes.log echo "Old pyflakes.log file removed" fi echo "Running pyflakes" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find -L $pkg -regex ".*.py"|xargs -r bin/pyflakes >> pyflakes.log done output = ${buildout:directory}/bin/pyflakes-test mode = 755 Wednesday, October 27, 2010

pylint Wednesday, October 27, 2010

pylint • Syntax checking • PEP8 compliance • Imports code • You’ve done everything wrong Wednesday, October 27, 2010

pylint [pylint] recipe = zc.recipe.egg eggs = logilab.pylintinstaller extra-paths = ${instance:location}/lib/python entry-points = pylint=pylint.lint:Run arguments = [ '--output-format=parseable', '--zope=y', '--reports=y', '--disable-msg= E0611,F0401,W0232,E1101,C0103,C0111,R0201,W0201,R0911,R0904,F0220, E1103,R0901,E0211,E0213,E1002,W0622', '--generated-members=objects', ] + sys.argv[1:] Wednesday, October 27, 2010

pylint • disable-msg • R0911: Too many return statements (%s/%s) • R0201: Method could be a function • W0201: Attribute %r deﬁned outside of __init__ • http://pylint-messages.wikidot.com/all-messages Wednesday, October 27, 2010

pylint [pylint-test] recipe = collective.recipe.template input = inline: #!/bin/sh if [ -s pylint.log ]; then rm pylint.log echo "Old pylint.log file removed" fi echo "Running pylint" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find -L $pkg -regex ".*.py" | xargs bin/pylint >> pylint.log done echo "Finished" output = ${buildout:directory}/bin/pylint-test mode = 755 Wednesday, October 27, 2010

pylint Wednesday, October 27, 2010

Selenium Wednesday, October 27, 2010

Selenium • Automated web application testing Wednesday, October 27, 2010

Selenium 1. Open this page 2. Click that thing 3. Fill out that form 4. Click submit Did it work? Wednesday, October 27, 2010

Selenium Testcase def test_login_overlay(self): self.open("/") self.wait() self.selenium.click('link=Log in') self.waitForElement('form#login_form') self.selenium.type("name=__ac_name", TEST_USER_NAME) self.selenium.type("name=__ac_password", TEST_USER_PASSWORD) self.selenium.click("submit") self.wait() self.failUnless(self.selenium.is_text_present("Log out")) Wednesday, October 27, 2010

Selenium Testcase class SeleniumLayer(PloneSandboxLayer): defaultBases = (HOST_ADJUSTABLE_ZSERVER_FIXTURE, PLONE_FIXTURE) # Connection parameters seleniumHost = os.environ.get('SELENIUM_HOST', 'localhost') seleniumPort = os.environ.get('SELENIUM_PORT', '4444') seleniumBrowser = os.environ.get('SELENIUM_BROWSER', '*firefox') def setUpZope(self, app, configurationContext): ... def setUpPloneSite(self, portal): ... url = "http://%s:%s/%s" % (self['host'], self['port'], PLONE_SITE_ID) self['selenium'] = selenium.selenium(self.seleniumHost, self.seleniumPort, self.seleniumBrowser, url) self['selenium'].start() def tearDownPloneSite(self, portal): self['selenium'].stop() del self['selenium'] Wednesday, October 27, 2010

Selenium RC • Firefox • Internet Explorer • Safari • Chrome • Opera • more... Wednesday, October 27, 2010

Selenium Grid Wednesday, October 27, 2010

“Firefox?” Wednesday, October 27, 2010

Thanks! Wednesday, October 27, 2010

Selenium Test Runner fsd/ core/ ... selenium/ base.py testSelenium.py tests/ ... Wednesday, October 27, 2010

Selenium Test Runner [selenium] recipe = collective.xmltestreport eggs = fsd.core [test] fsd.membrane [test] defaults = ['--tests-pattern', '^f?selenium$', '--xml'] Wednesday, October 27, 2010

Selenium Test Runner [selenium-firefox] recipe = ${selenium:recipe} eggs = ${selenium:eggs} defaults = ${selenium:defaults} [selenium-safari] recipe = ${selenium:recipe} eggs = ${selenium:eggs} defaults = ${selenium:defaults} Wednesday, October 27, 2010

Selenium Test Runner %> bin/selenium-firefox %> bin/selenium-safari Wednesday, October 27, 2010

Selenium Test Runner workspace/ bin/ ... parts/ selenium-firefox/ testreports/ *.xml selenium-safari/ testreports/ *.xml Wednesday, October 27, 2010

~ Wednesday, October 27, 2010

Sprint! • Saturday • QA Sprint • Selenium testing and more • Free food! Wednesday, October 27, 2010

Multi-Conﬁguration Projects Wednesday, October 27, 2010

Multi-Conﬁguration Projects • Run a build for each set of n x n ( x n x n) variables. Wednesday, October 27, 2010

Wish List Wednesday, October 27, 2010

Wish List • Load testing • Sphinx builds • Chameleon compatibility checking • Windmill Wednesday, October 27, 2010

Best Practices Wednesday, October 27, 2010

Check in Your Configs • Configuration is stored in XML files • Easy to back up to SVN Wednesday, October 27, 2010

Check in Your Configs hudson / config.xml hudson.plugins.*.xml jobs / myjob / builds / 2010-10-21_11-56-40 2010-10-21_12-20-26 config.xml workspace / Wednesday, October 27, 2010

Check in Your Conﬁgs Wednesday, October 27, 2010

Watch Your Disk Space • “Hudson disk-usage” plugin Wednesday, October 27, 2010

Watch Your Disk Space Wednesday, October 27, 2010

Use an Egg Cache • Set up a default.cfg for your hudson account. [buildout] eggs-directory = /home/hudson/.buildout/eggs download-cache = /home/hudson/.buildout/downloads Wednesday, October 27, 2010

Pin unittest2 • (If you’re using it) • plone.app.testing • 3x faster buildout Wednesday, October 27, 2010

Split Big Builds • How long does the build take? • How often does each part need to be run? • Tests? • Coverage? Wednesday, October 27, 2010

Split Big Builds • Build • Test • Syntax Checking • Coverage • Selenium Wednesday, October 27, 2010

Split Big Builds 1. “Build other projects” post-build action Wednesday, October 27, 2010

Split Big Builds 1. “Build other projects” post-build action 2. Custom workspace Wednesday, October 27, 2010

Split Big Builds 1. “Build other projects” post-build action 2. Custom workspace 3. Build Fingerprinting Wednesday, October 27, 2010

Fingerprinting • “These builds are related” • Archive an md5 hash of a ﬁle to act as an identiﬁer Wednesday, October 27, 2010

Fingerprinting In parent build: In parent and child builds: Wednesday, October 27, 2010

Reporting Wednesday, October 27, 2010

Hudson Report Views Wednesday, October 27, 2010

Email Wednesday, October 27, 2010

Chuck Wednesday, October 27, 2010

iPhone Wednesday, October 27, 2010

Extreme Feedback Devices Wednesday, October 27, 2010

hudson.plone.org • Currently builds Plone 4.0.x • Core developers have access • Add your core & collective projects Wednesday, October 27, 2010

Thanks • Martin Aspeli • Timo Stollenwerk • Hanno Schlichting Wednesday, October 27, 2010

QA Sprint! • Saturday • QA Sprint • Selenium testing and more • Free food! Wednesday, October 27, 2010

Continuous Integration Testing for Plone Using Hudson

by Eric Steele

Accessibility

Categories

Upload Details

Usage Rights

Statistics