Your SlideShare is downloading. ×
Making the most out
  of your test suite



                    Eric Holscher
         http://ericholscher.com
           ...
Overview


» Defining your tests
» Running your tests on every commit
» Monitoring and deployment
» Playing with test data
Defining your tests
I download your code...
Now I want to run your tests
How do I do that?
Ways to run Django tests

» ./manage.py test <app>
» ./manage.py test testapp
» nosetests
» setup.py test
» run_tests.py
»...
Python has even more
We need a standard way to
    run Python tests
setup.py test is the answer
...but it’s not ready
setup.py test


» Part of setuptools/distribute
» Will be a part of distutils2
» Django has no support for this yet
» We c...
Pony Build
Pony Build


» Started by Titus Brown
» Super simple Continuous Integration
  Server
» Interesting part is the client
» gi...
Define how to run tests in
        Python
Full Example

class PonyBuild(BaseBuild):

  def __init__(self):
    super(PonyBuild, self).__init__()
    self.name = "sp...
Supports standard ways of
running tests out of the box
Ships with support for


» ./manage.py test <app>
» ./manage.py test <testapp>
» nose
» setup.py test
Simple Pony Build


class PonyBuild(DjangoGitBuild):
   def __init__(self):
     super(PonyBuild, self).__init__()
     se...
Lets you run your build in
 different environments
Different Environments



» Local Environment
» Temporary Directory
» Virtualenv
At least ship with a shell
          script
Document how to run tests
Continuous Integration
“Run your tests on every
       commit”
A fancy wrapper around a
       shell script
Without this, your tests are
          useless
Make sure you setup.py
 install your package!
--noinput is your friend
Use Virtualenv
Watch Titus’s Pycon Talk
Three main options



» Buildbot
» Hudson
» Devmason
Hudson
Hudson stats


» Written in Java :x
» Great plugin community
» Super simple to setup
» Becoming Defacto
» Worthwhile if yo...
We use this at work and it
        kicks ass
hudson.djangoproject.com
Setting up your own
Useful Capabilities


» Coverage
» XML Output
» PyLint
» Profiling
Plugins



» Hudson Cobertura Plugin
» Hudson Violations Plugin
» IRC Plugin
pip install coverage
(Compiles a C extension)
Coverage.py



coverage run manage.py test
or
coverage run 1.1.X/tests/runtests.py
coverage xml -o 1.1.X/coverage.xml
XML Output

» Gareth Rushgrove’s django-test-
  extensions
» Requires a subclassed TestRunner
» github.com/garethr/django-...
Example Build Bits



from my_test_settings import *
TEST_RUNNER =
‘test_extensions.test_runners.xmloutput’
Nightly Tasks




» pylint
» profiling
django-nose makes a lot of
       this easier
Devmason
Language Agnostic Build
     Reporting API
<3 Piston
devmason.com/docs
Integrates with Github and
         Bitbucket
Have simple API support for
 Python unittest and Pony
          Build
Keep your builds in the
      Pony Barn
“Free” for Python
   community
Will hopefully help
standardize tests
github.com/ericholscher/
    devmason-server
github.com/ericholscher/
     devmason-utils
Monitoring & Deployment
“Django Kong”


» “Functional Testing”
» “Devops”
» Uses Twill: twill.idyll.org
» github.com/ericholscher/django-kong
Simple Example


go {{ site.url }}/
code 200
find “Awesome”
follow “Events”
code 200
find “Today’s Events”
Check to see if your code
  works on live sites
Easily run your tests across
      all types of a site
Catch errors users are
having but never reporting
Gives you a neat “feel” for
        your sites
We check all of our sites
    every 3 minutes
A neat hack allows you to
test your site locally too!
Stuff to do


» Nagios integration
» “Run levels”
» Better aggregates timing data
» Have tests live on disk/in VCS
Run nightly automated
tools to check your sites
Patu
Test Utils Crawler
Get a Dashboard!
Test Data
My Pony
We all run tests a lot.
There is valuable data about
    our software there.
CPANTS




» Perl’s Effort of aggregating test data
» Lots of pretty graphs
Cpants pivots



» Operating System
» Architecture
» Language Versions
We need this for Python
Devmason can serve as the
   reporting platform
We have all the tech we
need, just need to tie it
        together
Things to Remember
Document how to run your
tests (in executable form)
Get a CI server setup
What you don’t measure
  you don’t improve
Get some basic monitoring
for your sites functionality
URLS
» github.com/ericholscher/django-kong
» hudson-ci.org
» github.com/ctb/pony-build
» pypants.org
» cpantesters.org
» d...
Questions?




» eric@ericholscher.com
» twitter.com/ericholscher
Making the most of your Test Suite
Making the most of your Test Suite
Making the most of your Test Suite
Making the most of your Test Suite
Making the most of your Test Suite
Making the most of your Test Suite
Making the most of your Test Suite
Upcoming SlideShare
Loading in...5
×

Making the most of your Test Suite

3,403

Published on

Published in: Technology
1 Comment
11 Likes
Statistics
Notes
  • Where can I download this slides?
    Thanks!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
3,403
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
1
Likes
11
Embeds 0
No embeds

No notes for slide

Transcript of "Making the most of your Test Suite"

  1. 1. Making the most out of your test suite Eric Holscher http://ericholscher.com Djangocon.eu 2010
  2. 2. Overview » Defining your tests » Running your tests on every commit » Monitoring and deployment » Playing with test data
  3. 3. Defining your tests
  4. 4. I download your code...
  5. 5. Now I want to run your tests
  6. 6. How do I do that?
  7. 7. Ways to run Django tests » ./manage.py test <app> » ./manage.py test testapp » nosetests » setup.py test » run_tests.py » ...
  8. 8. Python has even more
  9. 9. We need a standard way to run Python tests
  10. 10. setup.py test is the answer
  11. 11. ...but it’s not ready
  12. 12. setup.py test » Part of setuptools/distribute » Will be a part of distutils2 » Django has no support for this yet » We can lead standardization efforts
  13. 13. Pony Build
  14. 14. Pony Build » Started by Titus Brown » Super simple Continuous Integration Server » Interesting part is the client » github.com/ericholscher/pony_barn
  15. 15. Define how to run tests in Python
  16. 16. Full Example class PonyBuild(BaseBuild): def __init__(self): super(PonyBuild, self).__init__() self.name = "sphinx" self.required = ['nose'] self.repo_url = 'http://bitbucket.org/birkenfeld/sphinx/' def define_commands(self): self.commands = [ pony.HgClone(self.repo_url, egg=self.get_name()), pony.BuildCommand([self.context.python, 'setup.py', 'install']), pony.TestCommand([self.context.python, 'tests/run.py']), ]
  17. 17. Supports standard ways of running tests out of the box
  18. 18. Ships with support for » ./manage.py test <app> » ./manage.py test <testapp> » nose » setup.py test
  19. 19. Simple Pony Build class PonyBuild(DjangoGitBuild): def __init__(self): super(PonyBuild, self).__init__() self.repo_url = "git://github.com/alex/django-filter.git" self.name = "django-filter" self.package_name = 'django_filters' self.installed_apps = ['django_filters', 'django_filters.tests']
  20. 20. Lets you run your build in different environments
  21. 21. Different Environments » Local Environment » Temporary Directory » Virtualenv
  22. 22. At least ship with a shell script
  23. 23. Document how to run tests
  24. 24. Continuous Integration
  25. 25. “Run your tests on every commit”
  26. 26. A fancy wrapper around a shell script
  27. 27. Without this, your tests are useless
  28. 28. Make sure you setup.py install your package!
  29. 29. --noinput is your friend
  30. 30. Use Virtualenv
  31. 31. Watch Titus’s Pycon Talk
  32. 32. Three main options » Buildbot » Hudson » Devmason
  33. 33. Hudson
  34. 34. Hudson stats » Written in Java :x » Great plugin community » Super simple to setup » Becoming Defacto » Worthwhile if you have resources
  35. 35. We use this at work and it kicks ass
  36. 36. hudson.djangoproject.com
  37. 37. Setting up your own
  38. 38. Useful Capabilities » Coverage » XML Output » PyLint » Profiling
  39. 39. Plugins » Hudson Cobertura Plugin » Hudson Violations Plugin » IRC Plugin
  40. 40. pip install coverage (Compiles a C extension)
  41. 41. Coverage.py coverage run manage.py test or coverage run 1.1.X/tests/runtests.py coverage xml -o 1.1.X/coverage.xml
  42. 42. XML Output » Gareth Rushgrove’s django-test- extensions » Requires a subclassed TestRunner » github.com/garethr/django-test- extensions » github.com/danielfm/unittest-xml- reporting
  43. 43. Example Build Bits from my_test_settings import * TEST_RUNNER = ‘test_extensions.test_runners.xmloutput’
  44. 44. Nightly Tasks » pylint » profiling
  45. 45. django-nose makes a lot of this easier
  46. 46. Devmason
  47. 47. Language Agnostic Build Reporting API
  48. 48. <3 Piston
  49. 49. devmason.com/docs
  50. 50. Integrates with Github and Bitbucket
  51. 51. Have simple API support for Python unittest and Pony Build
  52. 52. Keep your builds in the Pony Barn
  53. 53. “Free” for Python community
  54. 54. Will hopefully help standardize tests
  55. 55. github.com/ericholscher/ devmason-server
  56. 56. github.com/ericholscher/ devmason-utils
  57. 57. Monitoring & Deployment
  58. 58. “Django Kong” » “Functional Testing” » “Devops” » Uses Twill: twill.idyll.org » github.com/ericholscher/django-kong
  59. 59. Simple Example go {{ site.url }}/ code 200 find “Awesome” follow “Events” code 200 find “Today’s Events”
  60. 60. Check to see if your code works on live sites
  61. 61. Easily run your tests across all types of a site
  62. 62. Catch errors users are having but never reporting
  63. 63. Gives you a neat “feel” for your sites
  64. 64. We check all of our sites every 3 minutes
  65. 65. A neat hack allows you to test your site locally too!
  66. 66. Stuff to do » Nagios integration » “Run levels” » Better aggregates timing data » Have tests live on disk/in VCS
  67. 67. Run nightly automated tools to check your sites
  68. 68. Patu
  69. 69. Test Utils Crawler
  70. 70. Get a Dashboard!
  71. 71. Test Data
  72. 72. My Pony
  73. 73. We all run tests a lot.
  74. 74. There is valuable data about our software there.
  75. 75. CPANTS » Perl’s Effort of aggregating test data » Lots of pretty graphs
  76. 76. Cpants pivots » Operating System » Architecture » Language Versions
  77. 77. We need this for Python
  78. 78. Devmason can serve as the reporting platform
  79. 79. We have all the tech we need, just need to tie it together
  80. 80. Things to Remember
  81. 81. Document how to run your tests (in executable form)
  82. 82. Get a CI server setup
  83. 83. What you don’t measure you don’t improve
  84. 84. Get some basic monitoring for your sites functionality
  85. 85. URLS » github.com/ericholscher/django-kong » hudson-ci.org » github.com/ctb/pony-build » pypants.org » cpantesters.org » devmason.com » github.com/ericholscher/devmason-server » github.com/ericholscher/devmason-utils
  86. 86. Questions? » eric@ericholscher.com » twitter.com/ericholscher

×