Making the most of your Test Suite

  • 3,313 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Where can I download this slides?
    Thanks!
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
3,313
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
1
Likes
11

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Making the most out of your test suite Eric Holscher http://ericholscher.com Djangocon.eu 2010
  • 2. Overview » Defining your tests » Running your tests on every commit » Monitoring and deployment » Playing with test data
  • 3. Defining your tests
  • 4. I download your code...
  • 5. Now I want to run your tests
  • 6. How do I do that?
  • 7. Ways to run Django tests » ./manage.py test <app> » ./manage.py test testapp » nosetests » setup.py test » run_tests.py » ...
  • 8. Python has even more
  • 9. We need a standard way to run Python tests
  • 10. setup.py test is the answer
  • 11. ...but it’s not ready
  • 12. setup.py test » Part of setuptools/distribute » Will be a part of distutils2 » Django has no support for this yet » We can lead standardization efforts
  • 13. Pony Build
  • 14. Pony Build » Started by Titus Brown » Super simple Continuous Integration Server » Interesting part is the client » github.com/ericholscher/pony_barn
  • 15. Define how to run tests in Python
  • 16. Full Example class PonyBuild(BaseBuild): def __init__(self): super(PonyBuild, self).__init__() self.name = "sphinx" self.required = ['nose'] self.repo_url = 'http://bitbucket.org/birkenfeld/sphinx/' def define_commands(self): self.commands = [ pony.HgClone(self.repo_url, egg=self.get_name()), pony.BuildCommand([self.context.python, 'setup.py', 'install']), pony.TestCommand([self.context.python, 'tests/run.py']), ]
  • 17. Supports standard ways of running tests out of the box
  • 18. Ships with support for » ./manage.py test <app> » ./manage.py test <testapp> » nose » setup.py test
  • 19. Simple Pony Build class PonyBuild(DjangoGitBuild): def __init__(self): super(PonyBuild, self).__init__() self.repo_url = "git://github.com/alex/django-filter.git" self.name = "django-filter" self.package_name = 'django_filters' self.installed_apps = ['django_filters', 'django_filters.tests']
  • 20. Lets you run your build in different environments
  • 21. Different Environments » Local Environment » Temporary Directory » Virtualenv
  • 22. At least ship with a shell script
  • 23. Document how to run tests
  • 24. Continuous Integration
  • 25. “Run your tests on every commit”
  • 26. A fancy wrapper around a shell script
  • 27. Without this, your tests are useless
  • 28. Make sure you setup.py install your package!
  • 29. --noinput is your friend
  • 30. Use Virtualenv
  • 31. Watch Titus’s Pycon Talk
  • 32. Three main options » Buildbot » Hudson » Devmason
  • 33. Hudson
  • 34. Hudson stats » Written in Java :x » Great plugin community » Super simple to setup » Becoming Defacto » Worthwhile if you have resources
  • 35. We use this at work and it kicks ass
  • 36. hudson.djangoproject.com
  • 37. Setting up your own
  • 38. Useful Capabilities » Coverage » XML Output » PyLint » Profiling
  • 39. Plugins » Hudson Cobertura Plugin » Hudson Violations Plugin » IRC Plugin
  • 40. pip install coverage (Compiles a C extension)
  • 41. Coverage.py coverage run manage.py test or coverage run 1.1.X/tests/runtests.py coverage xml -o 1.1.X/coverage.xml
  • 42. XML Output » Gareth Rushgrove’s django-test- extensions » Requires a subclassed TestRunner » github.com/garethr/django-test- extensions » github.com/danielfm/unittest-xml- reporting
  • 43. Example Build Bits from my_test_settings import * TEST_RUNNER = ‘test_extensions.test_runners.xmloutput’
  • 44. Nightly Tasks » pylint » profiling
  • 45. django-nose makes a lot of this easier
  • 46. Devmason
  • 47. Language Agnostic Build Reporting API
  • 48. <3 Piston
  • 49. devmason.com/docs
  • 50. Integrates with Github and Bitbucket
  • 51. Have simple API support for Python unittest and Pony Build
  • 52. Keep your builds in the Pony Barn
  • 53. “Free” for Python community
  • 54. Will hopefully help standardize tests
  • 55. github.com/ericholscher/ devmason-server
  • 56. github.com/ericholscher/ devmason-utils
  • 57. Monitoring & Deployment
  • 58. “Django Kong” » “Functional Testing” » “Devops” » Uses Twill: twill.idyll.org » github.com/ericholscher/django-kong
  • 59. Simple Example go {{ site.url }}/ code 200 find “Awesome” follow “Events” code 200 find “Today’s Events”
  • 60. Check to see if your code works on live sites
  • 61. Easily run your tests across all types of a site
  • 62. Catch errors users are having but never reporting
  • 63. Gives you a neat “feel” for your sites
  • 64. We check all of our sites every 3 minutes
  • 65. A neat hack allows you to test your site locally too!
  • 66. Stuff to do » Nagios integration » “Run levels” » Better aggregates timing data » Have tests live on disk/in VCS
  • 67. Run nightly automated tools to check your sites
  • 68. Patu
  • 69. Test Utils Crawler
  • 70. Get a Dashboard!
  • 71. Test Data
  • 72. My Pony
  • 73. We all run tests a lot.
  • 74. There is valuable data about our software there.
  • 75. CPANTS » Perl’s Effort of aggregating test data » Lots of pretty graphs
  • 76. Cpants pivots » Operating System » Architecture » Language Versions
  • 77. We need this for Python
  • 78. Devmason can serve as the reporting platform
  • 79. We have all the tech we need, just need to tie it together
  • 80. Things to Remember
  • 81. Document how to run your tests (in executable form)
  • 82. Get a CI server setup
  • 83. What you don’t measure you don’t improve
  • 84. Get some basic monitoring for your sites functionality
  • 85. URLS » github.com/ericholscher/django-kong » hudson-ci.org » github.com/ctb/pony-build » pypants.org » cpantesters.org » devmason.com » github.com/ericholscher/devmason-server » github.com/ericholscher/devmason-utils
  • 86. Questions? » eric@ericholscher.com » twitter.com/ericholscher