Test Driven


Published on

Want to know the case for Test-Driven Development? Want to know style tips and gotchas for Testing and TDD? Let Alex Chaffee, former Mad Scientist at Pivotal Labs, tell you everything you didn't know you didn't know about testing.

Published in: Technology, Business
1 Comment
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Test Driven

  1. 1. <ul><li>by Alex Chaffee </li></ul><ul><ul><li>alex @ pivotallabs.com </li></ul></ul><ul><ul><li>Pivotal Labs </li></ul></ul>Test-Driven
  2. 2. <ul><li>Developers or QA Engineers </li></ul><ul><li>Familiar with JUnit </li></ul><ul><li>Want more detail on Automated Testing in general </li></ul><ul><li>Want to know the case for Test-Driven Development </li></ul><ul><li>Want to know style tips and gotchas for Testing and TDD </li></ul>Intended Audience
  3. 3. Part I: The Blank Page
  4. 4. Let's test-drive a utility class...
  5. 5. Red-Green-Refactor
  6. 6. <ul><li>Arrange (set up preconditions) </li></ul><ul><li>Act (call production code) </li></ul><ul><li>Assert (check the results) </li></ul>3A
  7. 7. <ul><li>The heart of a unit test </li></ul><ul><li>An Assert is a declaration of truth </li></ul><ul><li>Failed assert -> false (incorrect) behavior </li></ul><ul><li>“assert your postconditions” </li></ul><ul><li>Example: </li></ul><ul><ul><ul><li>Set set = new MySet(); </li></ul></ul></ul><ul><ul><ul><li>set.add(&quot;Ice Cream&quot;); </li></ul></ul></ul><ul><ul><ul><li>assertTrue(set.contains(&quot;Ice Cream&quot;)); </li></ul></ul></ul>Assert
  8. 8. <ul><li>Don't be over-ambitious </li></ul><ul><li>Each test -- especially each new test -- should add one brick to the wall of knowledge </li></ul><ul><ul><li>Code:Brick :: Test:Mortar </li></ul></ul><ul><li>Pick tests (features) in order of growth </li></ul>One Step At A Time
  9. 9. <ul><li>A great first test to write </li></ul><ul><li>Input and output are trivial </li></ul><ul><li>Helps you focus on infrastructure and API </li></ul>The Null Test
  10. 10. <ul><li>When you get stuck on a test, try starting with the assertion(s) and then work your way backwards to the setup </li></ul><ul><li>Start with the assert </li></ul><ul><ul><ul><li>assertTrue(set.contains(“alex”)); </li></ul></ul></ul><ul><li>Then add the code above it </li></ul><ul><ul><ul><li>Set set = new MySet(); </li></ul></ul></ul><ul><ul><ul><li>set.add(“alex”); </li></ul></ul></ul><ul><li>Helps focus on goal </li></ul>Assert First
  11. 11. <ul><li>Start with hardcoded results and wait until later tests to force them to become real </li></ul>Fake It 'Til You Make It
  12. 12. <ul><li>Make the code abstract only when you have two or more examples </li></ul>Triangulate To Abstraction public void testSum() { assertEquals(4, plus(3,1)); } --- int plus(int x, y) { return 4; } public void testSum() { assertEquals(4, plus(3,1)); assertEquals(5, plus(3,2)); } --- int plus(int x, y) { return x + y; }
  13. 13. <ul><li>Before you begin, make a TODO list </li></ul><ul><li>Write down a bunch of operations </li></ul><ul><li>For each operation, list the null test and some others </li></ul><ul><li>Also put down refactorings that come to mind </li></ul><ul><li>Why not write all the tests in code first? </li></ul><ul><ul><li>Could box you in </li></ul></ul><ul><ul><li>Interferes with red-green-refactor </li></ul></ul>Test List
  14. 14. <ul><li>aka Don't Be Stupid </li></ul><ul><li>If you really, really, honestly know the right way to implement it, then write it that way </li></ul><ul><li>But keep track of how many times your &quot;obvious implementation&quot; was broken or untested </li></ul><ul><li>Edge cases, off-by-one errors, null handling... all deserve tests and often the Obvious Implementation is not covered </li></ul>Obvious Implementation
  15. 15. Part II: Testing Philosophy
  16. 16. <ul><li>Unit </li></ul><ul><li>Integration </li></ul><ul><li>Acceptance </li></ul><ul><li>QA </li></ul><ul><li>UI </li></ul><ul><li>Performance </li></ul><ul><li>Monitoring </li></ul>Automated Testing Layers
  17. 17. <ul><li>Automated </li></ul><ul><li>Isolated </li></ul><ul><li>Repeatable </li></ul><ul><li>Fast </li></ul><ul><li>Easy to write and maintain </li></ul><ul><li>Focused </li></ul><ul><li>Easy to read </li></ul><ul><li>Robust (opposite: Brittle) </li></ul>A Good Test Is...
  18. 18. <ul><li>Someone should be able to understand your class by reading the tests </li></ul><ul><li>Live documentation (better than dead trees) </li></ul><ul><li>“ Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” </li></ul><ul><li>– Martin Fowler </li></ul>Tests are “Executable Specifications”
  19. 19. Why do you test?
  20. 20. <ul><li>Prevent bugs </li></ul><ul><li>Regress bugs (&quot;bug repellant&quot;) </li></ul><ul><li>Localize defects </li></ul><ul><li>Understand design </li></ul><ul><li>Document (or specify) design </li></ul><ul><li>Improve design </li></ul><ul><li>Support refactorings </li></ul><ul><li>Enable experimentation and change </li></ul><ul><li>Confidence </li></ul><ul><li>Catch errors the language can't </li></ul><ul><li>Long-term sustainability and maintainability </li></ul>Why do you test?
  21. 21. When do you test?
  22. 22. <ul><li>Before checkin </li></ul><ul><li>After update </li></ul><ul><li>Before deploy </li></ul><ul><li>While coding </li></ul><ul><li>In the background </li></ul>When do you test?
  23. 23. <ul><li>All the time </li></ul>When do you test?
  24. 24. <ul><li>Never </li></ul><ul><li>After coding </li></ul><ul><li>During coding </li></ul><ul><li>Before coding </li></ul>When do you write tests?
  25. 25. Why test first?
  26. 26. <ul><li>Gets tests written </li></ul><ul><ul><li>Easier than retrofitting tests onto an existing system </li></ul></ul><ul><ul><li>Guarantees 100% test coverage </li></ul></ul><ul><ul><li>In practice, you never have time after the code is written, but you always have the time before </li></ul></ul><ul><ul><ul><li>Go figure :-) </li></ul></ul></ul>Why test first?
  27. 27. Why test first? <ul><li>Reduces scope of production code </li></ul><ul><ul><li>Less scope -> less work </li></ul></ul><ul><li>Proves that your objects have usable interfaces </li></ul><ul><ul><li>more useful methods and fewer useless ones </li></ul></ul><ul><li>Guarantees testability </li></ul><ul><ul><li>Test-last code is often hard to test </li></ul></ul><ul><li>Sustainable Feature Velocity </li></ul>
  28. 28. <ul><li>Think of tests as examples or specifications </li></ul><ul><li>One trick: write code in a test class, then extract into the production class </li></ul><ul><li>&quot;If you can't write a test, then you don't know what the code should do. And what business do you have writing code in the first place when you can't say what it's supposed to do?&quot; - Rob Mee </li></ul>How can you write tests for code that doesn't exist?
  29. 29. <ul><li>Simple Rule: </li></ul><ul><ul><li>Test everything that could possibly break </li></ul></ul><ul><li>Depends on definitions of “everything” and “possibly” </li></ul><ul><ul><li>(and “break”) </li></ul></ul><ul><li>This means, don’t test things that couldn’t possibly break </li></ul><ul><ul><li>E.g. Getters and Setters </li></ul></ul><ul><ul><li>Unless you think they could fail! </li></ul></ul><ul><ul><li>Better safe than sorry </li></ul></ul><ul><li>Full-Range Testing (positive, negative, boundary, null, exception) </li></ul>What to test?
  30. 30. <ul><li>Personal judgement, skill, experience </li></ul><ul><li>Usually, you start by testing too little, then you let a bug through </li></ul><ul><li>Then you start testing a lot more, then you gradually test less and less, until you let a bug through </li></ul><ul><li>Then you start testing too much again :-) </li></ul><ul><li>Eventually you reach homeostasis </li></ul>How much to test?
  31. 31. <ul><li>Not too big, not too small </li></ul><ul><li>Same concept as high coherence, low coupling </li></ul>Test for &quot;essential complexity&quot;
  32. 32. <ul><li>Write the tests first </li></ul><ul><li>Design for testability </li></ul><ul><li>Use the front door first </li></ul><ul><li>Communicate intent </li></ul><ul><li>Don't modify the SUT </li></ul><ul><li>Keep tests independent </li></ul><ul><li>Isolate the SUT </li></ul><ul><li>Minimize test overlap </li></ul><ul><li>Minimize untestable code </li></ul><ul><li>Keep test logic out of production code </li></ul><ul><li>Verify one condition per test </li></ul><ul><li>Test separate concerns separately </li></ul><ul><li>Ensure commensurate effort and responsibility </li></ul>Meszaros' Principles of Test Automation
  33. 33. <ul><li>Every time you write code, you write tests that exercise it </li></ul><ul><ul><li>That means that if you change the code, and the tests break, you must either </li></ul></ul><ul><ul><ul><li>Change the tests to match the new spec </li></ul></ul></ul><ul><ul><ul><li>Change the code to meet the old spec </li></ul></ul></ul><ul><li>Do not remove the failing tests </li></ul><ul><ul><li>Unless they no longer apply to the new code’s design or API </li></ul></ul><ul><ul><li>Do not work around the failing tests </li></ul></ul><ul><li>Test code is not &quot;wasted&quot; or &quot;extra&quot; -- tests are first-class citizens </li></ul><ul><ul><li>If you feel like they're too much work, examine your process </li></ul></ul><ul><ul><li>Maybe you're writing the wrong tests, or your tests are too brittle, or they're not refactored enough </li></ul></ul>Tests Are An Extension of Code
  34. 34. <ul><li>It forces you to really understand the code </li></ul><ul><li>It forces you to really understand the tests </li></ul><ul><li>It forces you to create code that is truly reusable and modular and testable </li></ul><ul><ul><li>“put your money where your mouth is” </li></ul></ul><ul><li>These forces drive you to keep your code and your tests simple and easy to understand </li></ul>Unit Testing Is Hard
  35. 35. <ul><li>Need to spend time on infrastructure, fixtures, getting comfortable with TDD </li></ul><ul><li>Business case for TDD: sustainable velocity </li></ul><ul><ul><li>for feature velocity, stabilty > early oomph </li></ul></ul><ul><li>Famous graph </li></ul>Test-Driving Is Slower At First
  36. 36. <ul><li>Test-Driven Development </li></ul><ul><ul><li>Good old-fashioned coding, now with tests! </li></ul></ul><ul><li>Test-Driven Design </li></ul><ul><ul><li>Free your mind and the code will follow </li></ul></ul><ul><li>Quite a lot of overlap, but worth keeping difference in mind </li></ul><ul><li>Lots of XP gurus are all about the Zen, but you don't need to buy into that </li></ul><ul><ul><li>But it's actually pretty cool to try Zen Testing </li></ul></ul>Two D's
  37. 37. Part III: Advanced Techniques
  38. 38. <ul><li>Positive Tests </li></ul><ul><li>exercise normal conditions (“sunny day scenarios”) </li></ul><ul><li>E.g. Verify that after adding an element to a set, that element exists in the set </li></ul><ul><li>Negative Tests </li></ul><ul><li>Exercise failure conditions (“rainy day scenarios”) </li></ul><ul><li>E.g. verify that trying to remove an element from an empty set throws an exception </li></ul><ul><li>Boundary Conditions </li></ul><ul><li>Exercise the limits of the system (“cloudy day”) </li></ul><ul><li>E.g. adding the maximum number of elements to a set </li></ul><ul><li>E.g. test 0, -1, maximum, max+1 </li></ul>Full Range Testing
  39. 39. <ul><li>instead of SetTest.testEmpty </li></ul><ul><li>how about SetTest.testShouldHaveSizeZeroWhenEmpty </li></ul><ul><li>or EmptySetTest.testHasSizeZero </li></ul>Verbose Test Naming
  40. 40. <ul><li>Optional first parameter to JUnit asserts is &quot;message&quot; </li></ul><ul><li>Assertion messages can be confusing </li></ul><ul><li>Example: </li></ul><ul><li>assertEquals(“set is empty”, set.isEmpty()); </li></ul><ul><li>Does it mean “the set must be empty” or “the test is failing because the set is empty”? </li></ul><ul><li>Solution: should statements </li></ul><ul><li>assertEquals(“set should be empty”, set.isEmpty()) </li></ul><ul><li>or even better: </li></ul><ul><li>assertEquals(&quot;a newly-created set should be empty&quot;, set.isEmpty()) </li></ul>Should Statements
  41. 41. <ul><li>Philosophy: a test is a valid client of an object </li></ul><ul><li>Therefore don't be ashamed of adding a method just because it would make a test easier to write </li></ul><ul><li>Used -> Useful </li></ul><ul><li>Remember, tests are examples of use </li></ul>Test-Only Methods
  42. 42. <ul><li>Spend time refactoring your tests </li></ul><ul><li>It'll pay off later, when writing new tests or extending/debugging old ones </li></ul><ul><li>Refactor for readability, not necessarily for removing all duplication </li></ul><ul><ul><li>Different priorities than for production code </li></ul></ul><ul><li>Extract methods </li></ul><ul><li>Shorter lines </li></ul><ul><li>Break up long tests (scenario tests) into several short tests (feature tests) </li></ul><ul><li>One technique: &quot;Refactor production code on green, Refactor test code on red.&quot; </li></ul><ul><ul><li>for complex cases, break the code, make sure the refactored tests still reveal the breakage, then fix it </li></ul></ul>Refactor Test Code
  43. 43. <ul><li>assertEquals(86400, new Day().getSeconds()) </li></ul><ul><li>vs. </li></ul><ul><li>assertEquals(60 * 60 * 24, new Day().getSeconds()) </li></ul><ul><li>vs. </li></ul><ul><li>secondsPerMinute = 60 </li></ul><ul><li>minutesPerHour = 60 </li></ul><ul><li>hoursPerDay = 24 </li></ul><ul><li>assertEquals(secondsPerMinute * minutesPerHour * hoursPerDay, new Day().getSeconds()) </li></ul>Evident Data
  44. 44. <ul><li>Problem: several axes of variability, combinatorial explosion </li></ul><ul><li>Solution: Loop through a matrix of data in your test, call a &quot;check&quot; function on each row </li></ul>Matrix Tests
  45. 45. <ul><li>aka &quot;Golden Data Tests&quot; </li></ul><ul><li>Grab the complete output of a routine, put it into the test </li></ul><ul><li>Not amenable to test-driven development </li></ul><ul><li>Effective for large or risky refactorings </li></ul><ul><li>Quite brittle, so often thrown away after the refactoring is done </li></ul>Characterization Tests
  46. 46. <ul><li>public void testUnknownCountry() { </li></ul><ul><li>try { </li></ul><ul><li>currencyConverter.getRate(&quot;Snozistan&quot;); </li></ul><ul><li>fail(&quot;Should have thrown an exception for unknown country&quot;); </li></ul><ul><li>} catch (UnknownCountryException e) { </li></ul><ul><li>// ok </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>Exception Tests
  47. 47. <ul><li>A pair's job is to keep you focused </li></ul><ul><li>&quot;Wait, let's write a test first.&quot; </li></ul><ul><li>&quot;Wait, let's refactor first.&quot; </li></ul><ul><li>&quot;Wait, let's discuss this.&quot; </li></ul><ul><li>&quot;Can I drive?&quot; </li></ul>Pair Programming
  48. 48. <ul><li>One pair writes a test </li></ul><ul><li>The other pair makes it pass and writes the next test </li></ul><ul><li>Repeat </li></ul><ul><li>Good way to get out of a rut, or cure a keyboard hog </li></ul>Ping-Pong Pairing
  49. 49. <ul><li>When a defect is reported... </li></ul><ul><ul><li>The first step is to write a (failing) test that reproduces the bug </li></ul></ul><ul><ul><li>Fix the bug by writing code to make the test run successfully </li></ul></ul><ul><ul><li>Verify the bug in the running application </li></ul></ul><ul><ul><li>Add the bug test to the automated suite </li></ul></ul><ul><ul><li>Check in the bugfix code and test </li></ul></ul><ul><ul><li>Now it’s always run – instant regression test! </li></ul></ul><ul><li>&quot;Regression tests are test that you would have written originallly.&quot; - Kent Beck </li></ul><ul><li>May also want to write a failing Acceptance Test, but that's optional -- you definitely want a failing unit test </li></ul>Regression Test
  50. 50. <ul><li>Often the best thing to do is throw away your work and start again </li></ul>Do Over
  51. 51. <ul><li>At the end of the day, write a failing test and leave it there for tomorrow </li></ul><ul><li>Based on writer's trick: start a sentence and leave it unfinished </li></ul>Leave One For Tomorrow
  52. 52. <ul><li>Tests are only valuable if they're run all the time </li></ul><ul><li>If they're slow, people will not want to run them all the time </li></ul><ul><li>So keep them fast! </li></ul><ul><li>Difficult quest, but worth it </li></ul><ul><li>Don’t get stuck in molasses! </li></ul><ul><ul><li>Refactor your code to be easier to write fast tests on </li></ul></ul><ul><ul><li>Replace slow tests with (one or more) fast tests that cover the same area </li></ul></ul>The Need For Speed
  53. 53. Retrofitting <ul><li>What to do when you have an existing untested codebase? </li></ul><ul><li>Start small </li></ul><ul><ul><li>Write one test, make it pass, check it in </li></ul></ul><ul><ul><li>Write tests for all new code </li></ul></ul><ul><ul><li>Write tests for all new bugs </li></ul></ul><ul><ul><li>Write tests before attempting refactoring </li></ul></ul><ul><li>Usually easier to write characterization tests (UI/integration) </li></ul><ul><ul><li>But don’t fall into the slow test trap </li></ul></ul>
  54. 54. <ul><li>Any time all the tests are green, you can check in </li></ul><ul><li>Run all the tests all the time </li></ul><ul><li>Don’t check in until all tests pass </li></ul><ul><li>If you broke “someone else’s” tests, you are responsible for fixing “their” code </li></ul><ul><li>Remember, they are in the room, so go get them if you need help </li></ul><ul><li>Learn to Love the Orb </li></ul><ul><ul><li>ccmenu </li></ul></ul>Continuous Integration
  55. 55. <ul><li>Suites are a pain to maintain </li></ul><ul><li>Write code to automatically scan for tests and run them together </li></ul><ul><li>Possible to do in JUnit, but annoying </li></ul>Automatic Suites
  56. 56. <ul><li>Matter of preference </li></ul><ul><li>Both are useful at times </li></ul>Outside-in vs. Inside-out
  57. 57. <ul><li>Start with domain objects </li></ul><ul><li>Next layer of tests </li></ul>Inside-out
  58. 58. <ul><li>Start with customer story or user interface </li></ul><ul><li>Makes you think like a user </li></ul><ul><li>Tests capture these requirements </li></ul><ul><li>Lower layers implemented with Test Doubles (mocks) </li></ul><ul><ul><li>After you're done, either replace mocks with real objects, or leave them there (perhaps at higher maintenance cost) </li></ul></ul>Outside-in
  59. 59. <ul><li>Write a bunch of UI-level tests </li></ul><ul><li>Leave them there while you test-drive inside-out </li></ul><ul><li>When they all pass, you're done </li></ul>Outside-in design, inside-out development
  60. 60. <ul><li>A Test Double replaces the &quot;real&quot; instance of an object used by the production code with something suitable for the currently running test, but with the same interface </li></ul><ul><li>Stubs </li></ul><ul><ul><li>Hard-coded values </li></ul></ul><ul><li>Mocks </li></ul><ul><ul><li>Pre-programmed with expectations </li></ul></ul><ul><ul><li>Fail-fast </li></ul></ul><ul><ul><li>Test Doubles in general are often called Mock Objects, so be careful about terminology </li></ul></ul><ul><li>Fakes </li></ul><ul><ul><li>Can store values across calls, but don't really do what the live object would do </li></ul></ul><ul><ul><li>E.g. in-memory database </li></ul></ul>Test Doubles
  61. 61. More Test Doubles <ul><li>Spies </li></ul><ul><ul><li>Remember what methods were called with what values </li></ul></ul><ul><ul><li>Tests can inspect these lists after code returns </li></ul></ul><ul><li>Saboteurs </li></ul><ul><ul><li>Blow up in ways that would be difficult to make happen for real </li></ul></ul><ul><ul><li>To test what would happen when, e.g., the database goes away, or the disk is full </li></ul></ul><ul><li>Self Shunt </li></ul><ul><ul><li>The test itself declares methods or classes implementing the above, and passes in a pointer to itself </li></ul></ul>
  62. 62. <ul><li>Two types: </li></ul><ul><ul><li>DI Frameworks </li></ul></ul><ul><ul><li>Complete Construction </li></ul></ul><ul><ul><ul><li>This is the one I'm talking about </li></ul></ul></ul><ul><ul><ul><li>Pass in dependencies to the constructor (or, if necessary, to setters) </li></ul></ul></ul><ul><li>An object under test will receive references to all external services </li></ul><ul><li>Allows tests to inject Test Doubles at will </li></ul><ul><li>Forces objects to be isolated </li></ul><ul><li>Example: TBD </li></ul>Dependency Injection
  63. 63. <ul><li>Changes the language of tests to emphasize that they're specifications or examples </li></ul><ul><li>Replaces &quot;assert&quot; with &quot;should&quot; </li></ul>EDD/BDD (specs)
  64. 64. <ul><li>A natural progression of refactoring your test data </li></ul><ul><ul><li>literals </li></ul></ul><ul><ul><li>constants </li></ul></ul><ul><ul><li>local variables </li></ul></ul><ul><ul><li>instance variables (defined in setUp()) </li></ul></ul><ul><ul><li>creation methods </li></ul></ul><ul><ul><li>parameterized creation methods or objects (&quot;object mothers&quot;) </li></ul></ul><ul><li>Other patterns </li></ul><ul><ul><li>test objects / graphs (&quot;fixtures&quot; or &quot;cast of characters&quot; or &quot;menagerie&quot;) </li></ul></ul><ul><ul><li>external fixture files </li></ul></ul>Fixtures and Object Mothers
  65. 65. Mock Clock
  66. 66. Part IV: Test-Driving UI
  67. 67. Part V: Q&A
  68. 68. Thanks!