Downloaded 47 times
Uploaded bySteve Poole
Ten Simple Rules for Writing Great Testcases @ Devoxx US
We've all been there: debugging problems in a test case and silently screaming into the dark. (Sometimes not even silently.) Poor test case design can cost you significant time and effort let alone impact the quality of your application or product. Testing is vitally important, but so is having a test suite you can use effectively and can rely on. This session will take you through the top ten rules for writing effective and reliable testcases. The new kids on the block such as Cloud or Docker and general "Infrastructure as Code" style solutions may make you believe old rules are just old. This talk will make you think again. Knowledge gained from personal experience is always best. Learn from these old masters how to design great test cases and maybe you'll never have to visit the dark side again.
More Related Content
More from Steve Poole
Ten Simple Rules for Writing Great Testcases @ Devoxx US
- 1. #DevoxxUS @spoole167 @stuartmarks TENSIMPLE RULES FOR WRITING GREAT TESTCASES
- 2. #DevoxxUS @spoole167 @stuartmarks StevePoole : IBM JVM Developer DevOps practitioner Developer Advocate Stuart Marks : Oracle Principal MTS Java / OpenJDK Core Libraries
- 3. #DevoxxUS @spoole167 @stuartmarks “Iget paid for code that works, not for tests” How to maximize your effort and protect your investment in tests and testing now with added Cloud
- 4. #DevoxxUS @spoole167 @stuartmarks 1.Think before you act 2. Make your tests understandable 3. Keep your tests “small and simple” 4. Test one thing only 5. Fast tests only 6. Absolute repeatability 7. Independent tests only 8. Provide diagnostic data on failure 9. No hard-coding of your environment 10.No extraneous output
- 5. #DevoxxUS @spoole167 @stuartmarks 1.Think before you act What are you testing? Why are you testing? Planhttps://www.flickr.com/photos/ayurvedicmedicines/
- 6. #DevoxxUS @spoole167 @stuartmarks 2.Make your tests understandable Comments Expected behaviour Aid debug Diagnostics https://www.flickr.com/photos/83633410@N07/
- 7. #DevoxxUS @spoole167 @stuartmarks 3.Keep your tests “small and simple” Separate test logic / setup Much easier to debug Use setup / teardown https://www.flickr.com/photos/9266144@N02/
- 8. #DevoxxUS @spoole167 @stuartmarks 4.Test one thing only One scenario per test Enables fast debug Obvious why test failed https://www.flickr.com/photos/ryantron/
- 9. #DevoxxUS @spoole167 @stuartmarks 5.Fast tests only Run unit tests often as possible Maintain quality bar Quick results https://www.flickr.com/photos/mcleod/
- 10. #DevoxxUS @spoole167 @stuartmarks 6.Absolute repeatability Non-deterministic tests are a headache Fix intermittent tests immediately No value, waste of resource Must trust all tests https://www.flickr.com/photos/fdecomite/
- 11. #DevoxxUS @spoole167 @stuartmarks 7.Independent tests only Must run in any order Run subset, faster results No dependencies https://www.flickr.com/photos/sheila_sund/
- 12. #DevoxxUS @spoole167 @stuartmarks 8.Provide diagnostic data on failure Use message in asserts Make it simple to debug Reference input data Record test environment info https://www.flickr.com/photos/davidbaker/
- 13. #DevoxxUS @spoole167 @stuartmarks 9.No hard-coding of your environment No ports, IP addresses, data files, databases Use config files, system properties or mock objects Portable tests https://www.flickr.com/photos/pug50/
- 14. #DevoxxUS @spoole167 @stuartmarks 10.No extraneous output A passing test is a silent test Too much output = confusion Use option, config file to turn on debug, save output https://www.flickr.com/photos/3-bs/
- 15. #DevoxxUS @spoole167 @stuartmarks Piraterules Guidelines only Some may seen obvious. Some may seem pedantic Until you inherit a testsuite… Thank you. https://www.flickr.com/photos/gapic/
- 16. © 2017 IBMCorporation Mission Badge #6: SMS Text Your mission should you choose to accept it…. Join us at the IBM Booth for hands-on labs, demos, games and talk to our developers. Text Mission2017 to 41411 to get the booth giveaway and learn more about all the IBM sessions & speakers. Enter the raffle by completing missions for a chance to win • a Drone • TJBot Kit • VR glasses
Editor's Notes
- #3 Questions for the end or on twitter..
- #4 Steve: To say why we are doing. Intro Why do we write tests. I don’t need to my code is good enough. I only get paid for code. Stuart: Platitudes -> things are ‘better’ if we write tests. With not much complexity there was more knowledge in the code and tests than in my head. 200 lines of code Your brain is not big enough to keep even a simple code in your head.
- #5 Pirate Rules (guidelines only) Can we agree on rules vs guidelines ? This list is the top ish(10) from professional testers and developers. Most of our pain has come from from personal experience in breaking these rules.. Our experiences only. Valueable. Its not exclusive, just our opnion. And we don’t have time for more.. Non of these are unimportant
- #6 Steve – intro and substance --- Your first job is to capture the expected behaviour of your system so you can continue to show you have not broken bevaiour.. You cannot test quality in But without tests you cannot assert ‘quality’ you only have some much time to capture your systems behavior in tests - don’t waste it -- Stuart – your opinion “on testing in relation to quality” How do you measure this? Test coverage -> Don’t just test getters and setters Its not valueless but unless you’re looking at conformance don’t get too bent out of shape about coverage. Improving Test coveage without bugs is not necessary a good thing. JDK has jtreg. SQE has separate test suite (closed for no good reason) too much effort to open source the framework and tests. Is adding more tests to extend coverage a good idea? Not necessary - time spend on arguing how the code is supposed to work Do new tests have bugs as often as code has bugs?
- #7 Stuart: tests are an equally important part of a code base as the production code. Tests need to be maintained and enhanced alongside the production code, not left as an afterthought. Q: What happens if your tests are not understandable? Should tests be easier to understand than the code their testing? Tests should be coded and commented well. A test suite is the inverse of the code. Its codifing expected behaviour so write it down what your think the behavior is. When and a test fails you don’t want to be spending time figuring out if it’s the test. How would you report a bug if you can’t work out what the test.
- #8 Steve: tests should have a well defined setup Can you find the actual line of testing in a large test suite Why are we saying this - how often have you had to step a 1000times Or debug a nested loop inside a nested loop? Or inserted code to allow you to set a break point. If you have to hack the test its too big clean separate between getting to to the actual cost. Q: “you are in full agreement?”
- #9 Stuart: setup – operation – verification – teardown Don’t do multiple operations; they can spoil the fixture for subsequent op/verify steps. Separate into multiple tests. Q: “do you agree?”
- #10 Steve: Testing is domain / time context We’d all like all our tests to run all the time. We can’t do that so when backing off chose wisely. Running the whole testsuite is useful. Long tests encourage you to not run tests. You let more bugs through Even if part of a CI run.. Long running test suites are evil. They encourage corner cutting. The make you think about not running tests… Q: “Stuart – what happens if your tests are not fast?” War story about test suite run monthly
- #11 Stuart: An ideal to strive for but possibly impossible to achieve in practice. Nonetheless v important! (ask for show of hands) Noise level of intermittent test failures makes result analysis harder, masks real failures. Q: “Steve how does this work in a cloud environment?” Cloud: reliable system built from unreliable components. This manifests itself in intermittent test failures. How to distinguish "valid" failure (failure that can be handled by the larger system) vs an "invalid" failure? Do we care? "Offensive programming" – deliberate injection of failures. Useful for system tests. Random without a seed. Sleeping to wait for stuff to happen. Repeatability with timeouts is difficult. “Chaos Monkeys”
- #12 Steve: War story - junit test ordering. Exposed unexpected test interdependences… I want to one day run one of these tests singualry for debug purposes. I don’t want to run 50 others first! Run your tests in random order? Q: “ has this ever happened to you? Stuart: Yes. Don't write tests that have persistent side effects. CORBA test war story.
- #13 Stuart: Reduce debugging time. Exception stack traces usually help here, but not always. Consider: asserting that right exception was thrown at the right time. Timeout war story. Q: what’s your experience with this? Steve : War story FFDC works for me syndrome + Cloud: Remote tests are difficult remove env is always different and un reproducable locally..
- #14 Steve: War story “the App server test env and 1yr” + Cloud needs you not to do this.. Every instance could be on a different box, somewhere else in the world Reduce opportunities to be effected by using mock objects. Q: “can you think of any reasons why you would want to hard code your environment?” Stuart: Barely. Hard to avoid hard-coding 100%. Example: test that the default port is 80.
- #15 Stuart: Essential if you have 10,000 unit tests. Even saying ”passed” can generate too much output. But for complex system tests, esp with timeouts, sometimes verbose logging is necessary. Q: You like verbose output don’t you Steve? Steve: if you have too much output - > you invest in systems to analyse the log and look for the real ‘truth’
- #16 Steve: Just like the movie says – these are all guidelines. We think you should think before you ignore the rules. Wait until you inherit your first test suite.. Q: Stuart –any closing thoughts?