Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to White Box Testing: It’s Not Just for Developers Any More
Similar to White Box Testing: It’s Not Just for Developers Any More (20)
More from TechWell
More from TechWell (20)
Recently uploaded
Recently uploaded (20)
White Box Testing: It’s Not Just for Developers Any More
- 3. 5/10/16 1 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall 1 Unit Tes)ng: Not just for developers anymore Robert Vanderwall, Ul)mate So?ware Group June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Agenda • Current Trends • Test Pyramid • Code Patterns • Metrics • Adding Value • Wrap-up
- 4. 5/10/16 2 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Current Trends • Shift left • Death of Testing • Deep Collaboration June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Shift Left Many activities are moving earlier, including testing. • Testing begins at inception. • Testing is an integral part of the development process. Especially
- 5. 5/10/16 3 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Death of Testing Several companies have adopted a “No QA” approach • Engineers are becoming hybrids. • Generalists with area of specialization. • Developers do their own testing. • Ownership through to production. June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Deep Collaboration • Cross functional teams • Continuous conversations • Group ownership of quality
- 6. 5/10/16 4 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Quality Engineer • Understand ways to stay relevant • Enhance testing skills • Test with an understanding of what information developers would find useful June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Software Developer • Better appreciation for the value of exploratory testing • Boost ability to deliver value by collaborating well with testers • Improve ability to turn testing observations into useful ideas for unit testing
- 7. 5/10/16 5 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Test Pyramid • Most of the tests are unit tests • Fewer tests are integra)on tests • Least number of tests at system level June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Purpose • Unit tests are cheap and fast, they provide fast feedback. • Integration tests verify interactions between components. • System tests verify value is realized.
- 8. 5/10/16 6 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Inverted Pyramid • Many UI tests but few unit tests. • Automation effort without development support. • Lack of development practices for unit testing. June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Test Space Needle • Lots of unit tests, plenty of UI tests. • Almost no integration tests. • Overconfidence in unit testing. • Ends up with big-bang integration.
- 9. 5/10/16 7 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Coding Patterns • A few easy patterns • A few not so easy patterns • Pointers June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Easy Pattern - Sequence What could go wrong? • Operator may be wrong • z = x * y • Bad test x =2, y=2 • No discriminatory power def seq(x,y) : z = x + y return z
- 10. 5/10/16 8 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Test Pattern X Y Expected Output def seq(x,y) : z = x + y return z 0 0 7 0 0 9 6 5 Are these tests any good? What is the business need being sa)sfied? Total exemp)ons. these may be fine Total sales. these are not representa)ve. 3 8 Number, 0, > June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Test Pattern - combinations X Y Expected Output def seq(x,y) : z = x + y return z 0 0 7 0 0 9 6 5 What is the expected output? 0, 7, 9, 11, 11 ? Did you get the expecta)ons from the code? Could the tests ever fail? 3 8
- 11. 5/10/16 9 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Easy Pattern - Condition What could go wrong? • Relation may be wrong • x < 10, x >= 10 • Value may be wrong • x > 20 def condition (x) : if x > 10: z = x + 10 else: z = x -10 return z June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Test Pattern – Boundary tests X Expected Output -‐10 0 9 10 11 20 Edge -‐1, Edge, edge +1, mid-‐range def condition (x) : if x > 10: z = x + 10 else: z = x -10 return z
- 12. 5/10/16 10 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Test Pattern – Boundary tests X Y Expected Output 0 0 10 10 10 11 11 10 8 -‐8 -‐5 -‐6 def condition (x,y) : if x > y: z = x - y else: z = x + y return z Edge, edge +1, edge -‐1 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Easy Pattern - Loop What could go wrong? • Off-by-one • Didn’t iniPalize properly • Didn’t finalize properly def loop(L): for l in L: do_something(l) return results
- 13. 5/10/16 11 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Test Pattern – Loop L Expected Output empty 1 item 2 items Typical number of items Max number of items Max+1 items def loop(L): for l in L: do_something(l) return results June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Lessons • Test inputs should reflect expected uses. • Expected output should come from business function NOT from the code. • Try <, =, > for sequences. • Try edge-1, edge, edge+1, mid-range for boundary. • Try 0, 1,2, t, max. max+1 for loops.
- 14. 5/10/16 12 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall References https://msdn.microsoft.com/en-us/library/cc514239.aspx http://www.exampler.com/testing-com/test-patterns/patterns/ index.html June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Metrics If you can’t measure it, you can’t improve it – Peter Drucker • Code Coverage • Code Complexity • Mutation Score
- 15. 5/10/16 13 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Code Coverage Most IDEs have a way to measure coverage. Look for high coverage, esp: critical areas, complicated areas. Statement coverage first, then branch coverage. June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Code Coverage If a line of code is not executed, it is not tested. If a line of code is not tested, it may have a bug. Inspections help, a lot!
- 16. 5/10/16 14 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Code Coverage 1. def condition (items, y) : 2. for item in items: 3. if item.x > y: 4. z += a(item) 5. else: 6. z += b(item) 7. return z How many statements? How many tests to cover them? June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Code Coverage 1. def condition (items, y) : 2. for item in items: 3. if item.x > y: 4. z += a(item) 5. else: 6. z += b(item) 7. return z How many branches? How many tests to cover them? 1 2 3 4 6 7
- 17. 5/10/16 15 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Coverage caveats • I noPced that the good developers drink lots of coffee. • SO • I have all developers drink 6 cups a day. • Surely, I will get beber code, right? • NO: CorrelaPon is not causality. June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Code Coverage Caution Low coverage tells you that some code has not been tested. High coverage tells you nothing! Therefore, it should be a guidepost, NOT be a goal. hbp://www.exampler.com/tesPng-‐com/wriPngs/coverage.pdf
- 18. 5/10/16 16 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Code Complexity Many IDEs can measure Cyclomatic Complexity. 1-5 : Simple and easily understandable. 6-15: Rather complicated, better test it a lot. > 15: Quite complicated, probably hard to test and likely a haven for many bugs. June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Mutation Testing If I change the code, does a test fail? If not, then there are ways the code can be wrong that the tests will not find. A Few tools available, but manually altering the code works well, esp. for JS.
- 19. 5/10/16 17 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall X Y Expected Output 0 0 10 10 10 11 11 10 8 -‐8 -‐5 -‐6 def condition (x,y) : if x > y: z = x - y else: z = x + y return z June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Complexity Cautions If every module has very low complexity, where is the complex business logic? Not all statements are equally complex. A complex linq query may show CC=1.
- 20. 5/10/16 18 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall CRISP tests C – Clear purpose R – Repeatable I – Independent S – Simple P – Performant DRY – Don’t repeat yourself Tests may have duplicate code June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Adding Value In a left shifted highly collaborative environment, how can we add value? We just saw an overview of the test pyramid and a few testing patterns. How can we make that work for us?
- 21. 5/10/16 19 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Leverage Left Shift • Review unit tests for coverage, patterns, etc. • Pull together the tester-facing discipline of Exploratory testing and the developer-facing discipline of Unit testing. • Provide insights into the product and code through exploring and leverage these insights via unit tesPng. June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Leverage collaboration • Become more helpful for developers: • CommunicaPng bugs found • AND ideas from exploraPon that could be used to improve unit tests
- 22. 5/10/16 20 June 5–10, 2016 | Las Vegas, NV Robert Vanderwall Leverage Automation • Combine the forces of machines and humans. • Machines are parPcularly well suited for exploring large computaPonal space. • What they can’t do is have any insight or creaPve ideas. June 5–10, 2016 | Las Vegas, NV Robert Vanderwall