Principles Before Practices: Transform Your Testing by Understanding Key Concepts


Published on

It’s one thing to be exposed to new techniques from conferences and training courses, but it’s quite another thing to apply them in real life. A major reason is that people tend to focus on learning the technique without first grasping the underlying principles. Basic testing principles, such as the pesticide paradox of software defects and defect clustering, have been known for many years. Other principles, such as “Test automation is not automatic” and “Not every software failure is a defect,” are learned by experience. Once you grasp the principle, particular techniques become more applicable and extensible. However, principles take time to learn and much practice to apply well. Randy Rice explains why true learning and application are not instant and what it takes to really absorb what we learn. Randy shows how two specific techniques—pairwise testing and risk-based testing—can be misapplied unless the key concepts are first understood. Leave knowing how to build your own set of software testing principles that can be applied in many contexts

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Principles Before Practices: Transform Your Testing by Understanding Key Concepts

  1. 1. K1 Keynote 5/7/2014 8:30:00 AM Principles Before Practices: Transform Your Testing by Understanding Key Concepts Presented by: Randy Rice Rice Consulting Services, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ ∙
  2. 2. Randy Rice Rice Consulting Services, Inc. A leading author, speaker, and consultant with more than thirty years of experience in the field of software testing and software quality, Randy Rice has worked with organizations worldwide to improve the quality of their information systems and optimize their testing processes. He is coauthor (with William E. Perry) of Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems. Randy is an officer of the American Software Testing Qualifications Board (ASTQB). Founder, principal consultant, and trainer at Rice Consulting Services, Randy can be contacted at where he publishes articles, newsletters, and other content about software testing and software quality. Visit Randy’s blog.
  3. 3. 4/29/2014 1 PRINCIPLES BEFORE PRACTICES HOW TO TRANSFORM YOUR TESTING PRACTICES WITH BASIC AND ADVANCED PRINCIPLES RANDALL W. RICE, CTAL RICE CONSULTING SERVICES, INC. © 2013, Rice Consulting Services, Inc. 2 A LITTLE BACKGROUND • Many years ago, I heard a great analogy about conveying and applying knowledge. • If I were to teach someone how to wash dishes, I could: • Teach all the variations needed for every type of dish, cookware and utensil, or • I could teach principles and let the person adapt the methods to whatever they needed to wash.
  4. 4. 4/29/2014 2 3 THINGS LIKE… • Rinse off the big foodstuff first. • Save the really messy dishes until the end so you don't get everything else in the sink messy too. • Use hot water, but not too hot or else you will scald yourself. • Be careful with sharp knives in sudsy water 4 SIMILARLY, IN TESTING… • Take some sample tests early and find where the big problem areas seem to be. • Don't test the really complex areas at first. Get your bearings first. • Have strong tests, but if you make every test strong, you may not have time to finish. • Early testing is good, expect when the thing you are testing isn't ready even for early testing. • Test automation is not automatic.
  5. 5. 4/29/2014 3 5 COMMON QUESTIONS • Which tools do we use in certain situations? • Which techniques are “best” for a project? • How do we adjust tools and techniques? 6 WHAT WE WANT No Knowledge Mastery
  6. 6. 4/29/2014 4 7 REALISTIC VIEW No Knowledge Mastery “The Dip” Years 8 TESTING IS VERY NUANCED • There are small to large variables on any project that require you to: • Select certain techniques • Apply a technique differently • In essence, all testing is context-driven • And, there are small things that really make a big difference in how we may apply a technique.
  7. 7. 4/29/2014 5 9 FROM CLASSROOM TO PRACTICE Learning Objectives Retention Instruction Exercises Visual Aids Application 10 BUT, IT GETS WORSE Learning Objectives Instruction Visual Aids Exercises Retention for multiple people Application by multiple people Passage of time. Needs change, organizations change
  8. 8. 4/29/2014 6 11 WHY? • Let’s look at the lowest four levels of the Bloom Taxonomy: K4 (Analyze) - Assimilate K3 (Do) - Practices K2 (Understand) – Principles, concepts, theory K1 (Remember) - Facts The great majority of Foundation Level learning objectives 4 Foundation learning objectives 12 7 GENERAL TESTING PRINCIPLES • Testing shows the presence of defects • Exhaustive testing is impossible • Early testing • Defect clustering • Pesticide paradox • Testing is context dependent • Absence-of-errors fallacy Source: ISTQB Foundation Level Syllabus
  9. 9. 4/29/2014 7 13 THERE ARE MANY OTHER IMPLICIT PRINCIPLES • Not every failure is a defect. • Not every test can or should be automated. • Test automation doesn’t replace the need for manual testing • It doesn’t matter how good your test is, if you are testing the wrong version. 14 CLIMBING K2 • People normally dislike this level of learning. • Conceptual • Theoretical • Hands-off • Requires deeper understanding • Takes longer to build the basis of understanding
  10. 10. 4/29/2014 8 15 ANOTHER CHALLENGE • “Just show us how to use the tool.” • The students • “We want a course with all practical application and little or no theory.” • Management 16 A LITTLE TIME GOES BY… • “We don’t know really get how to design tests for our applications. We just learned on the case study example.” • Students • “The students passed the exam but we are still struggling in how to implement the techniques.” • Management
  11. 11. 4/29/2014 9 17 AN EXAMPLE FROM MUSIC • It is possible to learn to play an instrument from mimicking someone else. • The Suzuki Method • However, you are able to play only specific songs. • Professional musicians, even many rock stars, understand music theory! • Blues, heavy metal and other styles are all based on different types of scales and tunings. 18 WE SEE THE SAME THING IN SPORTS • Why do professional baseball players, golfers, football players need coaches? • Objectivity • Accountability • Teaching from a broader background of experience
  12. 12. 4/29/2014 10 19 WHAT SETS APART PROFESSIONAL MUSICIANS AND ATHLETES? • Practice! • Professional golfers are on the driving range every day and play most days. • Football players practice in extreme conditions. • Musicians may practice hours for one show. 20 HOW DO TESTERS PRACTICE? • Try new approaches • Delve deeper into an approach or technique you already know • Beta test a new technology • Play mental games
  13. 13. 4/29/2014 11 TWO EXAMPLES OF TEST TECHNIQUES THAT REQUIRE DEEPER UNDERSTANDING 22 EXAMPLE 1 – PAIRWISE TESTING • Key Principles • Used when there are distinct numbers of conditions that can be tested together, but the testing of all combinations would be infeasible. • Instead of testing all combinations of conditions, all unique pairs of conditions are tested. • Finds defects when: • The interaction between TWO conditions are incorrect • The conditions make sense to relate to each other • Primarily applied by using either orthogonal arrays or pairwise tools. • This can be a very interesting and effective way to deal with testing high numbers of combinations.
  14. 14. 4/29/2014 12 23 Teacher, why do we need to know this? 24 AN EXAMPLE The IE Tools>Advanced Options window 53 binary conditions 1 condition with 3 options 1 condition with 4 options 253 = 9,007,199,254,740,992 X 12 108,086,391,056,891,904 Possible combinations of conditions! Source: James Whittaker, How to Break Software
  15. 15. 4/29/2014 13 25 THIS WOULD REQUIRE… At one second per test execution: 108,086,391,056,891,904 360 300,239,975,158,033.067= hours 12,509,998,964,918.04 days 34,273,969,766.9 years To test all possible combinations! 26 WITH TEST CASE OPTIMIZATION • The number of tests that represent unique pairings = 7 • Obtained from using the AllPairs tool
  16. 16. 4/29/2014 14 27 WHAT RESEARCH INDICATES • There have been many studies done where defects were researched after the fact. • Research has shown that the highest likelihood of an integration defect relates to interaction between pairs of items or parameters. • 28 EXAMPLE OF RESEARCH FINDINGS • [Evaluating FDA recall class failures in medical devices we established that] [...] out of the 109 reports that [were] detailed [enough], 98% showed that the problem could have been detected by testing the device with all pairs of parameter settings. • D.R. Wallace and D.R. Kuhn, 2001 •
  17. 17. 4/29/2014 15 29 THE LIKELIHOOD OF TRIPLE- MODE OR HIGHER DEFECTS Condition B Condition DCondition CCondition A Condition E Double-mode defects are more likely than Triple-mode or higher defects. 30 EXAMPLE – AN OA. ARRAY 011Test case 4 101Test case 3 110Test case 2 000Test case 1 Condition 3Condition 2Condition 1
  18. 18. 4/29/2014 16 31 EXAMPLE – WITH CONDITIONS PoolFarmRuralTest case 4 PondHighriseRuralTest case 3 PondFarmUrbanTest case 2 PoolHighriseUrbanTest case 1 Condition 3Condition 2Condition 1 Some paired conditions may make no sense, or may be infeasible 32 ONE MORE SPIN ON THIS • A client was using orthogonal arrays to optimize the testing of stock trades. • 15 broker types • 3 trade types • 2 customer types • 7 amount levels (bands) • 4 account types • This would yield 15*3*2*7*4 = 2,520 tests for all combinations. • We noticed that the 15 broker types could be safely divided into 5 equivalence classes. • 5*3*2*7*4 = 840 • This resulted in a 67% reduction in test cases before any pairwise analysis!
  19. 19. 4/29/2014 17 33 WHAT WE’VE SEEN • Instead of going immediately to a tool or technique with a set of categories and conditions, let’s look first to see: • If the technique makes sense in the situation • How do we deal with the downsides? • In this example, can they be used as negative tests? • What do the stakeholders think of the technique? • I’ve seen some get very excited • I’ve seen some get very nervous! • Are we willing to accept the trade-offs of pairwise testing? • It becomes very important which categories and conditions we test together. • We still need an oracle to know the correct outcome. • Maybe there is a provision for a highrise building on a farm! 34 EXAMPLE #2 – RISK-BASED TESTING • Key Principles • Used when you can’t test the basic layer of functionality and/or when you need to prioritize testing to meet delivery schedules • The higher the risk, the more intense the test should be • Helps by: • Focusing on areas that might experience high levels of loss • Giving lesser attention to trivial or less important functions • Tracing and mitigating risks identified during testing • Can be used in just about any application as long as you are able to intelligently identify and measure risk. • It is possible to miss some risks and some defects. • Risk is a possibility, not a certainty • Our knowledge is limited • Some things will likely not get tested
  20. 20. 4/29/2014 18 35 Teacher, isn’t all testing risk-based? 36 Universe of Test Cases Case 1 Case 2 Case 3 Case 4 Case 5 Case n Case 6 Case 7 Case 11 Case 8 Case 12 Case 13 Case 10 Case 14 Case 9 CONSIDER THE FOLLOWING
  21. 21. 4/29/2014 19 37 Universe of Test Cases Case 1 Case 2 Case 3 Case 4 Case 5 Case n Case 6 Case 7 Case 11 Case 8 Case 12 Case 13 Case 10 Case 14 Case 9 High-risk cases Low-risk cases Moderate-risk cases THE TEST CASE UNIVERSE SEGMENTED BY RISK 38 Universe of Test Cases Case 1 Case 2 Case 3 Case 4 Case 5 Case n Case 6 Case 7 Case 11 Case 8 Case 12 Case 13 Case 10 Case 14 Case 9 High-risk cases Low-risk cases Moderate-risk cases TEST CASES THAT SPAN RISK LEVELS In testing this transaction, you span all levels of risk. How do you prioritize risk in this scenario?
  22. 22. 4/29/2014 20 39 HOW TO APPLY • At various levels: • Product risk • Functional risk • Attribute risk – Usability, performance, etc. • Project risk • But what about?: • Residual risks • Safety critical (when everything is a “high” risk) • When risks are: • Difficult to identify • Change often • Are an excuse for reduced testing • Difficult to mitigate and track This is the practice level! 40 A TIP TO HELP IDENTIFY YOUR PRACTICES Area/Principles Implications Test Automation • You can’t automate everything • Automation doesn’t replace people • The ROI of test automation is repetition • Select tests carefully to automate • Consider which skills will need to be built • To get payback, start with simple tests that are performed a lot. Test Design • Test conditions form the basis of test cases • Many things can serve as a basis for test design, including experience • Before writing test cases, have a smart way to identify test conditions. • If you don’t have a good written basis for testing, you can create one. Test Evaluation • If you can’t observe the outcome of a test, you can’t evaluate it • Testing is measurement • If we can’t define a way to see, report, or measure the outcome of a test, it does little good to plan it. • Since measurement of something (defects, performance, etc.) is the deliverable of testing, we should do it well.
  23. 23. 4/29/2014 21 HOW TO GAIN UNDERSTANDING GOING FROM PRINCIPLE TO PRACTICES 42 A FEW EXPECTATIONS • This is not easy, automatic or instant • This is a mental process more than anything else • This is also the result of practice and experience • This is not a classroom deliverable • Mentoring and coaching can help you see blind spots, but they can’t make you change.
  24. 24. 4/29/2014 22 43 WHAT REALLY HAPPENS Instructor says, “Do it like this ” You say ”OK” But think, “I’m not so sure” or “I like my way better.” 44 However, I know I need to improve my game. (I think)
  25. 25. 4/29/2014 23 45 ACTUALLY, THAT’S FINE • Each of us know internally the way we work best. • An external coach can provide tips and advice, but you know what works for you. • A great resource for this is the series, “The Inner Game” by Timothy Gallwey. 46 WHAT WE ARE TALKING ABOUT IS CHANGE No Knowledge Mastery “The Dip” Years Change is hard! We often fail
  26. 26. 4/29/2014 24 47 COMPETENCE LEVELS You do things badly but don’t know it. You do things badly but do know it. You do things well but have to think carefully about what you are doing. You do things well and really don’t think much about the mechanics. 48 Source: Will Taylor
  27. 27. 4/29/2014 25 49 IN TAKING EXAMS • Don’t overthink the questions. • Read it carefully and think it through, but don’t go back and forth too much on one question. • Don’t rely on practice exams. • They can be HUGELY misleading. • Instead, ask “Can I demonstrate the learning objectives?” • Read the book “Choke”. • It explains why we get all flustered and don’t do well on exams. 50 IMPLEMENTING PRACTICES • Pilot projects are great. • Learn in the small before going to the larger scale • Use the “planned organic” approach • Have senior management support • Let others copy your success!
  28. 28. 4/29/2014 26 51 FINAL THOUGHT • “the 10,000hr rule is a definite key in success” ― Malcolm Gladwell, Outliers: The Story of Success 52
  29. 29. 4/29/2014 27 53 BIO - RANDALL W. RICE • Over 35 years experience in building and testing information systems in a variety of industries and technical environments • ASTQB Certified Tester – Foundation level, Advanced level (Full) • Director, American Software Testing Qualification Board (ASTQB) • Chairperson, 1995 - 2000 QAI’’’’s annual software testing conference • Co-author with William E. Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems 54 CONTACT INFORMATION Randall W. Rice, CTAL Rice Consulting Services, Inc. P.O. Box 892003 Oklahoma City, OK 73170 Ph: 405-691-8075 Fax: 405-691-1441 Web site: e-mail: