0
IBM Haifa Research LabCombinatorial Test Design and the FoCuS Tool          Itai Segall          Software Testing Verifica...
IBM Haifa Research LabAbout us    Software Testing, Verification and Review    Methodologies group at IBM Haifa Research L...
IBM Haifa Research LabOutline    Combinatorial Test Design (CTD) Concepts and    Overview    The FoCuS Tool and the Modeli...
IBM Haifa Research LabObservation - Omission is a Major Risk in Testing    Choosing what to test out of an enormous test s...
IBM Haifa Research LabAn Example of a Test Design    The configuration space for a cellphone application     – Bluetooth –...
IBM Haifa Research LabDo we really need to test all 5120 configurations?    So, to balance cost and risk, we look to selec...
IBM Haifa Research LabHow do we reduce the 5120 potential test cases?    Interactions !      – Interaction level of every ...
IBM Haifa Research LabHow do we reduce the 5120 potential test cases?    The configuration space for a cellphone applicati...
IBM Haifa Research LabHow do we reduce the 5120 potential test cases? Interaction level of every 2 requires 16 tests – 99....
IBM Haifa Research LabHow do we reduce the 5120 potential test cases? Interaction level of every 2 requires 16 tests – 99....
IBM Haifa Research LabHow do we reduce the 5120 potential test cases? Interaction level of every 2 requires 16 tests – 99....
IBM Haifa Research LabHow do we reduce the 5120 potential test cases? Interaction level of every 2 requires 16 tests – 99....
IBM Haifa Research LabOutline     Combinatorial Test Design (CTD) Concepts and     Overview     The FoCuS Tool and The Mod...
IBM Haifa Research LabA Comment on the Use of CTD in the Industry     Many CTD tools were created      – The mathematics a...
IBM Haifa Research LabCTD within our FoCuS tool     Tackles complexity of modeling by Cartesian product methodology     Fu...
IBM Haifa Research LabThe cell phone example in FoCuS                                  © 2010 IBM Corporation
IBM Haifa Research LabExample – cont.    Cartesian product consists of 5120 possible    configurations:                   ...
IBM Haifa Research LabExample – Cont.    Some combinations do not make sense, and should    be removed:                   ...
IBM Haifa Research LabExample – Cont.    Now the model contains 2560 legal tasks:                                         ...
IBM Haifa Research LabExample – Cont.    A test plan can now be created:                                      © 2010 IBM C...
IBM Haifa Research LabExample – Cont.    16 tests suffice:                             © 2010 IBM Corporation
IBM Haifa Research LabExample – Cont.    Suppose I have only a few Bluetooth components                                   ...
IBM Haifa Research LabExample – Cont.    The new test plan:                             © 2010 IBM Corporation
IBM Haifa Research LabExample – Cont.    More complex coverage requirements may be stated:                                ...
IBM Haifa Research LabExample – cont.    And the resulting test plan:                                   © 2010 IBM Corpora...
IBM Haifa Research LabExample – cont.    “.. But I already have some tests implemented, I want    to use them”            ...
IBM Haifa Research LabExample – cont.    FoCuS can analyze what is not covered:                                           ...
IBM Haifa Research LabExample – cont.                             © 2010 IBM Corporation
IBM Haifa Research LabExample – cont.    .. and suggest a complementary test plan:                                        ...
IBM Haifa Research LabComparing Manually Created Test Plans to Usage ofModeling and CTD     Manual test plans will usually...
IBM Haifa Research LabAdditional Advantages of Systematic Test Plan Design     Once the methodology is learned, it is much...
IBM Haifa Research LabThank You   Trial download:   http://www.alphaworks.ibm.com/tech/focus   Contact me at: itais@il.ibm...
Upcoming SlideShare
Loading in...5
×

כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר) בכיסוי המערכת?

638

Published on

מצגת שהעביר איתי סגל ממעבדות המחקר של IBM בחיפה, בכנס בדיקות אוטומציה של טאקט שנערך בחיפה בשבוע שעבר.

איתי מציג טכניקה + כלי שמסייעת לתכנן כיסוי בדיקות בצורה הרבה יותר יעילה ואפקטיבית.

לשיטה קוראים All pairs והיא מתמקדת בבדיקת כל הצירופים של כל זוג משתנים במערכת בהנחה (שמגובה במחקרים) שמרבית התקלות יימצאו או כתוצאה מגורם אחד או בצירוף של שני גורמים (לדוגמא: מערכת הפעלה עם נגן מדיה כלשהו).

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
638
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר) בכיסוי המערכת?"

  1. 1. IBM Haifa Research LabCombinatorial Test Design and the FoCuS Tool Itai Segall Software Testing Verification and Reviews Group © 2010 IBM Corporation
  2. 2. IBM Haifa Research LabAbout us Software Testing, Verification and Review Methodologies group at IBM Haifa Research Labs – Concurrent testing – Code review methodology and tool support – Code coverage analysis – Test selection – Functional modeling and test planning © 2010 IBM Corporation
  3. 3. IBM Haifa Research LabOutline Combinatorial Test Design (CTD) Concepts and Overview The FoCuS Tool and the Modeling and Reduction Process3 © 2010 IBM Corporation
  4. 4. IBM Haifa Research LabObservation - Omission is a Major Risk in Testing Choosing what to test out of an enormous test space – What is a test space? The set of parameters/values that define the scenarios, inputs, configurations or conditions to be tested, such as • Customer type (business, residential, government, …) • Customer geography (provinces) • Transaction type (add service, delete service, request credit …) When you have a list of 3000 tests, can it be reviewed? How do you know what’s there and what is missing? – Coverage (in retrospect) – Preventive techniques using CTD4 © 2010 IBM Corporation
  5. 5. IBM Haifa Research LabAn Example of a Test Design The configuration space for a cellphone application – Bluetooth – Yes / No – GPS – Yes / No – GPS software – Support for different GSM frequencies – Support for different 3G frequencies – Support for EMS and MMS Total number of combinations – 2 x 2 x 5 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 5120 Too many!5 © 2010 IBM Corporation
  6. 6. IBM Haifa Research LabDo we really need to test all 5120 configurations? So, to balance cost and risk, we look to select a subset of tests that covers the interactions of configuration variables at some level of interaction (pairs, three- way, etc.) Source http://csrc.nist.gov/groups/SNS/acts/ftfi.html used in slides 6,8,96 © 2010 IBM Corporation
  7. 7. IBM Haifa Research LabHow do we reduce the 5120 potential test cases? Interactions ! – Interaction level of every 2 means that for any two variables all the combinations of their values have been tested together. – Every three is the same for every three variables IBM’s FoCuS tool models the variables, and allows us to adjust the level of interaction/coverage to achieve acceptable cost/risk tradeoff Explicit identification of what is tested and what is not tested © 2010 IBM Corporation
  8. 8. IBM Haifa Research LabHow do we reduce the 5120 potential test cases? The configuration space for a cellphone application – Bluetooth – Yes / No – GPS – Yes / No – GPS software – Support for 3 different GSM frequencies – Support for 2 different 3G frequencies – Support for EMS and MMS Total number of combinations – 2 x 2 x 5 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 5120 Question: How many tests are required for covering every combination of every 2 parameters? Answer: only 16 ! (99.7% reduction from 5120)8 © 2010 IBM Corporation
  9. 9. IBM Haifa Research LabHow do we reduce the 5120 potential test cases? Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120 GPS_Soft Bluetooth GPS ware Wi_Fi GSM850 GSM900 GSM1800 UMTS850 UMTS2100 EMS MMS FALSE TRUE Destinator TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE uDrive FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE None TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE mioMap TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE iGo TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE iGo TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE mioMap FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE Destinator FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE Destinator FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE mioMap TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE None TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE iGo FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE uDrive TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE None FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE © 2010 IBM Corporation
  10. 10. IBM Haifa Research LabHow do we reduce the 5120 potential test cases? Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120 GPS_Soft Bluetooth GPS ware Wi_Fi GSM850 GSM900 GSM1800 UMTS850 UMTS2100 EMS MMS FALSE TRUE Destinator TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE uDrive FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE None TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE mioMap TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE iGo TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE iGo TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE mioMap FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE Destinator FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE Destinator FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE mioMap TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE None TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE iGo FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE uDrive TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE None FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE © 2010 IBM Corporation
  11. 11. IBM Haifa Research LabHow do we reduce the 5120 potential test cases? Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120 GPS_Soft Bluetooth GPS ware Wi_Fi GSM850 GSM900 GSM1800 UMTS850 UMTS2100 EMS MMS FALSE TRUE Destinator TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE uDrive FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE None TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE mioMap TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE iGo TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE iGo TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE mioMap FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE Destinator FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE Destinator FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE mioMap TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE None TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE iGo FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE uDrive TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE None FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE © 2010 IBM Corporation
  12. 12. IBM Haifa Research LabHow do we reduce the 5120 potential test cases? Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120 GPS_Soft Bluetooth GPS ware Wi_Fi GSM850 GSM900 GSM1800 UMTS850 UMTS2100 EMS MMS FALSE TRUE Destinator TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE uDrive FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE None TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE FALSE TRUE mioMap TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE iGo TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE iGo TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE mioMap FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE Destinator FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE Destinator FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE mioMap TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE TRUE FALSE None TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE iGo FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE TRUE TRUE uDrive TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE FALSE FALSE None FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE © 2010 IBM Corporation
  13. 13. IBM Haifa Research LabOutline Combinatorial Test Design (CTD) Concepts and Overview The FoCuS Tool and The Modeling and Reduction Process13 © 2010 IBM Corporation
  14. 14. IBM Haifa Research LabA Comment on the Use of CTD in the Industry Many CTD tools were created – The mathematics are very nice to play with – The problem is solved with advanced combinatorics However, most are almost toy-tools – Real world problems are not toy examples – Real world problems may be hard to model – Not all value combinations are possible – there are restrictions – Existing tests need to be evaluated or taken into account – Advanced requirements such as distribution requirements on the values in the resulting tests Most tools miss many things testers need14 © 2010 IBM Corporation
  15. 15. IBM Haifa Research LabCTD within our FoCuS tool Tackles complexity of modeling by Cartesian product methodology Fully supports restrictions on the valid combinations Highly scalable Supports advanced features – Distribution requirements – Good path and bad path test planning Considers existing tests – Report uncovered areas – Suggest complementary test plans15 © 2010 IBM Corporation
  16. 16. IBM Haifa Research LabThe cell phone example in FoCuS © 2010 IBM Corporation
  17. 17. IBM Haifa Research LabExample – cont. Cartesian product consists of 5120 possible configurations: © 2010 IBM Corporation
  18. 18. IBM Haifa Research LabExample – Cont. Some combinations do not make sense, and should be removed: © 2010 IBM Corporation
  19. 19. IBM Haifa Research LabExample – Cont. Now the model contains 2560 legal tasks: © 2010 IBM Corporation
  20. 20. IBM Haifa Research LabExample – Cont. A test plan can now be created: © 2010 IBM Corporation
  21. 21. IBM Haifa Research LabExample – Cont. 16 tests suffice: © 2010 IBM Corporation
  22. 22. IBM Haifa Research LabExample – Cont. Suppose I have only a few Bluetooth components © 2010 IBM Corporation
  23. 23. IBM Haifa Research LabExample – Cont. The new test plan: © 2010 IBM Corporation
  24. 24. IBM Haifa Research LabExample – Cont. More complex coverage requirements may be stated: © 2010 IBM Corporation
  25. 25. IBM Haifa Research LabExample – cont. And the resulting test plan: © 2010 IBM Corporation
  26. 26. IBM Haifa Research LabExample – cont. “.. But I already have some tests implemented, I want to use them” © 2010 IBM Corporation
  27. 27. IBM Haifa Research LabExample – cont. FoCuS can analyze what is not covered: © 2010 IBM Corporation
  28. 28. IBM Haifa Research LabExample – cont. © 2010 IBM Corporation
  29. 29. IBM Haifa Research LabExample – cont. .. and suggest a complementary test plan: © 2010 IBM Corporation
  30. 30. IBM Haifa Research LabComparing Manually Created Test Plans to Usage ofModeling and CTD Manual test plans will usually have many large holes – And it is not known where they are Manual test plans are more expensive – Take a lot of time to design – Manual test plans contain more tests, so longer time to implement the tests and longer time to execute them Known quality situation – With CTD you can get the desired quality based on risk – With CTD you can compare alternatives, as once the model is created, creating alternatives takes minutes30 © 2010 IBM Corporation
  31. 31. IBM Haifa Research LabAdditional Advantages of Systematic Test Plan Design Once the methodology is learned, it is much less work – less tests to execute The tool tells you how many test configurations are needed and specific configurations that fulfill your requirements – You can discuss alternatives for how the budget will be used Having a model instead of a spreadsheet test list is much more useful – Less effort to maintain the model – From the model you can generate lists after changes are made • A new operating system was added • Need to add a new variable that interacts with the current ones • Need to remove variables or values and make sure that the test plan maintains quality – The model is much easier to review than traditional test plans as the generation scheme explicitly describes what is tested31 © 2010 IBM Corporation
  32. 32. IBM Haifa Research LabThank You Trial download: http://www.alphaworks.ibm.com/tech/focus Contact me at: itais@il.ibm.com © 2010 IBM Corporation
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×