SlideShare a Scribd company logo
1 of 76
Download to read offline
Combinatorial
Software Test
   Design

                (Beyond
                Pairwise
                Testing)




      1
Presented at STPCon
                                                              October, 2010

                                                          By Justin Hunter
                                                          CEO of Hexawise



Cover slide photo credit: "Mike" Michael L. Baird, flickr.bairdphotos.com
http://www.flickr.com/photos/mikebaird/2127310513

                                                                           2
L E M
      O B
P R
        3
The Problem


           You can’t test
            everything.




http://www.flickr.com/photos/maisonbisson/109211670/   2
The Problem



       “With so many
       possible tests to
       choose from...”




http://www.flickr.com/photos/madabandon/344158061/
                                                    2
The Problem



                                    “Things might
                                        seem
                                   uncontrollable...”




http://www.flickr.com/photos/maisonbisson/110867288/sizes/l/in/photostream/
                                                                             2
The Problem




                                                     ... discouraging,
                                                            even.


http://www.flickr.com/photos/cheo70/2702682262/   2
The Problem


                                                         So, how can you select
                                                            just a few tests...




http://www.flickr.com/photos/krawcowicz/3830467093/
                                                     8
The Problem




                        ... and still
                          achieve
                         adequate
                        coverage?




http://www.flickr.com/photos/editor/4861496460/   9
The Problem




                     Wait. So this is a test design method to
                  achieve greater coverage in fewer tests. Right?
http://www.flickr.com/photos/johncarleton/4890239248
                                                      10
Not Just Fewer, Better Tests

     If you prioritize                                But for those testers who
     speed, yes. You                                   prioritize thoroughness,
        can achieve                                    this method can also be
    greater coverage                                       used to increase
      in fewer tests.                                thoroughness dramatically.




                                                                It can help you
                                                               prioritize around
http://www.flickr.com/photos/legofenris/4212007962/
                                                                   either goal.
                                                        11
Not Just Fewer, Better Tests




                W  ay                                 The
                                                    choice is
                fewer                               yours to
                                                     make.
                 tes ts!
                                       etter
                             ligh tly b
                        ith s than we
                     (W
                            rage
                       cove e now.)
                            hav




http://www.flickr.com/photos/thomashawk/2402194148            http://www.flickr.com/photos/thomashawk/2402190314
                                                        12
Not Just Fewer, Better Tests


                                                           How can you create “curiously
                                                          strong” test cases that will allow
                                                           you to achieve either of these
                                                                 testing objectives?




http://www.flickr.com/photos/photobunny_earl/2309612944/
                                                                13
I O N
     L UT
S   O
       14
The Solution

                                                This presentation describes an
                                               approach that will help you create
                                                     powerful test cases.




http://www.flickr.com/photos/maisonbisson/110867008/
                                                        15
Structured Variation




 To do so, use
combinatorial
  test design
methods that
     build
  structured
   variation
into your test
     cases.
      16
Structured Variation?




                                                                 “Test One”




http://www.flickr.com/photos/gelrodandjodie/148759732
                                                       17
Structured Variation?

      This is not what I mean by structured
                    variation.




                                                                 “Test Two”




http://www.flickr.com/photos/gelrodandjodie/148759732
                                                       18
Structured Variation?

      This is not what I mean by structured
                    variation.




                                                                “Test Three”




http://www.flickr.com/photos/gelrodandjodie/148759732
                                                       19
Structured Variation?




 So if that’s not structured
     variation, what is?
20
Structured Variation



                                                  Structured
                                                   variation
                                                    means




http://www.flickr.com/photos/tk_five_0/2281768728     21
Structured Variation




                                                           ...each test
                                                              will be
                                                             different




http://www.flickr.com/photos/criben/2635618391
                                                22
Structured Variation



                                                   ...than the
                                                  tests before
                                                      it, and




http://www.flickr.com/photos/horrgakx/3186982135      23
Structured Variation




     ...testers
    will do their
     testing in
     different
        ways,




http://www.flickr.com/photos/pixelrobber/2626213081   24
Structured Variation




                                                          ... and
                                                         explore
                                                           from
                                                           more
                                                         angles.




http://www.flickr.com/photos/tk_five_0/2286693552   25
Structured Variation




          “OK. Maximum variation is better than redundantly
                repeated redundant repetition. Got it.
                   That it? Am I good to go now?”
http://www.flickr.com/photos/johncarleton/4890241530   26
Coverage



                          No.
                        There’s
                         more.
                      Much more.
                      Remember
                       coverage?




http://www.flickr.com/photos/30431232@N03/3610846363
                                                      27
Coverage




    Accidentally
    leaving gaps
     in coverage
     can lead to
        painful
       results.




http://www.flickr.com/photos/dwfromthepeg/240921271/
                                                      28
Coverage




                                                    Structured
                                                     variation
                                                    focuses on
                                                    covering as
                                                      much as
                                                     possible...




http://www.flickr.com/photos/localsurfer/38245658/
                                                             29
Coverage




                          ... with as
                              few
                          resources
                               as
                          possible...




http://www.flickr.com/photos/mikebaird/4177570453/
                                                    30
Coverage


                ...and in the
                    most
                  efficiently
                 structured
                way possible.




http://www.flickr.com/photos/dannohung/332507807
                                                  31
F
        O &
      S
     U N
   C IO
 O T
F U        L E
   L ONA
 O I
S T
 R A  32
Single-Mode Faults




                                                             Most software defects
                                                              in production today
                                                               can be triggered by
                                                               just one test input.



http://www.flickr.com/photos/vaguelyartistic/292083492
                                                        33
Dual-Mode Faults



     Software bugs caused
     by the interaction of
      two test inputs are
      very common too.




http://www.flickr.com/photos/kanaka/1798327442
                                                34
Triple-Mode Faults

                                      Defects that require three specific test
                                         inputs to trigger them are rare.




http://www.flickr.com/photos/kanaka/1798327442
                                                  35
Quadruple-Mode Faults


            Defects that require four more test inputs to
                  trigger them are extremely rare.




http://www.flickr.com/photos/novembering/3816524907
                                                     36
Implication on Testing Strategy




                 Aha! So, to find bugs efficiently, we should
                      focus first on testing all possible
                 combinations every set of TWO test inputs!
http://www.flickr.com/photos/johncarleton/4890236762      37
Implication on Testing Strategy

             That’s right, two.
            Focus on covering all
             possible two-way
            defects first. In fact,
            that’s so important,
              can you say it a
                second time?




http://www.flickr.com/photos/soulsurfingcrew/2273300213
                                                           38
Implication on Testing Strategy

     To find bugs efficiently                                      we should focus
                                                                 first on testing




all possible combinations of                                   TWO test inputs!




http://www.flickr.com/photos/johncarleton/4890236762      39
I C S
            AN
    E C   H
M
           40
Combinations of Test Inputs


                       The key
                     words here
                        are test
                      inputs and
                    combinations.




http://www.flickr.com/photos/12tribu/2750040244
                                                 41
Combinations of Test Inputs


      Test inputs
       will vary
     dramatically
     from project
      to project.




42
Combinatorial Software Test Design



                                                                           Well, duh!
                                                                          They would,
                                                                         wouldn’t they?
                                                                         But how does
                                                                          it all work?




http://www.flickr.com/photos/johncarleton/4377591743
                                                             43
Combinatorial Software Test Design


                Here’s how it works: First, people
                are required to identify what test
                     inputs should be tested.




http://www.flickr.com/photos/mikebaird/3939231212
                                                          44
Combinatorial Software Test Design




 Second, computer algorithms will take those
   inputs and quickly calculate the smallest
possible number of tests required to meet the
     tester’s specified coverage objectives.


                     45
Combinatorial Software Test Design

                                                     ... and again, the
                                                      algorithms will
                W  ay                                 create different
                                                           solutions
                fewer                                depending upon

                 tes ts!                                 your testing
                                                          objectives.
                                        etter
                              ligh tly b
                         ith s than we
                      (W
                             rage
                        cove e now.)
                             hav




http://www.flickr.com/photos/thomashawk/2402194148                 http://www.flickr.com/photos/thomashawk/2402190314

                                                              46
Combinatorial Software Test Design



                 W  ay                                                The woman
                                                                     wanting fewer
                 fewer                                                tests should

                  tes ts!                                            create 2-way
                                                                          tests
                                               r
                                          bette
                                     htly
                         Wi
                                  ig
                             th sl n we ha
                                                ve                       (AKA
                       (            ha
                              age t .)
                        c over     now                               “pairwise” or
                                                                    “AllPairs” tests)




http://www.flickr.com/photos/thomashawk/2402194148
                                                            47
Combinatorial Software Test Design



This guy, wanting more
coverage, should create
     more thorough
  combinatorial tests.
 (e.g., 3-way, 4-way, or
 multi-strength tests).




                                http://www.flickr.com/photos/thomashawk/2402190314


                           48
OT
      Y N STS
    H
 W NT ?   E
   S  G ND
     I A
 E
D YH
   B  49
Advantages of Combinatorial Software Test Design

                              Test Design: Man vs. Machine
                                        (Once test inputs are known)

                                        Speed:                          Coverage:                            Efficiency:

      Human                              Weeks                                Gaps                           Way More
                                                                                                              Tests




   Computer                            Seconds                                100%*                         Way Fewer
                                                                                                              Tests




*100% of what? 100% of the desired test input combinations (e.g., all pairs of values for 2-way tests solutions, all triplets of possible
values for 3-way solutions, etc.).
Coverage Advantages

            Whatever
            your test
              inputs,
         combinatorial
         testing makes
           it easy and
          fast to cover
            all of the
              specific
         combinations
         of test inputs
             that you
           suspect are
            relatively
           important.
http://www.flickr.com/photos/kanaka/1798327442   51
Advantages of Combinatorial Software Test Design




                                            OK, but wouldn’t I have tested
                                             those relatively important
                                              combinations on my own?
http://www.flickr.com/photos/johncarleton/4890239248
Advantages of Combinatorial Software Test Design



Perhaps, but maybe you...




               Forget        Run out of time



          Use more tests      Accidentally
           than needed       repeat yourself

                        53
Coverage Advantages




    Plus, using combinatorial testing, you’ll
also get additional coverage from combinations
 that wouldn’t have made your list, including...


http://www.flickr.com/photos/tk_five_0/2706278432   54
Coverage Advantages


                                         ...specialized use cases,
                    No, its way
                    cooler than
                      pareto.




http://www.flickr.com/photos/subspace-eddy/2766810770/
                                                        55
Coverage Advantages




  ... unexpected
  combinations,




http://www.flickr.com/photos/funky64/2875860978
                                                 56
Coverage Advantages




                                                                 ...notable,
                                                               combinations
                                                               that capture
                                                              your attention,
http://www.flickr.com/photos/brb_photography/3221049399
                                                         57
Coverage Advantages




                                                         ...even combinations that
                                                       rarely, if ever, actually occur,




http://www.flickr.com/photos/grassvalleylarry/5536758
                                                         58
Coverage Advantages

      ... and...
  combinations
  that will take
     you into
     different
   parts of the
    application
  (and different
   states) than
    you would
         have
    otherwise
    thought to
     explore.

http://www.flickr.com/photos/28041049@N00/227352411
                                                     59
Coverage Advantages




Quickly testing
unusual pairs is
     often
  surprisingly
  efficient and
effective. Many
  unexpected
    two-way
  interactions
     cause
   problems.
                   60
Constraint Handling

                                                                           Combinations
                                                                          that can’t ever
                                                                        appear in reality?
                                  Yes.                                  No problem. You
                                                                        can prevent them
                                                                        from appearing in
                                                                         your plans with
                                                                          “Invalid Pairs.”




http://www.flickr.com/photos/kaptainkobold/4741725788/lightbox/
                                                                 61
E T ?
         Y Y
  E AD
R
         62
Ready?




                                                           I’m ready
                                                            already!
                                                            Let’s do
                                                              this!
http://www.flickr.com/photos/johncarleton/4890241530
                                                      63
Hazards




                                                               Not so fast.
                                                              Hazards exist.
                                                               To succeed,
                                                              you must use
                                                                common
                                                                 sense.
http://www.flickr.com/photos/christianhaugen/3604106558
                                                         64
Limitations




                                                       And let’s
                                                        not kid
                                                       ourselves.
                                                        Not just
                                                        anyone
                                                        can do
http://www.flickr.com/photos/mikebaird/573368639
                                                       this stuff.
                                                  65
Prerequisites




                                                         It also requires
                                                           good critical
                                                         thinking skills.


http://www.flickr.com/photos/johncarleton/9511759/
                                                    66
Prerequisites




                                                    To thrive,
                                                    you must
                                                    be able to
                                                     develop
                                                     models
                                                      about
                                                     how the
                                                     systems
                                                      you’re
                                                    exploring
http://www.flickr.com/photos/10ch/3101275536/
                                               67     work.
Practice Makes Perfect



         Training
       and practice
         will help
           you
         succeed.




http://www.flickr.com/photos/mikebaird/2128091146
Questions




                        ?

http://www.flickr.com/photos/nathaninsandiego/3647136579
                                                          69
Ready?



                                                        Let’s get
                                                        started!




http://www.flickr.com/photos/mikebaird/2996548709
                                                   70
D I X
                                         P E N
                   AP
http://www.flickr.com/photos/dannohung/332507807
                                                  71
Thank you

      Special thanks to the folks at:
         and its talented contributing
      photographers, including: Mike Baird




        ... and the inimitable John Carleton
(who, let the record show, had no input into the cartoonish
  statements we added next to his striking expressions).




                            72
Why 2-way Testing is Effective

Multiple thorough studies show approximately 85% of
defects in production could have been detected by
simply testing all possible pairs of values.



                                                      Text




     Source: “Combinatorial Testing” IEEE Computer, Aug, 2009. Rick Kuhn, Raghu Kacker, Jeff Lei, Justin Hunter
Additional Information

Excellent introductory articles and instructional videos: www.CombinatorialTesting.com


Results of a ten-project study of effectiveness:
https://www.hexawise.com/Combinatorial-Softwar-Testing-Case-Studies-IEEE-Computer-Kuhn-Kacker-Lei-Hunter.pdf



Free trials of our firm’s combinatorial test design tool: www.hexawise.com/users/new (which stay free
indefinitely for companies using 1-4 licenses).


Also, please feel free to contact me if you have any questions. I’d be happy to quickly review a test
plan or two, answer your questions, give quick pointers to help you run a pilot, etc. Seriously.... I enjoy
helping people get started with this test design approach. Please don’t hesitate to reach out. There’s
no charge and no catch.




                                                     74
Practice Tips: 4-Step Process

     Four-Step Process to Design Efficient and Effective Tests

                                                              Scope                                                             Level	
  of	
  Detail                                          Passive	
  Field	
  Treatment
     1.	
  Plan                    Be	
  clear	
  about	
  single	
  or	
  mul3ple:                   Acceptable	
  op3ons:	
                                                      Dis3nguish	
  between	
  important	
  fields	
  (par3cularly	
  those	
  that	
  
                                                                                                                                                                                   will	
  trigger	
  business	
  rules)	
  and	
  unimportant	
  fields	
  in	
  the	
  
                                   -­‐	
  	
  Features	
  /	
  Func3ons	
  /	
  Capabili3es           -­‐	
  	
  High	
  level	
  “search	
  for	
  something”                     applica3on.
                                   -­‐	
  	
  User	
  types                                           -­‐	
  	
  Medium	
  level	
  “search	
  for	
  a	
  book”
                                                                                                                                                                                   Quickly	
  document	
  what	
  your	
  approach	
  will	
  be	
  towards	
  
                                   -­‐	
  	
  Business	
  Units                                       -­‐	
  	
  Detailed	
  “search	
  for	
  ‘Catcher	
  in	
  the	
  Rye’	
     passive	
  fields.	
  You	
  might	
  consider:	
  ignore	
  them	
  (e.g.,	
  don’t	
  
                                   -­‐	
  	
  H/W	
  or	
  S/W	
  Configura3ons	
  	
                  	
  	
  	
  	
  by	
  its	
  3tle”                                           select	
  any	
  Values	
  in	
  your	
  plan)	
  or	
  a	
  3	
  Value	
  approach	
  such	
  as	
  
                                                                                                                                                                                   “valid”	
  “Invalid	
  (then	
  fix)”	
  and	
  “Blank	
  (then	
  fix)”


                                                  Configura3ons                                                                                Users                                                                      Ac3ons
     2.	
  Create                  First	
  add	
  hardware	
  configura3ons                           Next,	
  add	
  mul3ple	
  types	
  of	
  users	
  (e.g.,	
                  Start	
  with	
  Big	
  Common	
  Ac3ons	
  made	
  by	
  users
                                                                                                      administrator,	
  customer,	
  special	
  customer)                    AZer	
  comple3ng	
  Big	
  Common	
  Ac3ons,	
  circle	
  back	
  and	
  add	
  
                                   Next	
  add	
  soZware	
  configura3ons
                                                                                                      Consider	
  permission	
  /	
  authority	
  levels	
  of	
  admin	
   Small	
  Ac3ons	
  and	
  Excep3ons
                                                                                                      users	
  as	
  well	
  as	
  business	
  rules	
  that	
  different	
   Remember	
  some	
  ac3ons	
  may	
  be	
  system-­‐generated
                                                                                                      users	
  might	
  trigger


                                                  Business	
  Rules                                                                        Gap	
  Filling                                                               Itera3on
     3.	
  Refine                   Select	
  Values	
  to	
  trigger	
  bus.	
  rules                 Iden3fy	
  gaps	
  by	
  analyzing	
  different	
  states,	
                  Refine	
  longest	
  lists	
  of	
  Values;	
  reduce	
  their	
  numbers	
  by	
  using	
  
                                                                                                                                                                                   equivalence	
  classes,	
  etc.
                                                                                                      orders	
  of	
  opera3ons,	
  decision-­‐tree	
  outcomes	
  
                                   Iden3fy	
  equivalence	
  classes                                  sought	
  vs.	
  delivered	
  by	
  tests,	
  “gap	
  hun3ng”	
              Create	
  Tests	
  with	
  and	
  w/out	
  borderline	
  Values;	
  consider	
  cost/
                                                                                                      conversa3ons	
  w/	
  SME’s,	
  etc.                                         benefit	
  tradeoffs	
  of	
  addi3onal	
  test	
  design	
  refinements
                                   Test	
  for	
  boundary	
  values
                                                                                                      Fill	
  gaps	
  by	
  either	
  (i)	
  adding	
  Parameters	
  and/or	
   Consider	
  stopping	
  tes3ng	
  aZer	
  reaching	
  ~80%	
  coverage	
  
                                   Mark	
  constraints	
  /	
  invalid	
  pairs
                                                                                                      Values	
  or	
  (ii)	
  crea3ng	
  “one-­‐off”	
  tests.	
                 Consider	
  2-­‐way,	
  3-­‐way,	
  and	
  Mixed-­‐Strength	
  op3ons


                                                  Auto-­‐Scrip3ng                                                        Expected	
  Results                                               Con3nuous	
  Improvement
     4.	
  Execute                 	
  	
  	
  




                                   Add	
  auto-­‐scrip3ng	
  instruc3ons	
  once;	
                     Export	
  the	
  tests	
  into	
  Excel	
  when	
  you’re	
  done	
  
                                                                                                                                                                                   If	
  possible	
  measure	
  defects	
  found	
  per	
  tester	
  hour	
  “with	
  and	
  
                                                                                                                                                                                   without	
  Hexawise”	
  and	
  share	
  the	
  results
                                   apply	
  those	
  instruc3ons	
  to	
  all	
  of	
  the	
  tests	
   itera3ng	
  the	
  plan
                                   in	
  your	
  plan	
  instantly                                                                                                                 Add	
  inputs	
  based	
  on	
  undetected	
  defects
                                                                                                        Add	
  complex	
  Expected	
  Results	
  in	
  Excel	
  post-­‐
                                   Don’t	
  include	
  complex	
  Expected	
  Results	
   export                                                                                   Share	
  good,	
  proven,	
  plan	
  templates	
  with	
  others
                                   in	
  auto-­‐scripts	
  

© Hexawise, 2010. All rights reserved.
                                                                                                                                            26
Practice Tips: Warning Signs

“You might be headed for trouble if...”

                                                                                                                     Scope                                                                                                                                                                Level	
  of	
  Detail                                                      Passive	
  Field	
  Treatment
1.	
  Plan                     ...	
  You	
  cannot	
  clearly	
  describe	
  both	
  the	
                                                                                                                                                                               ...	
  Parameters	
  with	
  the	
  most	
  Values	
  have	
  more	
   ...	
  You	
  cannot	
  clearly	
  describe	
  your	
  strategy	
  to	
  deal	
  with	
  
                               scope	
  of	
  your	
  test	
  plan	
  and	
  what	
  will	
  be	
                                                                                                                                                                         Values	
  than	
  they	
  require.	
  	
  365	
  values	
  for	
  “days	
   unimportant	
  details.	
  	
  If	
  Values	
  will	
  impact	
  business	
  rules,	
  focus	
  
                               leZ	
  out	
  of	
  scope.                                                                                                                                                                                                                 of	
  the	
  year”	
  is	
  bad.	
  	
  Instead,	
  use	
  equivalence	
    on	
  them.	
  	
  If	
  Values	
  don’t	
  impact	
  business	
  rules,	
  consider	
  
                                                                                                                                                                                                                                                                          class	
  Values	
  like	
  “weekend”	
  &	
  “weekday.”	
  	
  When	
   ignoring	
  them.	
  
                                                                                                                                                                                                                                                                          in	
  doubt,	
  choose	
  more	
  Parameters	
  and	
  fewer	
  
                                                                                                                                                                                                                                                                          Values.


                                                                   Configura3ons                                                                                                                                                                                                                         Users                                                                                   Ac3ons
2.	
  Create                   ...	
  You	
  have	
  ignored	
  hardware	
  and	
                                                                                                                                                                                         ...	
  You	
  have	
  not	
  included	
  all	
  the	
  different	
  types	
     ...	
  You	
  start	
  entering	
  Small	
  Ac3ons	
  (e.g.,	
  “search	
  for	
  a	
  
                               soZware	
  configura3ons	
  	
  without	
  first	
                                                                                                                                                                                           of	
  users	
  necessary	
  to	
  trigger	
  different	
  business	
            hardback	
  science	
  book	
  by	
  author	
  name”)	
  before	
  you	
  enter	
  Big	
  
                               confirming	
  this	
  approach	
  with	
                                                                                                                                                                                                    rules.	
  	
  What	
  user	
  types	
  might	
  create	
  different	
           Ac3ons	
  (e.g.,	
  “Put	
  Something	
  in	
  Cart.	
  	
  Buy	
  it.”)	
  	
  First	
  go	
  from	
  
                               stakeholders.                                                                                                                                                                                                                              outcomes?	
  	
  Authority	
  level?	
  Age?	
  Loca3on?	
  	
                 beginning	
  to	
  end	
  at	
  a	
  high	
  level.	
  	
  AZer	
  you’ve	
  done	
  that,	
  feel	
  
                                                                                                                                                                                                                                                                          Income?	
  	
  Customer	
  status?                                             free	
  to	
  add	
  more	
  details.	
  



                                                                   Business	
  Rules                                                                                                                                                                                                            Gap	
  Filling                                                                                Itera3on
3.	
  Refine                    ...	
  You	
  forget	
  to	
  iden3fy	
  invalid	
  pairs.	
  	
  	
  	
  	
  -­‐	
                                                                                                                                                        ...	
  You	
  assume	
  that	
  the	
  test	
  condi3ons	
  coming	
           ...	
  You	
  forget	
  to	
  look	
  at	
  the	
  Coverage	
  Analysis	
  charts.	
  	
  If	
  you	
  
                               or	
  -­‐	
  	
  ...	
  You	
  rely	
  only	
  on	
  Func3onal	
                                                                                                                                                                           out	
  of	
  Hexawise	
  will	
  be	
  100%	
  of	
  the	
  tests	
  you	
     achieve	
  80%	
  coverage	
  in	
  the	
  first	
  quarter	
  of	
  the	
  tests,	
  you	
  
                               Requirements	
  and	
  Tech	
  Specs	
  w/out	
                                                                                                                                                                                            should	
  run.	
  	
  There	
  might	
  well	
  be	
  addi3onal	
              should	
  measure	
  the	
  cost/benefit	
  implica3ons	
  of	
  execu3ng	
  the	
  
                               thinking	
  hard	
  yourself	
  and	
  asking	
                                                                                                                                                                                            “one-­‐off”	
  things	
  that	
  you	
  should	
  test	
  and/or	
  a	
         last	
  3/4	
  of	
  the	
  tests.
                               ques3ons	
  to	
  SME’s	
  about	
  business	
  rules	
                                                                                                                                                                                    few	
  nega3ve	
  tests	
  to	
  design	
  by	
  hand.
                               and	
  outcomes	
  that	
  are	
  not	
  yet	
  triggered.


                                                                     Auto-­‐Scrip3ng                                                                                                                                                                                                  Expected	
  Results                                                         Con3nuous	
  Improvement
4.	
  Execute                  ...	
  You	
  add	
  detailed	
  Expected	
  Results	
  in	
                                                                                                                                                                               ...	
  You	
  invest	
  a	
  lot	
  of	
  3me	
  in	
  calcula3ng	
  and	
     ...	
  You	
  don’t	
  ask	
  (when	
  defects	
  that	
  the	
  tests	
  missed	
  are	
  found	
  
                               the	
  tests.	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  -­‐	
     documen3ng	
  Expected	
  Results	
  before	
  you	
  have	
                   post-­‐tes3ng)	
  “What	
  input	
  could	
  have	
  been	
  added	
  to	
  the	
  test	
  
                               or	
  -­‐	
  ...	
  You	
  forget	
  that	
  this	
  feature	
  exists	
                                                                                                                                                                                                                                                  plan	
  to	
  detected	
  this?”	
  “Should	
  I	
  add	
  that	
  input	
  to	
  the	
  
                               and	
  find	
  yourself	
  typing	
  out	
  test-­‐by-­‐test	
                                                                                                                                                                              determined	
  your	
  “final	
  version”	
  Parameters	
  and	
  
                               instruc3ons	
  one-­‐by-­‐one.                                                                                                                                                                                                             Values.	
  	
  Last	
  minute	
  addi3ons	
  to	
  inputs	
  will	
            Hexawise	
  test	
  plan	
  now	
  to	
  improve	
  it	
  in	
  advance	
  of	
  the	
  next	
  
                                                                                                                                                                                                                                                                          jumble	
  up	
  test	
  condi3ons	
  for	
  most	
  test	
  cases.             3me	
  it	
  is	
  used?”	
  


© Hexawise, 2010. All rights reserved.
                                                                                                                                                                                                                                                                                                 27

More Related Content

What's hot

What's hot (20)

Test Automation Strategies For Agile
Test Automation Strategies For AgileTest Automation Strategies For Agile
Test Automation Strategies For Agile
 
AI Testing What Why and How To Do It?
AI Testing What Why and How To Do It?AI Testing What Why and How To Do It?
AI Testing What Why and How To Do It?
 
Building a Test Automation Strategy for Success
Building a Test Automation Strategy for SuccessBuilding a Test Automation Strategy for Success
Building a Test Automation Strategy for Success
 
TDD And Refactoring
TDD And RefactoringTDD And Refactoring
TDD And Refactoring
 
Automation Testing
Automation TestingAutomation Testing
Automation Testing
 
Regression Testing - An Overview
Regression Testing - An OverviewRegression Testing - An Overview
Regression Testing - An Overview
 
QA Best Practices in Agile World_new
QA Best Practices in Agile World_newQA Best Practices in Agile World_new
QA Best Practices in Agile World_new
 
[Infographic] Challenges in Test Automation
[Infographic] Challenges in Test Automation [Infographic] Challenges in Test Automation
[Infographic] Challenges in Test Automation
 
An Introduction to Performance Testing
An Introduction to Performance TestingAn Introduction to Performance Testing
An Introduction to Performance Testing
 
TDD vs. ATDD - What, Why, Which, When & Where
TDD vs. ATDD - What, Why, Which, When & WhereTDD vs. ATDD - What, Why, Which, When & Where
TDD vs. ATDD - What, Why, Which, When & Where
 
Test NG Framework Complete Walk Through
Test NG Framework Complete Walk ThroughTest NG Framework Complete Walk Through
Test NG Framework Complete Walk Through
 
Cypress, Playwright, Selenium, or WebdriverIO? Let the Engineers Speak!
Cypress, Playwright, Selenium, or WebdriverIO? Let the Engineers Speak!Cypress, Playwright, Selenium, or WebdriverIO? Let the Engineers Speak!
Cypress, Playwright, Selenium, or WebdriverIO? Let the Engineers Speak!
 
Test Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & ToolsTest Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & Tools
 
Test automation - What? Why? How?
Test automation - What? Why? How?Test automation - What? Why? How?
Test automation - What? Why? How?
 
Top ten software testing tools
Top ten software testing toolsTop ten software testing tools
Top ten software testing tools
 
Test automation process
Test automation processTest automation process
Test automation process
 
Types of software testing
Types of software testingTypes of software testing
Types of software testing
 
Test Automation
Test AutomationTest Automation
Test Automation
 
Agile Testing by Example
Agile Testing by ExampleAgile Testing by Example
Agile Testing by Example
 
What is Regression Testing? | Edureka
What is Regression Testing? | EdurekaWhat is Regression Testing? | Edureka
What is Regression Testing? | Edureka
 

Similar to Combinatorial software test design beyond pairwise testing

Comment j'ai mis ma suite de tests au régime en 5 minutes par jour
Comment j'ai mis ma suite de tests au régime en 5 minutes par jourComment j'ai mis ma suite de tests au régime en 5 minutes par jour
Comment j'ai mis ma suite de tests au régime en 5 minutes par jour
CARA_Lyon
 
Let's make this test suite run faster - Paris JUG 2011
Let's make this test suite run faster - Paris JUG 2011Let's make this test suite run faster - Paris JUG 2011
Let's make this test suite run faster - Paris JUG 2011
David Gageot
 

Similar to Combinatorial software test design beyond pairwise testing (20)

Tests are Nothing, Testing is Everything
Tests are Nothing, Testing is EverythingTests are Nothing, Testing is Everything
Tests are Nothing, Testing is Everything
 
21st Century Bricoleurs v4
21st Century Bricoleurs v421st Century Bricoleurs v4
21st Century Bricoleurs v4
 
Let's make this test suite run faster! SoftShake 2010
Let's make this test suite run faster! SoftShake 2010Let's make this test suite run faster! SoftShake 2010
Let's make this test suite run faster! SoftShake 2010
 
Comment j'ai mis ma suite de tests au régime en 5 minutes par jour
Comment j'ai mis ma suite de tests au régime en 5 minutes par jourComment j'ai mis ma suite de tests au régime en 5 minutes par jour
Comment j'ai mis ma suite de tests au régime en 5 minutes par jour
 
What Can I Do Now? (web 2.0 pedagogy) v4
What Can I Do Now? (web 2.0 pedagogy) v4What Can I Do Now? (web 2.0 pedagogy) v4
What Can I Do Now? (web 2.0 pedagogy) v4
 
The web as a tool - rather than a threat
The web as a tool - rather than a threatThe web as a tool - rather than a threat
The web as a tool - rather than a threat
 
Pluginblc11 post
Pluginblc11 postPluginblc11 post
Pluginblc11 post
 
Pluginblc11 post
Pluginblc11 postPluginblc11 post
Pluginblc11 post
 
The Community Tools You Need (Can't Be Built)
The Community Tools You Need (Can't Be Built)The Community Tools You Need (Can't Be Built)
The Community Tools You Need (Can't Be Built)
 
Infinite Possibilities - STC Summit 2015
Infinite Possibilities - STC Summit 2015Infinite Possibilities - STC Summit 2015
Infinite Possibilities - STC Summit 2015
 
Let's make this test suite run faster - Paris JUG 2011
Let's make this test suite run faster - Paris JUG 2011Let's make this test suite run faster - Paris JUG 2011
Let's make this test suite run faster - Paris JUG 2011
 
What's Wrong With This Picture?
What's Wrong With This Picture?What's Wrong With This Picture?
What's Wrong With This Picture?
 
Designing The Digital Experience
Designing The Digital ExperienceDesigning The Digital Experience
Designing The Digital Experience
 
Plugging in: Leveraging Technology For Engagement
Plugging in: Leveraging Technology For EngagementPlugging in: Leveraging Technology For Engagement
Plugging in: Leveraging Technology For Engagement
 
What Can I Do Now? (web 2.0 pedagogy) v3
What Can I Do Now? (web 2.0 pedagogy) v3What Can I Do Now? (web 2.0 pedagogy) v3
What Can I Do Now? (web 2.0 pedagogy) v3
 
What Can I Do Now? (web 2.0 pedagogy) v3.4
What Can I Do Now? (web 2.0 pedagogy) v3.4What Can I Do Now? (web 2.0 pedagogy) v3.4
What Can I Do Now? (web 2.0 pedagogy) v3.4
 
Infinite Possibilities - FIU Online's InnovEd Conference 2017
Infinite Possibilities - FIU Online's InnovEd Conference 2017Infinite Possibilities - FIU Online's InnovEd Conference 2017
Infinite Possibilities - FIU Online's InnovEd Conference 2017
 
What Can I Do Now? (web 2.0 pedagogy) v3.2
What Can I Do Now? (web 2.0 pedagogy) v3.2What Can I Do Now? (web 2.0 pedagogy) v3.2
What Can I Do Now? (web 2.0 pedagogy) v3.2
 
What Can I Do Now? (web 2.0 pedagogy) v3.9
What Can I Do Now? (web 2.0 pedagogy) v3.9What Can I Do Now? (web 2.0 pedagogy) v3.9
What Can I Do Now? (web 2.0 pedagogy) v3.9
 
What Can I Do Now? (web 2.0 pedagogy) v3.1
What Can I Do Now? (web 2.0 pedagogy) v3.1What Can I Do Now? (web 2.0 pedagogy) v3.1
What Can I Do Now? (web 2.0 pedagogy) v3.1
 

Recently uploaded

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc
 
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial IntelligenceRevolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Precisely
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Recently uploaded (20)

WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial IntelligenceRevolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 

Combinatorial software test design beyond pairwise testing

  • 1. Combinatorial Software Test Design (Beyond Pairwise Testing) 1
  • 2. Presented at STPCon October, 2010 By Justin Hunter CEO of Hexawise Cover slide photo credit: "Mike" Michael L. Baird, flickr.bairdphotos.com http://www.flickr.com/photos/mikebaird/2127310513 2
  • 3. L E M O B P R 3
  • 4. The Problem You can’t test everything. http://www.flickr.com/photos/maisonbisson/109211670/ 2
  • 5. The Problem “With so many possible tests to choose from...” http://www.flickr.com/photos/madabandon/344158061/ 2
  • 6. The Problem “Things might seem uncontrollable...” http://www.flickr.com/photos/maisonbisson/110867288/sizes/l/in/photostream/ 2
  • 7. The Problem ... discouraging, even. http://www.flickr.com/photos/cheo70/2702682262/ 2
  • 8. The Problem So, how can you select just a few tests... http://www.flickr.com/photos/krawcowicz/3830467093/ 8
  • 9. The Problem ... and still achieve adequate coverage? http://www.flickr.com/photos/editor/4861496460/ 9
  • 10. The Problem Wait. So this is a test design method to achieve greater coverage in fewer tests. Right? http://www.flickr.com/photos/johncarleton/4890239248 10
  • 11. Not Just Fewer, Better Tests If you prioritize But for those testers who speed, yes. You prioritize thoroughness, can achieve this method can also be greater coverage used to increase in fewer tests. thoroughness dramatically. It can help you prioritize around http://www.flickr.com/photos/legofenris/4212007962/ either goal. 11
  • 12. Not Just Fewer, Better Tests W ay The choice is fewer yours to make. tes ts! etter ligh tly b ith s than we (W rage cove e now.) hav http://www.flickr.com/photos/thomashawk/2402194148 http://www.flickr.com/photos/thomashawk/2402190314 12
  • 13. Not Just Fewer, Better Tests How can you create “curiously strong” test cases that will allow you to achieve either of these testing objectives? http://www.flickr.com/photos/photobunny_earl/2309612944/ 13
  • 14. I O N L UT S O 14
  • 15. The Solution This presentation describes an approach that will help you create powerful test cases. http://www.flickr.com/photos/maisonbisson/110867008/ 15
  • 16. Structured Variation To do so, use combinatorial test design methods that build structured variation into your test cases. 16
  • 17. Structured Variation? “Test One” http://www.flickr.com/photos/gelrodandjodie/148759732 17
  • 18. Structured Variation? This is not what I mean by structured variation. “Test Two” http://www.flickr.com/photos/gelrodandjodie/148759732 18
  • 19. Structured Variation? This is not what I mean by structured variation. “Test Three” http://www.flickr.com/photos/gelrodandjodie/148759732 19
  • 20. Structured Variation? So if that’s not structured variation, what is? 20
  • 21. Structured Variation Structured variation means http://www.flickr.com/photos/tk_five_0/2281768728 21
  • 22. Structured Variation ...each test will be different http://www.flickr.com/photos/criben/2635618391 22
  • 23. Structured Variation ...than the tests before it, and http://www.flickr.com/photos/horrgakx/3186982135 23
  • 24. Structured Variation ...testers will do their testing in different ways, http://www.flickr.com/photos/pixelrobber/2626213081 24
  • 25. Structured Variation ... and explore from more angles. http://www.flickr.com/photos/tk_five_0/2286693552 25
  • 26. Structured Variation “OK. Maximum variation is better than redundantly repeated redundant repetition. Got it. That it? Am I good to go now?” http://www.flickr.com/photos/johncarleton/4890241530 26
  • 27. Coverage No. There’s more. Much more. Remember coverage? http://www.flickr.com/photos/30431232@N03/3610846363 27
  • 28. Coverage Accidentally leaving gaps in coverage can lead to painful results. http://www.flickr.com/photos/dwfromthepeg/240921271/ 28
  • 29. Coverage Structured variation focuses on covering as much as possible... http://www.flickr.com/photos/localsurfer/38245658/ 29
  • 30. Coverage ... with as few resources as possible... http://www.flickr.com/photos/mikebaird/4177570453/ 30
  • 31. Coverage ...and in the most efficiently structured way possible. http://www.flickr.com/photos/dannohung/332507807 31
  • 32. F O & S U N C IO O T F U L E L ONA O I S T R A 32
  • 33. Single-Mode Faults Most software defects in production today can be triggered by just one test input. http://www.flickr.com/photos/vaguelyartistic/292083492 33
  • 34. Dual-Mode Faults Software bugs caused by the interaction of two test inputs are very common too. http://www.flickr.com/photos/kanaka/1798327442 34
  • 35. Triple-Mode Faults Defects that require three specific test inputs to trigger them are rare. http://www.flickr.com/photos/kanaka/1798327442 35
  • 36. Quadruple-Mode Faults Defects that require four more test inputs to trigger them are extremely rare. http://www.flickr.com/photos/novembering/3816524907 36
  • 37. Implication on Testing Strategy Aha! So, to find bugs efficiently, we should focus first on testing all possible combinations every set of TWO test inputs! http://www.flickr.com/photos/johncarleton/4890236762 37
  • 38. Implication on Testing Strategy That’s right, two. Focus on covering all possible two-way defects first. In fact, that’s so important, can you say it a second time? http://www.flickr.com/photos/soulsurfingcrew/2273300213 38
  • 39. Implication on Testing Strategy To find bugs efficiently we should focus first on testing all possible combinations of TWO test inputs! http://www.flickr.com/photos/johncarleton/4890236762 39
  • 40. I C S AN E C H M 40
  • 41. Combinations of Test Inputs The key words here are test inputs and combinations. http://www.flickr.com/photos/12tribu/2750040244 41
  • 42. Combinations of Test Inputs Test inputs will vary dramatically from project to project. 42
  • 43. Combinatorial Software Test Design Well, duh! They would, wouldn’t they? But how does it all work? http://www.flickr.com/photos/johncarleton/4377591743 43
  • 44. Combinatorial Software Test Design Here’s how it works: First, people are required to identify what test inputs should be tested. http://www.flickr.com/photos/mikebaird/3939231212 44
  • 45. Combinatorial Software Test Design Second, computer algorithms will take those inputs and quickly calculate the smallest possible number of tests required to meet the tester’s specified coverage objectives. 45
  • 46. Combinatorial Software Test Design ... and again, the algorithms will W ay create different solutions fewer depending upon tes ts! your testing objectives. etter ligh tly b ith s than we (W rage cove e now.) hav http://www.flickr.com/photos/thomashawk/2402194148 http://www.flickr.com/photos/thomashawk/2402190314 46
  • 47. Combinatorial Software Test Design W ay The woman wanting fewer fewer tests should tes ts! create 2-way tests r bette htly Wi ig th sl n we ha ve (AKA ( ha age t .) c over now “pairwise” or “AllPairs” tests) http://www.flickr.com/photos/thomashawk/2402194148 47
  • 48. Combinatorial Software Test Design This guy, wanting more coverage, should create more thorough combinatorial tests. (e.g., 3-way, 4-way, or multi-strength tests). http://www.flickr.com/photos/thomashawk/2402190314 48
  • 49. OT Y N STS H W NT ? E S G ND I A E D YH B 49
  • 50. Advantages of Combinatorial Software Test Design Test Design: Man vs. Machine (Once test inputs are known) Speed: Coverage: Efficiency: Human Weeks Gaps Way More Tests Computer Seconds 100%* Way Fewer Tests *100% of what? 100% of the desired test input combinations (e.g., all pairs of values for 2-way tests solutions, all triplets of possible values for 3-way solutions, etc.).
  • 51. Coverage Advantages Whatever your test inputs, combinatorial testing makes it easy and fast to cover all of the specific combinations of test inputs that you suspect are relatively important. http://www.flickr.com/photos/kanaka/1798327442 51
  • 52. Advantages of Combinatorial Software Test Design OK, but wouldn’t I have tested those relatively important combinations on my own? http://www.flickr.com/photos/johncarleton/4890239248
  • 53. Advantages of Combinatorial Software Test Design Perhaps, but maybe you... Forget Run out of time Use more tests Accidentally than needed repeat yourself 53
  • 54. Coverage Advantages Plus, using combinatorial testing, you’ll also get additional coverage from combinations that wouldn’t have made your list, including... http://www.flickr.com/photos/tk_five_0/2706278432 54
  • 55. Coverage Advantages ...specialized use cases, No, its way cooler than pareto. http://www.flickr.com/photos/subspace-eddy/2766810770/ 55
  • 56. Coverage Advantages ... unexpected combinations, http://www.flickr.com/photos/funky64/2875860978 56
  • 57. Coverage Advantages ...notable, combinations that capture your attention, http://www.flickr.com/photos/brb_photography/3221049399 57
  • 58. Coverage Advantages ...even combinations that rarely, if ever, actually occur, http://www.flickr.com/photos/grassvalleylarry/5536758 58
  • 59. Coverage Advantages ... and... combinations that will take you into different parts of the application (and different states) than you would have otherwise thought to explore. http://www.flickr.com/photos/28041049@N00/227352411 59
  • 60. Coverage Advantages Quickly testing unusual pairs is often surprisingly efficient and effective. Many unexpected two-way interactions cause problems. 60
  • 61. Constraint Handling Combinations that can’t ever appear in reality? Yes. No problem. You can prevent them from appearing in your plans with “Invalid Pairs.” http://www.flickr.com/photos/kaptainkobold/4741725788/lightbox/ 61
  • 62. E T ? Y Y E AD R 62
  • 63. Ready? I’m ready already! Let’s do this! http://www.flickr.com/photos/johncarleton/4890241530 63
  • 64. Hazards Not so fast. Hazards exist. To succeed, you must use common sense. http://www.flickr.com/photos/christianhaugen/3604106558 64
  • 65. Limitations And let’s not kid ourselves. Not just anyone can do http://www.flickr.com/photos/mikebaird/573368639 this stuff. 65
  • 66. Prerequisites It also requires good critical thinking skills. http://www.flickr.com/photos/johncarleton/9511759/ 66
  • 67. Prerequisites To thrive, you must be able to develop models about how the systems you’re exploring http://www.flickr.com/photos/10ch/3101275536/ 67 work.
  • 68. Practice Makes Perfect Training and practice will help you succeed. http://www.flickr.com/photos/mikebaird/2128091146
  • 69. Questions ? http://www.flickr.com/photos/nathaninsandiego/3647136579 69
  • 70. Ready? Let’s get started! http://www.flickr.com/photos/mikebaird/2996548709 70
  • 71. D I X P E N AP http://www.flickr.com/photos/dannohung/332507807 71
  • 72. Thank you Special thanks to the folks at: and its talented contributing photographers, including: Mike Baird ... and the inimitable John Carleton (who, let the record show, had no input into the cartoonish statements we added next to his striking expressions). 72
  • 73. Why 2-way Testing is Effective Multiple thorough studies show approximately 85% of defects in production could have been detected by simply testing all possible pairs of values. Text Source: “Combinatorial Testing” IEEE Computer, Aug, 2009. Rick Kuhn, Raghu Kacker, Jeff Lei, Justin Hunter
  • 74. Additional Information Excellent introductory articles and instructional videos: www.CombinatorialTesting.com Results of a ten-project study of effectiveness: https://www.hexawise.com/Combinatorial-Softwar-Testing-Case-Studies-IEEE-Computer-Kuhn-Kacker-Lei-Hunter.pdf Free trials of our firm’s combinatorial test design tool: www.hexawise.com/users/new (which stay free indefinitely for companies using 1-4 licenses). Also, please feel free to contact me if you have any questions. I’d be happy to quickly review a test plan or two, answer your questions, give quick pointers to help you run a pilot, etc. Seriously.... I enjoy helping people get started with this test design approach. Please don’t hesitate to reach out. There’s no charge and no catch. 74
  • 75. Practice Tips: 4-Step Process Four-Step Process to Design Efficient and Effective Tests Scope Level  of  Detail Passive  Field  Treatment 1.  Plan Be  clear  about  single  or  mul3ple: Acceptable  op3ons:   Dis3nguish  between  important  fields  (par3cularly  those  that   will  trigger  business  rules)  and  unimportant  fields  in  the   -­‐    Features  /  Func3ons  /  Capabili3es -­‐    High  level  “search  for  something” applica3on. -­‐    User  types -­‐    Medium  level  “search  for  a  book” Quickly  document  what  your  approach  will  be  towards   -­‐    Business  Units -­‐    Detailed  “search  for  ‘Catcher  in  the  Rye’   passive  fields.  You  might  consider:  ignore  them  (e.g.,  don’t   -­‐    H/W  or  S/W  Configura3ons            by  its  3tle” select  any  Values  in  your  plan)  or  a  3  Value  approach  such  as   “valid”  “Invalid  (then  fix)”  and  “Blank  (then  fix)” Configura3ons Users Ac3ons 2.  Create First  add  hardware  configura3ons Next,  add  mul3ple  types  of  users  (e.g.,   Start  with  Big  Common  Ac3ons  made  by  users administrator,  customer,  special  customer) AZer  comple3ng  Big  Common  Ac3ons,  circle  back  and  add   Next  add  soZware  configura3ons Consider  permission  /  authority  levels  of  admin   Small  Ac3ons  and  Excep3ons users  as  well  as  business  rules  that  different   Remember  some  ac3ons  may  be  system-­‐generated users  might  trigger Business  Rules Gap  Filling Itera3on 3.  Refine Select  Values  to  trigger  bus.  rules Iden3fy  gaps  by  analyzing  different  states,   Refine  longest  lists  of  Values;  reduce  their  numbers  by  using   equivalence  classes,  etc. orders  of  opera3ons,  decision-­‐tree  outcomes   Iden3fy  equivalence  classes sought  vs.  delivered  by  tests,  “gap  hun3ng”   Create  Tests  with  and  w/out  borderline  Values;  consider  cost/ conversa3ons  w/  SME’s,  etc. benefit  tradeoffs  of  addi3onal  test  design  refinements Test  for  boundary  values Fill  gaps  by  either  (i)  adding  Parameters  and/or   Consider  stopping  tes3ng  aZer  reaching  ~80%  coverage   Mark  constraints  /  invalid  pairs Values  or  (ii)  crea3ng  “one-­‐off”  tests.   Consider  2-­‐way,  3-­‐way,  and  Mixed-­‐Strength  op3ons Auto-­‐Scrip3ng Expected  Results Con3nuous  Improvement 4.  Execute       Add  auto-­‐scrip3ng  instruc3ons  once;   Export  the  tests  into  Excel  when  you’re  done   If  possible  measure  defects  found  per  tester  hour  “with  and   without  Hexawise”  and  share  the  results apply  those  instruc3ons  to  all  of  the  tests   itera3ng  the  plan in  your  plan  instantly Add  inputs  based  on  undetected  defects Add  complex  Expected  Results  in  Excel  post-­‐ Don’t  include  complex  Expected  Results   export Share  good,  proven,  plan  templates  with  others in  auto-­‐scripts   © Hexawise, 2010. All rights reserved. 26
  • 76. Practice Tips: Warning Signs “You might be headed for trouble if...” Scope Level  of  Detail Passive  Field  Treatment 1.  Plan ...  You  cannot  clearly  describe  both  the   ...  Parameters  with  the  most  Values  have  more   ...  You  cannot  clearly  describe  your  strategy  to  deal  with   scope  of  your  test  plan  and  what  will  be   Values  than  they  require.    365  values  for  “days   unimportant  details.    If  Values  will  impact  business  rules,  focus   leZ  out  of  scope. of  the  year”  is  bad.    Instead,  use  equivalence   on  them.    If  Values  don’t  impact  business  rules,  consider   class  Values  like  “weekend”  &  “weekday.”    When   ignoring  them.   in  doubt,  choose  more  Parameters  and  fewer   Values. Configura3ons Users Ac3ons 2.  Create ...  You  have  ignored  hardware  and   ...  You  have  not  included  all  the  different  types   ...  You  start  entering  Small  Ac3ons  (e.g.,  “search  for  a   soZware  configura3ons    without  first   of  users  necessary  to  trigger  different  business   hardback  science  book  by  author  name”)  before  you  enter  Big   confirming  this  approach  with   rules.    What  user  types  might  create  different   Ac3ons  (e.g.,  “Put  Something  in  Cart.    Buy  it.”)    First  go  from   stakeholders. outcomes?    Authority  level?  Age?  Loca3on?     beginning  to  end  at  a  high  level.    AZer  you’ve  done  that,  feel   Income?    Customer  status? free  to  add  more  details.   Business  Rules Gap  Filling Itera3on 3.  Refine ...  You  forget  to  iden3fy  invalid  pairs.          -­‐   ...  You  assume  that  the  test  condi3ons  coming   ...  You  forget  to  look  at  the  Coverage  Analysis  charts.    If  you   or  -­‐    ...  You  rely  only  on  Func3onal   out  of  Hexawise  will  be  100%  of  the  tests  you   achieve  80%  coverage  in  the  first  quarter  of  the  tests,  you   Requirements  and  Tech  Specs  w/out   should  run.    There  might  well  be  addi3onal   should  measure  the  cost/benefit  implica3ons  of  execu3ng  the   thinking  hard  yourself  and  asking   “one-­‐off”  things  that  you  should  test  and/or  a   last  3/4  of  the  tests. ques3ons  to  SME’s  about  business  rules   few  nega3ve  tests  to  design  by  hand. and  outcomes  that  are  not  yet  triggered. Auto-­‐Scrip3ng Expected  Results Con3nuous  Improvement 4.  Execute ...  You  add  detailed  Expected  Results  in   ...  You  invest  a  lot  of  3me  in  calcula3ng  and   ...  You  don’t  ask  (when  defects  that  the  tests  missed  are  found   the  tests.                                                                                                          -­‐   documen3ng  Expected  Results  before  you  have   post-­‐tes3ng)  “What  input  could  have  been  added  to  the  test   or  -­‐  ...  You  forget  that  this  feature  exists   plan  to  detected  this?”  “Should  I  add  that  input  to  the   and  find  yourself  typing  out  test-­‐by-­‐test   determined  your  “final  version”  Parameters  and   instruc3ons  one-­‐by-­‐one. Values.    Last  minute  addi3ons  to  inputs  will   Hexawise  test  plan  now  to  improve  it  in  advance  of  the  next   jumble  up  test  condi3ons  for  most  test  cases. 3me  it  is  used?”   © Hexawise, 2010. All rights reserved. 27