Your SlideShare is downloading. ×
0
The Costs of Agile Testing


               Schalk W. Cronjé
              ysb33r@gmail.com



                           ...
Agile Today



                          ACCU 2010
2 / 45
                       © Schalk W. Cronjé
Agile Today

  ●      Mature
  ●      Well-known methodologies          agile
                                           t...
Institutions
  ●      Sets of internalised rules                 Institution
         supported by values
         –   Has...
Scott's Model
                                                                        interpret
  Societal Institutions   ...
"Excellent system qualities are a
    continuous management and engineering
      challenge, with no perfect solutions"

 ...
The Economic View




                            ACCU 2010
7 / 45
                         © Schalk W. Cronjé
"You are not here to produce software,
             you are here to provide value
                   to the business"

   ...
Why take an economic view?

  ●      It helps to quantify the effects of multiple
         interacting variables
  ●      ...
Cost of Quality

  ●       Appraisal costs
          –   Discovering condition of hardware & 3rd-party
              softw...
Batch Development
    Feature 1 Dev               Feature 1 QA
    Feature 2 Dev               Feature 2 QA
    Feature 3 ...
Batch Development
Total Time = devt1 .. devt4 + qat1 …qat4 + fixdelay + fixtime + retest time

      Feature 1 Dev        ...
Time Factors
                                                                             Building test
                  ...
Feedback

  ●       Faster feedback makes learning faster and
          more efficient
  ●       Co-location improves comm...
Optimising Batch Size
                                                 Total
  ●       U-curve optimisation               ...
"If you only quantify one thing,
             quantify the cost of delay"


                              Don Reinertsen

...
The Attention Principle



          "Time counts more than money"


                           Don Reinertsen




       ...
Building an Agile Testing
             Model



                           ACCU 2010
18 / 45
                        © Sch...
Simple Flow Model
          Specify                  Develop                        QA



                                ...
Reducing Learning Time

  ●       Writing test specifications after the
          development is costly in time
      –   ...
Reducing Learning Time
          Specify             Develop               QA


                                        Te...
Quantifying Learning Time

      ●      Can you convert time saved to a monetary
             value?                      ...
Change the Flow Model
          Specify    Develop & Check   Verify



  ●   Ensure the all automated checks are included
...
Change the Flow Model
          Specify                 Develop & Check      Verify




           +Time to develop       ...
The Role of the Tester

  ●       Up-front involvement to build in quality
  ●       Help to build test infrastructure up-...
Apply Lean Principles
          Specify      Develop & Check     Verify



  ●   Make the system pull-based
  ●   Reduce t...
Apply Lean Principles
          Specify             Develop & Check      Verify




                                      ...
Team Composition



                            ACCU 2010
28 / 45
                         © Schalk W. Cronjé
The Classic Dev-QA Team

  ●       This is the easiest team to adapt
  ●       Requires the will to change to a new workin...
The "No Testers" Team

  ●       All development + testing done by same
          people.
  ●       Requires T-skilled peo...
Paired Teams

  ●       Pair up people with different skills to work on
          same feature
      –   Feature teams are...
Real-world measurements



                       ACCU 2010
32 / 45
                    © Schalk W. Cronjé
Real-world measurements
            Specify            Develop & Check             Verify


                         M1   ...
What is wrong?


                      M1           M2          NF1          NF2
READY                  4            6    ...
What is wrong?
Excessive long time between feature availability and testing. Is there enough
testing bandwidth? Or is this...
What is wrong?


                      M1           M2          NF1          NF2
READY                  4            6    ...
What is wrong?
Is this 2 days due to re-work? Is there a problem in the development phase?



                            ...
How do these teams compare?


                      M1           M2          NF1          NF2
READY                  4    ...
Exercise

  ●       Analyse the previous listed teams and
          comment on their process (with liberty)
  ●       Is i...
The End Game



                            ACCU 2010
40 / 45
                         © Schalk W. Cronjé
The institution hits back
                                                                           speaking at public
  ...
Making Testing more Agile

  ●       Reduce learning time - bring test specification
          forward
  ●       Pair up d...
Investing in Agile Testing

  ●       Building infrastructure
  ●       Good testers demand equivalent
          remunerat...
Cost of Agile Testing

  ●       Appraisal costs
  ●       Internal failure costs
  ●       External failure costs
  ●    ...
Summary

  ●       Balanced teams reduce costs
  ●       Shared problem-solving can reduce feature
          cycle time
  ...
Upcoming SlideShare
Loading in...5
×

Costs Of Agile Testing

1,379

Published on

ACCU 2010 Talk on various costs surrounding a move to agile testing. Demonstrates use of "cycle time" and "cost of delay"

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,379
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
27
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Costs Of Agile Testing"

  1. 1. The Costs of Agile Testing Schalk W. Cronjé ysb33r@gmail.com ACCU 2010 1 / 45 © Schalk W. Cronjé
  2. 2. Agile Today ACCU 2010 2 / 45 © Schalk W. Cronjé
  3. 3. Agile Today ● Mature ● Well-known methodologies agile today ● Embraced by many – Even if only by lip service ● Misunderstood by many ● Too easy to tick the boxes than to deliver value ● Tyranny of the urgent – Discipline ACCU 2010 3 / 45 © Schalk W. Cronjé
  4. 4. Institutions ● Sets of internalised rules Institution supported by values – Has tacit influence – Rules will be contested ● Shape understanding of Organisations social meaning + order – Provides a framework for performance ● Shapes of rights + duties People – Political authority – Economic opportunities An organisation can become "institutionalised" Institutions don't last forever ACCU 2010 4 / 45 © Schalk W. Cronjé
  5. 5. Scott's Model interpret Societal Institutions innovate invent, error negotiate diffuse, impose Organisational Fields invent, negotiate diffuse, impose Organisations invent, sanction negotiate behaviour diffuse, impose Actors Limitations of cognitive / social rationality (groups/individuals) Selective perception ACCU 2010 5 / 45 © Schalk W. Cronjé
  6. 6. "Excellent system qualities are a continuous management and engineering challenge, with no perfect solutions" Tom Gilb, Overload 85, June 2008 ACCU 2010 6 / 45 © Schalk W. Cronjé
  7. 7. The Economic View ACCU 2010 7 / 45 © Schalk W. Cronjé
  8. 8. "You are not here to produce software, you are here to provide value to the business" Schalk W. Cronjé ACCU 2010 8 / 45 © Schalk W. Cronjé
  9. 9. Why take an economic view? ● It helps to quantify the effects of multiple interacting variables ● It helps us to understand that the customer is not the only judge of value ● By using an economic framework it can allow us to maximise value, including – cycle time – product cost – development expense ● It helps to communicate with non-technical decision makers ACCU 2010 9 / 45 © Schalk W. Cronjé
  10. 10. Cost of Quality ● Appraisal costs – Discovering condition of hardware & 3rd-party software components ● Internal failure costs – Defects found before shipment ● External failure costs – Defects found after shipment ● Prevention costs – Costs for preventing all of the above ACCU 2010 10 / 45 © Schalk W. Cronjé
  11. 11. Batch Development Feature 1 Dev Feature 1 QA Feature 2 Dev Feature 2 QA Feature 3 Dev Feature 3 QA Feature 4 Dev Feature 4 QA delivered build ACCU 2010 11 / 45 © Schalk W. Cronjé
  12. 12. Batch Development Total Time = devt1 .. devt4 + qat1 …qat4 + fixdelay + fixtime + retest time Feature 1 Dev Feature 1 QA Feature 2 Dev Feature 2 QA Feature 3 Dev Feature 3 QA Feature 4 Dev Feature 4 QA delivered build defect trickle feed Bug Feature 5+ QA DB Retest fixes QA delivered build Feature 5+ Dev ACCU 2010 12 / 45 © Schalk W. Cronjé
  13. 13. Time Factors Building test understanding infrastructure specs Feature 1 Dev Feature 1 QA Feature 2 Dev Feature 2 QA Feature 3 Dev Feature 3 QA Feature 4 Dev Feature 4 QA basic build verification delivered build time from raising defect until it is available for testing defect trickle feed time to re-test Bug Feature 5+ QA DB Retest fixes QA delivered build Feature 5+ Dev ACCU 2010 13 / 45 © Schalk W. Cronjé
  14. 14. Feedback ● Faster feedback makes learning faster and more efficient ● Co-location improves communication ● Faster feedback provides a sense of control ● Large batches leads to slower feedback ACCU 2010 14 / 45 © Schalk W. Cronjé
  15. 15. Optimising Batch Size Total ● U-curve optimisation Cost Holding cost ● Holding cost includes delay Cost in time to learn Transaction cost ● Transaction cost includes overhead in releasing a build Batch size for testing Optimal Batch Size _________________________________________________ √ 2 x Cost per Batch x Total features in project -------------------------------------------------- Holding cost per feature ACCU 2010 15 / 45 © Schalk W. Cronjé
  16. 16. "If you only quantify one thing, quantify the cost of delay" Don Reinertsen ACCU 2010 16 / 45 © Schalk W. Cronjé
  17. 17. The Attention Principle "Time counts more than money" Don Reinertsen ACCU 2010 17 / 45 © Schalk W. Cronjé
  18. 18. Building an Agile Testing Model ACCU 2010 18 / 45 © Schalk W. Cronjé
  19. 19. Simple Flow Model Specify Develop QA Isn't this Kanban waterfall? Board Ready Spec Spec Dev Dev QA QA Released Complete Complete Complete ACCU 2010 19 / 45 © Schalk W. Cronjé
  20. 20. Reducing Learning Time ● Writing test specifications after the development is costly in time – Knowledge decay – Leads to incomplete designs ● Move the test specification up-front before any development starts – This is part of requirements discovery – It broadens the perspective of the programmer – Allows us to distinguish between automated checks and human testing ACCU 2010 20 / 45 © Schalk W. Cronjé
  21. 21. Reducing Learning Time Specify Develop QA Test specification +Time to specify -Time to re-work +Time to develop -Time to re-test ACCU 2010 21 / 45 © Schalk W. Cronjé
  22. 22. Quantifying Learning Time ● Can you convert time saved to a monetary value? Before After Avg cycle days to spec 2 4 Avg cycle days to develop 8 10 Avg cycle days to test 13 8 Avg rework days during 5 1 test Total 23 22 Cost $8,118.00 $7,872.00 (*) Data in graph is from an insignificant data set and provides no conclusive proof. It is for illustrative purposes only ACCU 2010 22 / 45 © Schalk W. Cronjé
  23. 23. Change the Flow Model Specify Develop & Check Verify ● Ensure the all automated checks are included in the development stage ● Free up a Verification stage for exploratory testing and validating that process has been fulfilled. ACCU 2010 23 / 45 © Schalk W. Cronjé
  24. 24. Change the Flow Model Specify Develop & Check Verify +Time to develop -Time to re-work +More testing people -Time to re-test ACCU 2010 24 / 45 © Schalk W. Cronjé
  25. 25. The Role of the Tester ● Up-front involvement to build in quality ● Help to build test infrastructure up-front during the development phase ● Work with developer to critically review what is being automated. ACCU 2010 25 / 45 © Schalk W. Cronjé
  26. 26. Apply Lean Principles Specify Develop & Check Verify ● Make the system pull-based ● Reduce the batch size ● Limit the work-in-progress ● One feature end-to-end ● If the flow breaks, fix it immediately ACCU 2010 26 / 45 © Schalk W. Cronjé
  27. 27. Apply Lean Principles Specify Develop & Check Verify -Average Cycle time +Time to develop -Inherited delay cost +Time to test -Waiting time ACCU 2010 27 / 45 © Schalk W. Cronjé
  28. 28. Team Composition ACCU 2010 28 / 45 © Schalk W. Cronjé
  29. 29. The Classic Dev-QA Team ● This is the easiest team to adapt ● Requires the will to change to a new working practices ● If teams are distributed then groups of programmers and testers must be co-located. ACCU 2010 29 / 45 © Schalk W. Cronjé
  30. 30. The "No Testers" Team ● All development + testing done by same people. ● Requires T-skilled people ● Initial slow down of cycle time due to learning of new skills OR reduction in quality delivered ACCU 2010 30 / 45 © Schalk W. Cronjé
  31. 31. Paired Teams ● Pair up people with different skills to work on same feature – Feature teams are not a new concept ● Could be perceived to have higher person cost per feature – Instead of distributing the person cost over multiple queues, the cost is combined into a single queue – Can actually lead to reduced cycle time. ACCU 2010 31 / 45 © Schalk W. Cronjé
  32. 32. Real-world measurements ACCU 2010 32 / 45 © Schalk W. Cronjé
  33. 33. Real-world measurements Specify Develop & Check Verify M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 33 / 45 © Schalk W. Cronjé
  34. 34. What is wrong? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 34 / 45 © Schalk W. Cronjé
  35. 35. What is wrong? Excessive long time between feature availability and testing. Is there enough testing bandwidth? Or is this team just cheating? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 35 / 45 © Schalk W. Cronjé
  36. 36. What is wrong? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 36 / 45 © Schalk W. Cronjé
  37. 37. What is wrong? Is this 2 days due to re-work? Is there a problem in the development phase? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 37 / 45 © Schalk W. Cronjé
  38. 38. How do these teams compare? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 38 / 45 © Schalk W. Cronjé
  39. 39. Exercise ● Analyse the previous listed teams and comment on their process (with liberty) ● Is it possible to identify points where cost of delay can be calculated? ● How would you identify the holding costs and transaction costs? ● Can you improve testing/quality control and what is the expected and hidden costs? ACCU 2010 39 / 45 © Schalk W. Cronjé
  40. 40. The End Game ACCU 2010 40 / 45 © Schalk W. Cronjé
  41. 41. The institution hits back speaking at public Societal Institutions conferences, success publishing articles propagates from known champions slow diffusion or no diffusion Organisational Fields invent, negotiate diffuse, impose attempting to Organisations introduce change sanction behaviour imposing existing failed processes Actors Limitations of cognitive / social rationality (groups/individuals) Selective perception ACCU 2010 41 / 45 © Schalk W. Cronjé
  42. 42. Making Testing more Agile ● Reduce learning time - bring test specification forward ● Pair up developers and testers ● Be intelligent about what is being automated. ACCU 2010 42 / 45 © Schalk W. Cronjé
  43. 43. Investing in Agile Testing ● Building infrastructure ● Good testers demand equivalent remuneration to good programmers ● Put measurements in place ● Cross-train people to create T-skilled sets ● Smooth the Flow ● Recognise that transition will take months, even a year ACCU 2010 43 / 45 © Schalk W. Cronjé
  44. 44. Cost of Agile Testing ● Appraisal costs ● Internal failure costs ● External failure costs ● Prevention costs – Checks against external deliverables – Automating checks (TDD) – Building the capacity for addressing critical issues without too much interruption to other development. – Training & Practising the mind-set ACCU 2010 44 / 45 © Schalk W. Cronjé
  45. 45. Summary ● Balanced teams reduce costs ● Shared problem-solving can reduce feature cycle time ● Reduced cycle time will have a direct influence on delay cost. ● Following a process does not make you testing agile ● Measure to improve ● You will operate within an institutional context ACCU 2010 45 / 45 © Schalk W. Cronjé
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×