Costs Of Agile Testing
Upcoming SlideShare
Loading in...5
×
 

Costs Of Agile Testing

on

  • 1,654 views

ACCU 2010 Talk on various costs surrounding a move to agile testing. Demonstrates use of "cycle time" and "cost of delay"

ACCU 2010 Talk on various costs surrounding a move to agile testing. Demonstrates use of "cycle time" and "cost of delay"

Statistics

Views

Total Views
1,654
Views on SlideShare
1,626
Embed Views
28

Actions

Likes
1
Downloads
25
Comments
0

5 Embeds 28

http://www.linkedin.com 18
https://www.linkedin.com 4
http://www.slideshare.net 3
https://twitter.com 2
http://www.lmodules.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Costs Of Agile Testing Costs Of Agile Testing Presentation Transcript

  • The Costs of Agile Testing Schalk W. Cronjé ysb33r@gmail.com ACCU 2010 1 / 45 © Schalk W. Cronjé
  • Agile Today ACCU 2010 2 / 45 © Schalk W. Cronjé
  • Agile Today ● Mature ● Well-known methodologies agile today ● Embraced by many – Even if only by lip service ● Misunderstood by many ● Too easy to tick the boxes than to deliver value ● Tyranny of the urgent – Discipline ACCU 2010 3 / 45 © Schalk W. Cronjé
  • Institutions ● Sets of internalised rules Institution supported by values – Has tacit influence – Rules will be contested ● Shape understanding of Organisations social meaning + order – Provides a framework for performance ● Shapes of rights + duties People – Political authority – Economic opportunities An organisation can become "institutionalised" Institutions don't last forever ACCU 2010 4 / 45 © Schalk W. Cronjé
  • Scott's Model interpret Societal Institutions innovate invent, error negotiate diffuse, impose Organisational Fields invent, negotiate diffuse, impose Organisations invent, sanction negotiate behaviour diffuse, impose Actors Limitations of cognitive / social rationality (groups/individuals) Selective perception ACCU 2010 5 / 45 © Schalk W. Cronjé
  • "Excellent system qualities are a continuous management and engineering challenge, with no perfect solutions" Tom Gilb, Overload 85, June 2008 ACCU 2010 6 / 45 © Schalk W. Cronjé
  • The Economic View ACCU 2010 7 / 45 © Schalk W. Cronjé
  • "You are not here to produce software, you are here to provide value to the business" Schalk W. Cronjé ACCU 2010 8 / 45 © Schalk W. Cronjé
  • Why take an economic view? ● It helps to quantify the effects of multiple interacting variables ● It helps us to understand that the customer is not the only judge of value ● By using an economic framework it can allow us to maximise value, including – cycle time – product cost – development expense ● It helps to communicate with non-technical decision makers ACCU 2010 9 / 45 © Schalk W. Cronjé
  • Cost of Quality ● Appraisal costs – Discovering condition of hardware & 3rd-party software components ● Internal failure costs – Defects found before shipment ● External failure costs – Defects found after shipment ● Prevention costs – Costs for preventing all of the above ACCU 2010 10 / 45 © Schalk W. Cronjé
  • Batch Development Feature 1 Dev Feature 1 QA Feature 2 Dev Feature 2 QA Feature 3 Dev Feature 3 QA Feature 4 Dev Feature 4 QA delivered build ACCU 2010 11 / 45 © Schalk W. Cronjé
  • Batch Development Total Time = devt1 .. devt4 + qat1 …qat4 + fixdelay + fixtime + retest time Feature 1 Dev Feature 1 QA Feature 2 Dev Feature 2 QA Feature 3 Dev Feature 3 QA Feature 4 Dev Feature 4 QA delivered build defect trickle feed Bug Feature 5+ QA DB Retest fixes QA delivered build Feature 5+ Dev ACCU 2010 12 / 45 © Schalk W. Cronjé
  • Time Factors Building test understanding infrastructure specs Feature 1 Dev Feature 1 QA Feature 2 Dev Feature 2 QA Feature 3 Dev Feature 3 QA Feature 4 Dev Feature 4 QA basic build verification delivered build time from raising defect until it is available for testing defect trickle feed time to re-test Bug Feature 5+ QA DB Retest fixes QA delivered build Feature 5+ Dev ACCU 2010 13 / 45 © Schalk W. Cronjé
  • Feedback ● Faster feedback makes learning faster and more efficient ● Co-location improves communication ● Faster feedback provides a sense of control ● Large batches leads to slower feedback ACCU 2010 14 / 45 © Schalk W. Cronjé
  • Optimising Batch Size Total ● U-curve optimisation Cost Holding cost ● Holding cost includes delay Cost in time to learn Transaction cost ● Transaction cost includes overhead in releasing a build Batch size for testing Optimal Batch Size _________________________________________________ √ 2 x Cost per Batch x Total features in project -------------------------------------------------- Holding cost per feature ACCU 2010 15 / 45 © Schalk W. Cronjé
  • "If you only quantify one thing, quantify the cost of delay" Don Reinertsen ACCU 2010 16 / 45 © Schalk W. Cronjé
  • The Attention Principle "Time counts more than money" Don Reinertsen ACCU 2010 17 / 45 © Schalk W. Cronjé
  • Building an Agile Testing Model ACCU 2010 18 / 45 © Schalk W. Cronjé
  • Simple Flow Model Specify Develop QA Isn't this Kanban waterfall? Board Ready Spec Spec Dev Dev QA QA Released Complete Complete Complete ACCU 2010 19 / 45 © Schalk W. Cronjé
  • Reducing Learning Time ● Writing test specifications after the development is costly in time – Knowledge decay – Leads to incomplete designs ● Move the test specification up-front before any development starts – This is part of requirements discovery – It broadens the perspective of the programmer – Allows us to distinguish between automated checks and human testing ACCU 2010 20 / 45 © Schalk W. Cronjé
  • Reducing Learning Time Specify Develop QA Test specification +Time to specify -Time to re-work +Time to develop -Time to re-test ACCU 2010 21 / 45 © Schalk W. Cronjé
  • Quantifying Learning Time ● Can you convert time saved to a monetary value? Before After Avg cycle days to spec 2 4 Avg cycle days to develop 8 10 Avg cycle days to test 13 8 Avg rework days during 5 1 test Total 23 22 Cost $8,118.00 $7,872.00 (*) Data in graph is from an insignificant data set and provides no conclusive proof. It is for illustrative purposes only ACCU 2010 22 / 45 © Schalk W. Cronjé
  • Change the Flow Model Specify Develop & Check Verify ● Ensure the all automated checks are included in the development stage ● Free up a Verification stage for exploratory testing and validating that process has been fulfilled. ACCU 2010 23 / 45 © Schalk W. Cronjé
  • Change the Flow Model Specify Develop & Check Verify +Time to develop -Time to re-work +More testing people -Time to re-test ACCU 2010 24 / 45 © Schalk W. Cronjé
  • The Role of the Tester ● Up-front involvement to build in quality ● Help to build test infrastructure up-front during the development phase ● Work with developer to critically review what is being automated. ACCU 2010 25 / 45 © Schalk W. Cronjé
  • Apply Lean Principles Specify Develop & Check Verify ● Make the system pull-based ● Reduce the batch size ● Limit the work-in-progress ● One feature end-to-end ● If the flow breaks, fix it immediately ACCU 2010 26 / 45 © Schalk W. Cronjé
  • Apply Lean Principles Specify Develop & Check Verify -Average Cycle time +Time to develop -Inherited delay cost +Time to test -Waiting time ACCU 2010 27 / 45 © Schalk W. Cronjé
  • Team Composition ACCU 2010 28 / 45 © Schalk W. Cronjé
  • The Classic Dev-QA Team ● This is the easiest team to adapt ● Requires the will to change to a new working practices ● If teams are distributed then groups of programmers and testers must be co-located. ACCU 2010 29 / 45 © Schalk W. Cronjé
  • The "No Testers" Team ● All development + testing done by same people. ● Requires T-skilled people ● Initial slow down of cycle time due to learning of new skills OR reduction in quality delivered ACCU 2010 30 / 45 © Schalk W. Cronjé
  • Paired Teams ● Pair up people with different skills to work on same feature – Feature teams are not a new concept ● Could be perceived to have higher person cost per feature – Instead of distributing the person cost over multiple queues, the cost is combined into a single queue – Can actually lead to reduced cycle time. ACCU 2010 31 / 45 © Schalk W. Cronjé
  • Real-world measurements ACCU 2010 32 / 45 © Schalk W. Cronjé
  • Real-world measurements Specify Develop & Check Verify M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 33 / 45 © Schalk W. Cronjé
  • What is wrong? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 34 / 45 © Schalk W. Cronjé
  • What is wrong? Excessive long time between feature availability and testing. Is there enough testing bandwidth? Or is this team just cheating? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 35 / 45 © Schalk W. Cronjé
  • What is wrong? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 36 / 45 © Schalk W. Cronjé
  • What is wrong? Is this 2 days due to re-work? Is there a problem in the development phase? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 37 / 45 © Schalk W. Cronjé
  • How do these teams compare? M1 M2 NF1 NF2 READY 4 6 5 1h SPEC 1 0.5 6 0.5 SPEC/Completed 2 0.5 1 0.5 DEV + Check 2 1 13 4 DEV + Check / 3 6 4 18 Completed VERIFY 3 7 8 1 VERIFY/Completed 2 5 6 18 Blockages 0.5 1h 2 1h Total 17 26 43 42 Average cycle days per feature ACCU 2010 38 / 45 © Schalk W. Cronjé
  • Exercise ● Analyse the previous listed teams and comment on their process (with liberty) ● Is it possible to identify points where cost of delay can be calculated? ● How would you identify the holding costs and transaction costs? ● Can you improve testing/quality control and what is the expected and hidden costs? ACCU 2010 39 / 45 © Schalk W. Cronjé
  • The End Game ACCU 2010 40 / 45 © Schalk W. Cronjé
  • The institution hits back speaking at public Societal Institutions conferences, success publishing articles propagates from known champions slow diffusion or no diffusion Organisational Fields invent, negotiate diffuse, impose attempting to Organisations introduce change sanction behaviour imposing existing failed processes Actors Limitations of cognitive / social rationality (groups/individuals) Selective perception ACCU 2010 41 / 45 © Schalk W. Cronjé
  • Making Testing more Agile ● Reduce learning time - bring test specification forward ● Pair up developers and testers ● Be intelligent about what is being automated. ACCU 2010 42 / 45 © Schalk W. Cronjé
  • Investing in Agile Testing ● Building infrastructure ● Good testers demand equivalent remuneration to good programmers ● Put measurements in place ● Cross-train people to create T-skilled sets ● Smooth the Flow ● Recognise that transition will take months, even a year ACCU 2010 43 / 45 © Schalk W. Cronjé
  • Cost of Agile Testing ● Appraisal costs ● Internal failure costs ● External failure costs ● Prevention costs – Checks against external deliverables – Automating checks (TDD) – Building the capacity for addressing critical issues without too much interruption to other development. – Training & Practising the mind-set ACCU 2010 44 / 45 © Schalk W. Cronjé
  • Summary ● Balanced teams reduce costs ● Shared problem-solving can reduce feature cycle time ● Reduced cycle time will have a direct influence on delay cost. ● Following a process does not make you testing agile ● Measure to improve ● You will operate within an institutional context ACCU 2010 45 / 45 © Schalk W. Cronjé