Lean qa enabling quality through tools and technology lean quality assurance v3.3 spin 17 nov

1,311 views
1,215 views

Published on

This is a conference lecture, held March 5 2013 at Scandinavian Developers Conference, Gothenburg, Sweden. It is also the basis for a 1 to 3 day course on the same subject. Main theme is that there are much more cost effective ways to get all qualities assured in systems than 'testing them in'.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,311
On SlideShare
0
From Embeds
0
Number of Embeds
67
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Lean qa enabling quality through tools and technology lean quality assurance v3.3 spin 17 nov

  1. 1. ‘Lean Quality Assurance’ :Much More Effective QA Methods than Testing Scandinavian Developers Conference, Gothenburg, Sweden March 5th 2013, 1030 to 1120 (50 mins) By Tom Gilb Tom@Gilb.com www.GILB.com © www.Gilb.com Version 8- Sep. 2010 1
  2. 2. Main Take-away PointsQuality Assurance is far more than ‘test’, and it can be far more cost-effective‘Quality’ is far more than ‘bugs’You probably have a lot to learn, if you want real competitive quality © www.Gilb.com Version 8- Sep. 2010 2
  3. 3. Begin: Quality Assuranceis far more than ‘test’and it can be far more cost- effective © www.Gilb.com Version 8- Sep. 2010 3
  4. 4. Test and Inspection Effectiveness MeasuresCapers Jones © www.Gilb.com Version 8- Sep. 2010 4
  5. 5. © www.Gilb.com Version 8- Sep. 2010 5
  6. 6. © www.Gilb.com Version 8- Sep. 2010 6
  7. 7. © www.Gilb.com Version 8- Sep. 2010 7
  8. 8. © www.Gilb.com Version 8- Sep. 2010 8
  9. 9. All Defects Best Practice Testing CombinedDesign Remaining Defects © www.Gilb.com Version 8- Sep. 2010 9
  10. 10. Little hope of ‘zero defects’“Between8 10anddefect removalstages requiredto achieveremovaleffectiveness of95% ” © www.Gilb.com Version 8- Sep. 2010 10
  11. 11. Testing Capability (C. Jones)120 %100 80 60 40 In Field 20 Test 0 1st year 2nd year 5th year © www.Gilb.com Version 8- Sep. 2010 11
  12. 12. Defect Detection Capability (C. Jones)120 %100 In Field 80 Test 60 Inspection 40 20 0 1st year 2nd year 5th year © www.Gilb.com Version 8- Sep. 2010 12
  13. 13. IBM Defect Avoidance Experience120 %100 In Field 80 Test Inspection 60 Defect Prevention Effectiveness 40 20 0 1st year 2nd year 5th year © www.Gilb.com Version 8- Sep. 2010 13
  14. 14. Design Quality In © www.Gilb.com Version 8- Sep. 2010 14
  15. 15. You don’t get quality by testing it in © www.Gilb.com Version 8- Sep. 2010 15
  16. 16. but by ‘Engineering’ Quality In Reliability PerformanceWork hours Security$ € Kr. Usability Maintenance © www.Gilb.com Version 8- Sep. 2010 16
  17. 17. Setting Quality Goals simple exampleUsability.Learn Scale: average time to Learn how to operate the computer, from .. to .. Status [today] 3 hours Goal [next year] 10 min. © www.Gilb.com Version 8- Sep. 2010 17
  18. 18. Designing to meet Quality within Costs A systematic Quantitative Method Using ‘Impact Estimation’ Tables Design IdeasQualities€$ © www.Gilb.com Version 8- Sep. 2010 18
  19. 19. Quality Assurance is far more than ‘test’and, QA can be far more cost-effective Than ‘test’ approachesCost-Effective = Quality Delivered / Cost © www.Gilb.com Version 8- Sep. 2010 19
  20. 20. Quality is far more than ‘bugs’ © www.Gilb.com Version 8- Sep. 2010 20
  21. 21. System Performance Capacity Quality ‘How ‘How Well’ Much’ Resource Saving ‘Efficiency’ © www.Gilb.com Version 8- Sep. 2010 21
  22. 22. Qualities are many and variable •  Learning Usability •  Doing •  Error Rate •  Portability Adaptability •  Enhancability •  Compatibility •  Threat Type and Frequency Integrity •  Security Mitigation •  Reliability Availability •  Maintainability (fault fix speed)http://www.gilb.com/tiki-download_file.php?fileId=26 www.Gilb.com/Downloads/Esstentials/Ce Ch5 © www.Gilb.com Version 8- Sep. 2010 22
  23. 23. Quantify the Quality to ‘Assure’ It“…I often say thatwhen you can measure what you are speaking about,and express it in numbers,you know something about it;but when you cannot measure it,when you cannot express it in numbers,your knowledge is of a meagre and unsatisfactorykind;…” - Lord Kelvin, 1893 © www.Gilb.com Version 8- Sep. 2010 23
  24. 24. Main Idea, again• There are many much smarter ways to get quality than ‘testing it in’• For example, at Google ….. © www.Gilb.com Version 8- Sep. 2010 24
  25. 25. Google, is now experimenting in real Google projects. No Professional Testers He has totally eliminated the use of professional testers on his team, replacing them with a set of more cost effective means for ‘testing’ the software.. (Construx Summit Talk, Oct 2011, Seattle) © www.Gilb.com Version 8- Sep. 2010 25
  26. 26. Google/Whittaker Summary 2011“Where does testing fit in this world” JW © www.Gilb.com Version 8- Sep. 2010 26
  27. 27. However• Optimizing the testing process is great….• But, –  a lean, upstream, proactive approach is even far more powerful• (for getting critical qualities, cost- effectively) © www.Gilb.com Version 8- Sep. 2010 27
  28. 28. 7Competitive Lean QA methods to Learn © www.Gilb.com Version 8- Sep. 2010 28
  29. 29. 1. Stakeholders Decide Qualities (Geoff Thompson mentioned ‘Stakeholders’ earlier today) Suzanne Robertson & James Robertson © www.Gilb.com Version 8- Sep. 2010 29
  30. 30. 2. © www.Gilb.com Version 8- Sep. 2010 30
  31. 31. CMM Level 4 Basis • “As I see it Tom Gilb was the inspiration for much of what is defined in CMM Level 4.”•  Ron Radice (CMM Inventor at IBM) 1996 Salt lake City (agreed orally by Watts Humpreys - his IBM Director)•  stt@stt.com, www.stt.com © www.Gilb.com Version 8- Sep. 2010 31
  32. 32. Lack of clear top level project objectives has seen real projects fail for $100+ million: personal experience, real case Quantified Objectives (in Planguage), What they should have done Bad Objectives, for 8 years 8 years earlier!1. Central to The Corporations business strategy is to be Robustness.Testability:the world’s premier integrated <domain> serviceprovider. Type: Software Quality Requirement.2. Will provide a much more efficient user experience Version: 20 Oct 2006-10-203. Dramatically scale back the time frequently needed Status: Demo draft,after the last data is acquired to time align, depth correct, Stakeholder: {Operator, Tester}.splice, merge, recompute and/or do whatever else isneeded to generate the desired products Ambition: Rapid-duration automatic testing of <critical complex tests>, with extreme operator setup4. Make the system much easier to understand and use and initiation.than has been the case for previous system.5. A primary goal is to provide a much more productive Scale: the duration of a definedsystem development environment than was previously [Volume] of testing, or a defined [Type],the case. by a defined [Skill Level] of system operator, under defined [Operating6. Will provide a richer set of functionality for supporting Conditions].next-generation logging tools and applications.7. Robustness is an essential system requirement (see Goal [All Customer Use, Volume = 1,000,000 datapartial rewrite in example at right) items, Type = WireXXXX Vs DXX, Skill = First Time Novice, Operating Conditions = Field, {Sea Or8. Major improvements in data quality over currentpractice Desert}. <10 mins. 5 March 2013 © Gilb.com Gilb.com 32 © www. Version 8- Sep. 2010 32
  33. 33. VALUE CLARITY: Quantify the most-critical project objectives on day 1P&L-Consistency&T P&L: Scale: total adjustments btw Flash/ times per day the intraday P&L process is delayed more than 0.5Predict and Actual (T+1) signed off P&L. per day. Past 60 Goal: sec.15 Operational-Control.Timely.Trade-Bookings Scale: number of trades per day that are not booked on trade date. Past [April 20xx]Speed-To-Deliver: Scale: average Calendar days needed from 20 ?New Idea Approved until Idea Operational, for given Tasks, ongiven Markets. Front-Office-Trade-Management-Efficiency Scale: Time fromPast [2009, Market = EURex, Task =Bond Execution] 2-3 Ticket Launch to trade updating real-time risk viewmonths ? Past [20xx, Function = Risk Mgt, Region = Global] ~ 80s +/-Goal [Deadline =End 20xz, Market = EURex, Task =Bond 45s ??Execution] 5 days Goal [End 20xz, Function = Risk Mgt, Region = Global] ~ 50% better?Operational-Control: Scale: % of trades per day, where the Managing Risk – Accurate – Consolidated – Real Timecalculated economic difference between OUR CO andMarketplace/Clients, is less than “1 Yen”(or equivalent). Risk.Cross-Product Scale: % of financial products that riskPast [April 20xx] 10% change this to 90% NH Goal [Dec. 20xy] metrics can be displayed in a single position blotter in a way100% appropriate for the trader (i.e. – around a benchmark vs. across the curve).Operational-Control.Consistent: Scale: % of defined [Trades] Past [April 20xx] 0% 95%. Goal [Dec. 20xy] 100%failing full STP across the transaction cycle. Past [April 20xx, Risk.Low-latency Scale: number of times per day the intradayTrades=Voice Trades] 95% risk metrics is delayed by more than 0.5 sec. Past [April 20xx, NA]Past [April 20xx, Trades=eTrades] 93% 1% Past [April 20xx, EMEA] ??% Past [April 20xx, AP] 100%Goal [April 20xz, Trades=Voice Trades] <95 ± 2%> Goal [Dec. 20xy] 0%Goal [April 20xz, Trades=eTrades] 98.5 ± 0.5 % Risk.Accuracy Risk. user-configurable Scale: ??? pretty binary – feature isOperational-Control.Timely.End&OvernightP&L Scale: numberthere or not – how do we represent?of times, per quarter, the P&L information is not delivered timely to Past [April 20xx] 1% Goal [Dec. 20xy] 0%the defined [Bach-Run]. Operational Cost Efficiency Scale: <Increased efficiencyPast [April 20xx, Batch-Run=Overnight] 1 Goal [Dec. 20xy, Batch- (Straight through processing STP Rates )>Run=Overnight] <0.5> Past [April 20xx, Batch-Run= T+1] 1 Goal[Dec. 20xy, Batch-Run=End-Of-Day, Delay<1hour] 1 Cost-Per-Trade Scale: % reduction in Cost-Per-Trade Goal (EOY 20xy, cost type = I 1 – REGION = ALL) Reduce costOperational-Control.Timely.IntradayP&L Scale: number of by 60% (BW) Goal (EOY 20xy, cost type = I 2 – REGION = ALL) Reduce cost 5 March 2013 © Gilb.com Gilb.com © www. by x % Version 8- Sep. 2010 33 Goal (EOY 20xy, cost type = E1 – REGION = ALL) Reduce cost 33
  34. 34. Example of Estimating the ‘Business Value’of a Technical IT System Improvement (20xx) This is an example made to reason about specification standards and is not supposed to be a real spec. Just realistic.5 March 2013 © Gilb.com Gilb.com 34 © www. Version 8- Sep. 2010 34
  35. 35. 3. Assuring that Designs give Qualities Usability © www.Gilb.com Version 8- Sep. 2010 35
  36. 36. 4. Measure Quality Levels in Specifications with Inspection © www.Gilb.com Version 8- Sep. 2010 36
  37. 37. Value for Money Inspection and CMMI David Rico, http://davidfrico.com © www.Gilb.com Version 8- Sep. 2010 37
  38. 38. A Recent Example Source Eric Simmons, erik.simmons@intel.com 25  Oct  2011   Personal  Public  Communica*on  Applica*on  of  Specifica*on  Quality  Control  (Gilb  Inspec*ons)  by  a  SW  team  resulted  in  the  following  defect  density  reduc*on  in  requirements  (largely  Planguage,  15,000  Engineers  trained)    over  several  months:   Rev. # of Defects of PagesDefects/ Page # % Change in (DPP) DPP 0.3 312 31 10.06   0.5 209 44 4.75 -53% 0.6 247 60 4.12 -13% 0.7 114 33 3.45 -16% 0.8 45 38 1.18 -66% 1.0 10 45 0.22 -81% Overall % change in DPP revision 0.3 to 1.0: -98%Downstream  benefits:  • Scope  delivered  at  the  Alpha  milestone  increased  300%,  released  scope  up  233%  • SW  defects  reduced  by  ~50%  • Defects  that  did  occur  were  resolved  in  far  less  *me  on  average   38   © www.Gilb.com Version 8- Sep. 2010 38
  39. 39. 5 a. Numeric Quality Gateways © www.Gilb.com Version 8- Sep. 2010 39
  40. 40. 5 a. Numeric Quality Gateways Improve Quality of work Defects/Page 100 “Gary” at 80 80 Majors Found McDonnell-Douglas (~160-240 exist!) 60 40 40 20 23 8 0 0 0 1 2 3 4 5 February April Inspections of Gary’s Designs © www.Gilb.com Version 8- Sep. 2010 40
  41. 41. 643% DPP (=CMM 5) Improves Quality by 10x: Raytheon Start of Effort % CONC40% % COC The individual35% learning curve ?? COC30% Cost of Conformance25%20%15% CONC10% Cost of Rework (non-conformance) Bad Process 5%5% Change0% 1st year 2nd year 4rd year 5th year 6th year 7th year 8th year © www.Gilb.com Version 8- Sep. 2010 www.sei.cmu.edu/publications/documents/95.reports/95.tr.017.html 41
  42. 42. 7 a Frequent feedback and improvement assure quality Stake- Realized Potential Value Value Stake- holders holders Plan DoAct Study Perceived-­‐Value  Info   Realized-­‐Value  Informa;on   Stake- Stake- Stake- Stake- Other   holders holders holders holders Cri;cal   Factors   •  2 Kinds of Feedback from Stakeholders, when value increment is really exploited in practice after delivery. •  Combined with other information from the relevant environment. Like budget, deadline, technology, politics, laws, marketing changes. © www.Gilb.com Version 8- Sep. 2010 42
  43. 43. Recent (20 Sept, 2011) Report on Gilb Evo method (Richard Smith, Citigroup)•  http://rsbatechnology.co.uk/blog:8•  Back in 2004, I was employed by a large investment bank in their FX e-commerce IT department as a business analyst.•  The wider IT organisation used a complex waterfall-based project methodology that required use of an intranet application to manage and report progress.•  However, its main failings were that it almost totally missed the ability to track delivery of actual value improvements to a projects stakeholders, and the ability to react to changes in requirements and priority for the projects duration.•  The toolset generated lots of charts and stats that provided the illusion of risk control. but actually provided very little help to the analysts, developers and testers actually doing the work at the coal face.•  The proof is in the pudding; –  I have used Evo (albeit in disguise sometimes) on two large, high-risk projects in front-office investment banking businesses, and several smaller tasks. requirements –  On the largest critical project, the original business functions & performance objective document, which included no design, essentially remained unchanged over the 14 months the project took to deliver, –  but the detailed designs (of the GUI, business logic, performance characteristics) changed many many times, guided by lessons learnt and feedback gained by delivering a succession of early deliveries to real users. successfully went –  In the end, the new system responsible for 10s of USD billions of notional risk, live over over one weekend for 800 users worldwide, was seen as a big success by the sponsoring and stakeholders. “ I attended a 3-day course with you and Kai whilst at Citigroup in 2006” 5 March 2013 © Gilb.com Gilb.com 43 © www. Version 8- Sep. 2010 43
  44. 44. 7b Learn Stakeholders Measure Values Value Management Deliver Process Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 44
  45. 45. 7b Learn Stakeholders Measure Values Identify Stakeholders Who and what cares about the outcome of our project? Deliver Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 45
  46. 46. 7b Learn Stakeholders Measure Values Value Capturing Find & specify quantitatively Stakeholder Values, Product Qualities & Resource improvements. Deliver Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 46
  47. 47. 7b Learn Stakeholders Measure Values Solution Prioritization Find, Evaluate & Prioritize Solutions to satisfy Requirements. Deliver Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 47
  48. 48. 7b Learn Stakeholders Measure Values Evo Cycles Decompose the winning Solutions down into smaller entities, then package them so they deliver maximum Value. Deliver Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 48
  49. 49. 7b Learn Stakeholders Measure Values Develop Develop the packages that deliver the Value. Deliver Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 49
  50. 50. 7b Learn Stakeholders Measure Values Deliver Deliver to Stakeholders improved Value. (not always a thing or code) Deliver Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 50
  51. 51. 7b Learn Stakeholders Measure Values Measure Change Measure how much the Values changed. Deliver Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 51
  52. 52. 7b Learn Stakeholders Measure Values Learn & Change Learning is defined as a change in behavior. Deliver Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 52
  53. 53. 7b Learn Stakeholders Measure Values Value Management Deliver Process Solutions Develop Decompose © www.Gilb.com Version 8- Sep. 2010 53
  54. 54. End 7Competitive Lean QA methods to Learn © www.Gilb.com Version 8- Sep. 2010 54
  55. 55. Main Take-away PointsQuality Assurance is far more than ‘test’, and it can be far more cost-effective‘Quality’ is far more than ‘bugs’You probably have a lot to learn, if you want real competitive quality © www.Gilb.com Version 8- Sep. 2010 55
  56. 56. The Lean Quality Assurance Methods Definition of Lean• Everything  ‘not  adding  value  to  the  Customer’  is  considered  to  be  waste.     – This  includes:   •  unnecessary  code  and  func*onality   •  Delay  in  the  soVware  development  process   •  Unclear  requirements   •  Bureaucracy   •  Slow  internal  communica*on   – Amplify  Learning   •  The  learning  process  is  sped  up  by  usage  of  short  itera*on  cycles  –  each  one  coupled   with  refactoring  and  integra*on  tes*ng.  Increasing  feedback  via  short  feedback   sessions  with  Customers  helps  when  determining  the  current  phase  of  development   and  adjus*ng  efforts  for  future  improvements.   – Decide  as  late  as  possible     – Deliver  as  fast  as  possible   – Empower  the  team   – Build  integrity  in   •  separate  components  work  well  together  as  a  whole  with  balance  between  flexibility,   maintainability,  efficiency,  and  responsiveness.   – See  the  whole     •  “Think  big,  act  small,  fail  fast;  learn  rapidly”     © www.Gilb.com Version 8- Sep. 2010 56
  57. 57. What you can do immediately①  Identify the 5 most critical qualities of your system.②  Quantify the 5 qualities.③  For each quality, ①  set a Current level ②  and a Goal level © www.Gilb.com Version 8- Sep. 2010 57
  58. 58. Thanks! Thanks! Tack Takk If you request by email, Subject: ‘Lean QA Books/Papers’ or BOOKS Tom@Gilb.com I’ll send you 2 free books and some papers(you will not ever be put on a mailing list !) Discussion After lecture, all during the conference. Tom@Gilb.com Mobile: +47 920 66 705 www.Gilb.com Copy of these slides will be in Gilb.com Downloads/Slides: http://www.gilb.com/tiki-download_file.php?fileId=489 Version March 3 2013 © www.Gilb.com Version 8- Sep. 2010 58
  59. 59. Thanks! GO BACK LAST SLIDE If you request by email, Subject: ‘Lean QA Books/Papers’ or BOOKS Tom@Gilb.com I’ll send you 2 free books and some papers (you will not ever be put on a mailing list !) PS 6-7 March 2013, Wed-Thurs This weekTom will hold a 2 day Course on Requirements (Vinnande kravdesign) in Gothenburg in Scandinavian Language, arranged by www.inceptive.se/tomgilb/ Discussion After lecture, all during the conference. Tom@Gilb.com Mobile: +47 920 66 705 www.Gilb.com Copy of these slides will be in Gilb.com Downloads/Slides: http://www.gilb.com/tiki-download_file.php?fileId=489 Version March 3 2013 © www.Gilb.com Version 8- Sep. 2010 59
  60. 60. The End ! © www.Gilb.com Version 8- Sep. 2010 60

×