Test Managers: How You Can Really Make a Difference

542 views

Published on

When leading a test team or working in an agile team, becoming a trusted advisor to other stakeholders is paramount. This requires three key skills: earning trust, giving advice, and building relationships. Join Julie Gardiner as she explores each of these skills, describing why and how a trusted advisor develops different “mindsets.” Julie shares a framework of “quick-wins” for test managers and team leaders who need to show the value of testing on projects. To help provide timely, relevant information to stakeholders, she shares seven powerful monitoring and predicting techniques. Julie demonstrates three objective measures showing how testing adds value to organizations. To make sure that everyone is on the same page, Julie urges managers to establish a foundation for testing through well-defined policy statements, agreed to and sanctioned by senior management. Receive a set of spreadsheets and utilities to support your activities as a test manager who really makes a difference.

Published in: Technology, Business
  • Be the first to comment

Test Managers: How You Can Really Make a Difference

  1. 1. TP PM Tutorial 10/1/2013 1:00:00 PM "Test Managers: How You Can Really Make a Difference" Presented by: Julie Gardiner The Test People Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  2. 2. Julie Gardiner The Test People In the IT industry for more than twenty years, Julie Gardiner has held positions as an analyst programmer, Oracle DBA, and project manager. She has first-hand experience as a test analyst, test team leader, test consultant, and test manager. Previously with UK-based Grove Consultants for seven years, Julie now is head of agile, talent and transition management with The Test People. A certified ScrumMaster and agile coach, Julie also specializes in risk-based testing, agile testing, test management, and people issues.
  3. 3. Passionate about Testing, Excited about Technology Passionate about Testing, Excited by Technology Test Managers: How you can really make a difference 13th August 2013 Julie Gardiner Email: julie.gardiner@thetestpeople.com Twitter: @cheekytester ©  Julie  Gardiner   Passionate about Testing, Excited about Technology Contents What makes us valuable on project How to keep testing valuable on projects Showing the value of testing 1  
  4. 4. Passionate about Testing, Excited about Technology What does a Trusted Advisor look like? Passionate about Testing, Excited about Technology What senior managers are looking for §  commitment not just involvement plus… a –  knowledgeable enthusiasm sense of –  high quality teams humour §  respect for their decisions J –  an understanding of office politics –  an appreciation of other pressures and issues §  honesty and courage –  eyes on the deadline but hands firmly on quality –  a realist with a hint of optimism 2  
  5. 5. Passionate about Testing, Excited about Technology What if senior managers seem uninterested? §  lack of understanding solu%on:  educa%on   –  what do we do with our time? –  misconceptions regarding testing/development §  boredom solu%on:   enquire   –  information overload – too much detail –  the wrong information presented (does not understand) “The bitterness of poor quality is remembered long after the sweetness of meeting the schedule is forgotten” Passionate about Testing, Excited about Technology Being a “Trusted Advisor” requires three core skills Source: The Trusted Advisor David Maister, Charles Green Robert Galford building relationships 3  
  6. 6. Passionate about Testing, Excited about Technology Trust must be earned & deserved §  trust grows over time rather than just appears §  beware that trust can be lost in an instant §  trust is personal §  it involves both rational and emotional factors §  trust is a two-way relationship §  trust is different for the test manager & senior manager §  trust will entail risk §  if risks are not created, trust will flounder earning trust to be trusted is never a right… ©  Julie  Gardiner   it is always a privilege and honour Passionate about Testing, Excited about Technology Building relationships §  deal with senior management as “human beings” §  formula to building a strong relationship §  BSR = AQ + LTA building relationships §  be open and honest – not a sycophant §  when you need help – ask –  this can be a sign of strength rather than weakness §  show appreciation make it your goal to be someone managers turn to first when issues arise and advice is required. 4  
  7. 7. Passionate about Testing, Excited about Technology Giving advice effectively §  be prepared –  look beyond the “now” & understand the whole situation giving advice §  advice is almost never a purely logical process –  don’t exaggerate or lie – ever! (this destroys trust) –  give good news before bad news (don’t hide bad news) –  be fair with information §  know your audience –  the advice we give must be helpful –  different people need different information Passionate about Testing, Excited about Technology Developing the right mindset having a right attitude and mindset is a massive step closer to becoming a “Trusted Advisor” to Senior Management six key attitudes to consider §  focus on the other person §  being sincere §  having confidence (in oneself) §  adopting a blameless culture §  taking responsibility §  recognise strengths and weaknesses –  in you and your team 5  
  8. 8. Passionate about Testing, Excited about Technology Contents What makes us valuable on project How to keep testing valuable on projects Showing the value of testing Passionate about Testing, Excited about Technology Company Objectives? Quality Priority QUALITY TIME Cost Time COST ©  Julie  Gardiner   6  
  9. 9. Passionate about Testing, Excited about Technology Managing testing Policy   overall approach to testing Plan   Measure   Strategy   Review   Goal   Passionate about Testing, Excited about Technology Test policies •  represents testing philosophy for organisation –  outlines the corporate goals for testing •  guides to stated goals and objectives •  communicates good practice for the organisation –  approach, culture and standards •  •  •  •  developed by the IT, R&D or Product Development apply to both new development and maintenance short, static, high-level document some areas covered include –  test process, levels of testing, success factors, measurements, test process improvement 7  
  10. 10. Passionate about Testing, Excited about Technology Test policy examples •  testing is to be riskdriven •  reviews will include testers •  auditable test documentation records will be kept •  entry and exit criteria will be specified and enforced •  the value of testing will be monitored and reported What policy statements do you have in your organisation? Passionate about Testing, Excited about Technology Policy, Strategy and Plans Company Policy for Testing Company/Project Strategy for Testing Project (High Level) Test Plan 8  
  11. 11. Passionate about Testing, Excited about Technology Monitoring •  regular and timely supply of information •  what is actually happening •  to compare with the plan •  right use of words, graphs and tables •  interpretation & explanation is often required •  enables effective and timely control •  used to predict what is likely to happen now to introduce some powerful monitoring techniques to display on your dashboard… Passionate about Testing, Excited about Technology One: S-curves How  to  use  S-­‐curves   S-­‐curves  for:      tests  run/a@empted      tests  passed/complete    bugs  found      bugs  fixed   Pme   Plot  the  points  you  know,  and  planned  end  point   e.g.  no.  tests  run  so  far  &  planned  final  no.  at  end  date   Plot  trend  line:  3rd  order  polynomial     Useful  approximaPon:  straight  line   Source:  Marnie  Hutcheson   9  
  12. 12. Passionate about Testing, Excited about Technology Example using S-Curves Source: Tim Trew, Philips Passionate about Testing, Excited about Technology Two: Defect Measure (DM) Rate DM = 10*H+5*M+L DM Rate = DM/(hours of test effort) 10  
  13. 13. Passionate about Testing, Excited about Technology Three: System Availability Host System Availability Wed Tue Mon Sun Days Sat Available Fri Down time Thu Wed Tue Mon 0 2 4 6 8 10 Hours available in a day Based on 10 hour day we lose 2 days of down time! Passionate about Testing, Excited about Technology Four: Regression Test Coverage Test  Procedure   Solo  Personnel   Solo  Payroll   Personnel  Breadth   Payroll  Breadth   Personnel  Scenario  1   Payroll  Scenario  1   Key:   .0   Release   .1   Release  1   Release  1.0   Release  11.1   Release  1.2.2   not  run   defects  found   no  defects   11  
  14. 14. Passionate about Testing, Excited about Technology Five: Radar Diagram Rating Code Turmoil(loc) Defects found this week Total Open Defects Test Success Rate Test Completion Rate TOTAL RATING Raw Data Code Turmoil Defects found this week Total Open Defects Test Success Rate Test Completion Rate 10/23/2013 1127 23 24 95.50% 44.80% Ranges 1 = 0-50, 2 = 51-100, 3 = 101-150, 4 = 151-200, 5 = 201-250, 6 > 250 1 = 0-2, 2 = 3-4, 3 = 5-6, 4 = 7-8, 5 = 9-10, 6 > 10 1 = 0-10, 2 = 11-20, 3 = 21-30, 4 = 31-40, 5 = 41-50, 6 > 50 1 = 100%-98%, 2 = 97%-95%, 3 = 94%-92%, 4 = 91%-89%, 5 = 88%-86%, 6 < 86% 1 = 100%-95%, 2 = 94%-90%, 3 = 89%-85%, 4 = 84%-80%, 5 = 79%-75%, 6 < 75% 5 to 10 11 to 15 >15 6 6 2 1 5 20 Test Completion Rate 17-Feb-13 1409 16 32 98.70% 70.00% 24-Feb-13 1020 6 7 99.50% 71.80% Code Turmoil(loc) 6 5 4 3 2 1 0 Test Success Rate 03-Mar-13 799 13 12 99.80% 78.90% Defects found this week Total Open Defects Source: Mike Ennis, Managing the End Game of a Software Project, STAREast Passionate about Testing, Excited about Technology Six: Outstanding Risks start Residual Risks today end date residual risks of releasing TODAY all risks ‘open’ at the start Progress through the test plan Source: Risk Based E-Business Testing – Paul Gerrard & Neil Thompson 12  
  15. 15. Passionate about Testing, Excited about Technology Objective Objective Objective Objective Objective Benefit Benefit Benefit Benefit Benefit Benefit Benefit (or objective) based test reporting Open Risks Closed Open Closed Closed Open Open Closed Source:  Paul  Gerrard   Benefits available for release Passionate about Testing, Excited about Technology Seven: Reliability/Confidence Curve alternative 1 alternative 2 Time (minutes) Reliability/Confidence Curve 100 80 60 40 20 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Release 1 1.1 Time to failure R 1.0 R 1.1 R 1.2 R 1.3 13  
  16. 16. Passionate about Testing, Excited about Technology Contents What makes us valuable on project How to keep testing valuable on projects Showing the value of testing Passionate about Testing, Excited about Technology Three key measures to consider cost saving per bug Defect Detection Percentage (DDP) Pre-release Defect Fixed Percentage (DFP) 14  
  17. 17. 1000   Passionate about Testing, Excited about Technology 100   10   “Testing is expensive” 1   Red   Des   Test   Use   •  compared to what? •  what is the cost of NOT testing, or of bugs missed that should have been found in test? –  cost to fix bugs escalates the later the bug is found –  poor quality software costs more to use o  users take more time to understand what to do o  users make more mistakes in using it o  morale suffers o  => lower productivity •  what does a bug cost in your organisation? Passionate about Testing, Excited about Technology How much does it cost (based on Boehm) How much do defects cost? 18000 16000 14000 12000 10000 8000 6000 4000 2000 0 Folksam ru nn ing li v e sy st em /a cc ep ta nc e in iti a l te st at ion sp ec i fic re qu i re m en ts Boehm Source: Kristina Bihlar, Folksam BR 15  
  18. 18. Passionate about Testing, Excited about Technology How much do our defects cost Defect in: requirements specification initial test system/acceptance test live running Manhour to fix Cost 1 2.5 4 8 15 Cost per hour in SEK: 665 1663 2660 5320 9975 665 What if we found 100 defects during testing: During system/acceptance test what could have been found in initial test what could have been found in analysis But it could have cost: Source: Kristina Bihlar, Folksam BR We saved: We saved: 532,000 266,000 465,500 997,500 … Passionate about Testing, Excited about Technology How expensive for you? •  do your own calculation –  calculate cost to fix bugs missed by testing –  calculate cost to fix bugs found in testing –  calculate cost of testing o people’s time, machines, tools •  estimate if no data available –  your figures will be the best your company has! 16  
  19. 19. Passionate about Testing, Excited about Technology Questions you may be asked How good is the testing anyway? Can you prove you are doing a good job? Your testing can still be just as good in less time, can’t it? (That deadline pressure really didn’t matter, did it?) Is the testing any better for this release? (Have we learned anything?) (Have we really improved our testing?) How many bugs have we missed? Passionate about Testing, Excited about Technology Some questions for you •  do you keep track of defects? –  defects found in testing o  different test stages (e.g. system test, user acceptance test), o  different releases (e.g. testing for an incremental release) –  defects found in live running o  reported by users / customers •  can you find these numbers from a previous project and your current project? •  do you have a reasonable number of defects found? if so, you can use DDP to measure your test effectiveness 17  
  20. 20. Passionate about Testing, Excited about Technology Defect Detection Percentage (DDP) Defects  found  by  this  tesPng   Total  defects  including  those  found  aYerwards     •  "this" testing could be –  a test stage, e.g. component, integration, acceptance, regression, etc. –  all testing for a function or subsystem –  all testing for a system Passionate about Testing, Excited about Technology Effectiveness at finding defects Defects Found 50 45 40 35 30 25 20 15 10 5 0 Release 100% 90% 80% 70% 60% DDP 50% 40% 30% 20% 10% 0% Time Defects found Defects found Total defects 10 50 38 37 35 27 24 19 0 88 87 85 77 74 69 62 50 42 after testing: 12 found: in testing: DDP = 50 69 62 87 85 77 74 50 88 57 59 65 68 72 81 = 100 % 18  
  21. 21. Passionate about Testing, Excited about Technology Options for measuring DDP •  what defects? –  simplest: all test defects / all defects so far –  by severity level •  how "deep" to go? (how much analysis) –  deeper analysis gives more detailed information o but is it really worth the extra effort that would be needed? •  start simple –  simple information is much better than none –  learn from what information you have Passionate about Testing, Excited about Technology DDP versus DFP Defects  found  by  tesPng   Defect  DetecPon  Percentage  =   all  defects  found     Testing Defects   found   and  fixed   Pre-­‐release   Defect  Fix  Percentage  =   Defects     found   and  not  fixed   Defects  found   aYer  release   defects  fixed  before  release   all  defects  found  before  release   19  
  22. 22. Passionate about Testing, Excited about Technology Top 10 common responses from the survey “what  do  senior  managers  actually    look  for  in  a  test  manager?”   Passionate about Testing, Excited about Technology Top 10 surprise responses from the survey “what  do  senior  managers  actually    look  for  in  a  test  manager?”   20  
  23. 23. Passionate about Testing, Excited about Technology Ways forward for you •  set yourself an agenda –  choose most important improvement areas –  be practical –  plan the assessment of the changes •  plan actions and dates –  assign responsibility –  how to support the initiative –  timetable for improvement milestones •  the value of today is what you do next Passionate about Testing, Excited about Technology Passionate about Testing, Excited by Technology Questions ? Phone: +44 (0)113 8151779 Email: julie.gardiner@thetestpeople.com Web:Julie  Ghttp://www.thetestpeople.com ©   ardiner   21  

×