Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

To requirements and beyond...

744 views

Published on

Доклад Olivier Denoo на конференции Analyst Days-5, 22-23 апреля 2016 г., Санкт-Петербург
www.analystdays.com

Published in: Education
  • Be the first to comment

  • Be the first to like this

To requirements and beyond...

  1. 1. To Requirements and Beyond
  2. 2. 2 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
  3. 3. 3 WHO’S WHO? Mais que diable allait-il faire dans cette galère? - Molière
  4. 4. 4 A Schizophrenic View on Me, Myself and I, … I’m 20 ! Olivier Denoo I’m Belgian
  5. 5. 5 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
  6. 6. 6 Functional Defect Project : Louvois Late payments, wrong salary calculation… The French Ministry was unable to fix the software issue… Families are appalled 346 Mio € Loss – no usage French Servicemen hit by a payroll software issue
  7. 7. 7 Security Defect Orange TelCo blamed – Privacy rules over hacking
  8. 8. 8 User Satisfaction issue Windows 8 – Even Vista was better!
  9. 9. 9 User / Customer Issue Technical Issue Functional Issue Ow! Let’s go fishing! Production Maintenance Poor Non- Functional Testing Poor Code quality Poor Technical architecture Bug - Defect Failure - Issue Insufficient involvement Poor business processes Communication Needs Market Poor Requirements Poor Functional Architecture Poor functional Testing Insufficient involvement Poor Business Processes Communication Needs Market Poor Non- Functional Testing Poor Code quality Poor Functional Testing Production Maintenance Poor Technical Architecture Poor Requirements Poor Functional Architecture CMMI® for DEV
  10. 10. 10 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
  11. 11. 11 LANGUAGE & PATTERNS A matter of
  12. 12. 12 How Do We (testers) speak? • Constraints: – Time – Costs – Resources • Requirements: – Coverage – Test depth Coverage Depth Costs, Resources Time
  13. 13. 13 And What About the Others?
  14. 14. 14 Understand them Speak their language Fulfil their needs Put yourself into their shoes
  15. 15. 15 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
  16. 16. 16 Use Patterns… • …and make TESTABLE requirements: – FUNCTIONAL • As ROLE I need ACTION in order to OBJECTIVE + context qualifyers • As user I need to access my personal file in order to complete it with the necessary info – SYSTEM • When TRIGGER the system NAME must ACTION/PERMISSION to WHOM/WHAT in order to OBJECTIVE + qualifyers • When book is available, BorrowBook must allow subscribers to borrow if subscription paid and limit not reached
  17. 17. 17 MEASURING START
  18. 18. 18 Often (very often)… • Absence of references • Few measures and metrics (or none) – Unknown coverage – Unclear efficiency – No capitalization • Decisions = estimations • Management cannot evaluate properly
  19. 19. 19 What Would we Want to Know? • What’s left to do? • Am I on track? • What can I improve? • What is MY quality? • Do defects get solved? • What did I learn? Not measuring = Not knowing
  20. 20. 20 • No clear specification • What is really needed? – Basic or advanced functionality? – Usability? – Performance? – Security? • When do we have enough tests? – How critical is the functionality? – Deep or shallow test? • How long to design, execute the tests? • What should we test first? • How can we draw a conclusion from the tests? Difficulties Encountered
  21. 21. 21 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me…what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
  22. 22. 22 A TRH??? « Logic : a good tool that is sold to us with no user manual » - Pierre Véron
  23. 23. 23 Hierarchical tree view of application requirements Business process thinking How to reach business goals (notion of benefit) Process-oriented Tracing down from process to functions Quality reference Structuring with a TRH test requirement 1 test requirement 1.1 test requirement 1.2 test requirement 1.2.1 test requirement 1.2.2 test requirement 1.2.3 test requirement 1.3 test requirement 1.4 test requirement 1.3.1 test requirement 1.3.2
  24. 24. 24 The Fundamental Test Process Test planning and control Test analysis and design Test implementation and execution Evaluating exit criteria and reporting Test closure activities TRH used throughout these stages
  25. 25. 25 Unit testing Integration testing System testing Acceptance testing Requirements Specification Functional Specification Global design Detailed design Code Business Objectives The TRH and the V-model Test requirement Business requirement System requirement
  26. 26. 26 The Problem with Requirements • Requirements form the basis for testing • Requirements are not always testable – Often imprecise, ambiguous – Often incomplete – Can be contradicting or totally wrong – Sometimes, requirements simply don’t exist • Challenges – make imprecise, incomplete requirements testable – Identify wrong, missing requirements
  27. 27. 27 How would you test a train ticket application? A Small Example …
  28. 28. 28 Constructing the Hierarchy HOW? Let’s bring the Money In Sell tickets Exchange tickets Refund tickets
  29. 29. 29 Constructing the Hierarchy HOW? Let’s bring the Money In Sell tickets Exchange tickets Refund tickets International National Full price Off peak Reduced - Young Reduced - Sr Reduced Loyalty HOW?
  30. 30. 30 Asking How, Creatively • Use alternative formulations: – What needs to happen? – Are there any related problems? – What are you likely to have done previously/next? – What needs to happen when this fails? – How does the user know this? – How could things be screwed up? • Use basic test specification techniques to stimulate responses. – “Manage things” → CRUD (create/read/update/delete) – Equivalence partitioning – Boundaries – Negative cases • Make sure sub-requirements cover parent 100%
  31. 31. 31 Establish Business Context Let’s bring the Money In Sell tickets Exchange tickets Refund tickets International National Full price Off peak Reduced - Young Reduced - Sr Reduced Loyalty WHY? WHY? WHY?
  32. 32. 32 Asking Why, Unaggressively • The question “why?” by itself can be quite intimidating (or even look foolish) • Use alternative formulations: – What is the purpose of that? – Can you explain why you need it to do that? – What’s the relation to your business? – What business problem is this solving? – What was the thinking behind that? – What is the underlying reason for that? – What would happen if that feature was not available? – …
  33. 33. 33 How to Get There? • Previous Knowledge & Experience • Reading and retro-analysis • Face-to-face meetings • Workshops NEVER ASSUME !!! - Loyalty is not about calculating & redeeming points It is about COMMUNICATION
  34. 34. 34 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
  35. 35. 35 A TRH What for? • Prioritise • Communicate • Pilot • Estimate • Test strategy backbone test requirement 1 test requirement 1.1 test requirement 1.2 test requirement 1.2.1 test requirement 1.2.2 test requirement 1.2.3 test requirement 1.3 test requirement 1.4 test requirement 1.3.1 test requirement 1.3.2 1 1 1 2 2 3 3
  36. 36. 36 Test Implementation and Execution TRH Test design spec Test design spec Test procedure spec Test case spec Test procedure spec Test Plan bugs Baseline planning Progress report
  37. 37. 37 TRH and Test Design TEST SPECIFICATION ID: XYZ PURPOSE: Verify that search function examines articles. APPROACH REFINEMENTS: …
  38. 38. 38 TRH and Process Test procedure Scenario Data Validation points Physical Design WHAT ? Test procedure Scenario Data Validation points Test case1 Test case2 Test case3 Test case5 Test case4 Test Requirements Hierarchy WHY / WHAT? test requirement 1 test requirement 1.1 test requirement 1.2 test requirement 1.2.1 test requirement 1.2.2 test requirement 1.2.3 test requirement 1.3 test requirement 1.4 test requirement 1.3.1 test requirement 1.3.2 Logical Design WHAT?
  39. 39. 39 Who’s the Pilot?…Damn’ it’s the TRH! • Nb testable requirements (leaves) – Identified / Developed / written – % Coverage(s) – % Executed Progress (t vs. t-1 vs. planned) – Risk / Impact (before and after execution) – Planning / Effort
  40. 40. 40 In Practice • Planning - Execution – 100 requirements to test – 100 persons*days budget – Writing / Execution = 60/40 • Quality – 50% requirements coverage – 50% passed product – Grey zones (50 shades)?
  41. 41. 41 Software Tools…Who Cares? • There are (too) many of them – From Excel to Doors via TestLink & others – Integrated in suites (HP, IBM) or not – OpenSource or Commercial • Most (at least the known ones) are good • Tools are NOT the issue
  42. 42. 42 …Provided they Allow… • Real-time share and follow • Information and reporting (collect and process) • Decisions • Anticipation • Adapt the tool to YOUR organisation
  43. 43. 43 Word Processor in Outline Mode
  44. 44. 44 Mind-Mapping Software
  45. 45. 45 Spreadsheets
  46. 46. 46 …but How to Use them Matters • They must be robust… – The « in-house » requirements tracker • …fit to your needs and organisation • …and should allow to share and pilot – Customer / supplier « encode twice »
  47. 47. 47 Requirements • Total Nb & Testable >> coverage • Repartition / severity • Repartition / phase / version • Repartition / priority • Nb incorrect or maintained requirements • Nb scenarios / Requirement • Nb unused requirements • Nb added requirements • Nb changed requirements
  48. 48. 48 Test Scenarii – Test Designs • Total & Testable (Nb) • Répartition / severity • Repartition / phase or version • Repartition / priority • Nb scenarii reviewed and/or incorrect • Nb scenarii / Requirement • Nb scenarii unused • Nb scenarii added Measurements details Fine but complex vs.coarse but blurry estimations Differs from phase to phase depending on goals
  49. 49. 49 Defects • Total Nb • Repartition / severity • Repartition / priority • Repartition / status • Repartition / responsible • Nb found / unit time • Nb false positive • Nb unsolved • Nb found next phase
  50. 50. 50 Adjust your Strategy You are HERE Time Scope EXPECTED Deadline EXTENDED Deadline 100% 0% 50% 1 3 2 66% Productivity
  51. 51. 51 Different Mitigation Options • Option 2 – Limit Scope (shrink till you drop) • Quality drop • Increased risk – Let it slip (the whoosh sound of dealines) • Late market presence • Extra costs • Option 3 – Increase resources (Stakhanov – Taylor) • Extra costs • 2 women + 4,5 mths = NO baby
  52. 52. 52 One Example • Mitigating options 2 (slippage) and 3 (increase ressources) – Use what you already know to predict and plan • Risk oriented strategy – Sanity check – Flexible resources – Planning by roles and capability • Use existing resources • Identify and manage activities – Test new functionality – Re-test (bug fix) – Regression test (re-validation) – …/… • Multiply PC (automation) – Execute based on risk / priority • Most important first • Most important = deeper and longer BA
  53. 53. 53 Estimation Model Estimated (found) Fixed Planning Exported (next phase) Inherited (previous phase) TEST Phase n
  54. 54. 54 Test Campaigns 1 2 3 4 5 6 7 Go / No Go SP 2.1 SP 2.2 SP 2.3 ? Production? Production? h1 h2 h3 Inherited from SP1.x h1 h2 Σhi Σhi d2 d3 Manual Tests (n) Manual tests (n+1) + automated (n) • ! Performance • ! Availability • ! Stability 27/03 02/05 21/05 Inherited from SP2.0 d1 0 ? ? ? Correction > c1 Correction > c2 SC 21/03 18/04 8 ? 2/04 di = Nb estimated defcts / phase ci = Nb fixed defects in the same phase hi = Nb inherited defects Σ’hi Σ’hi
  55. 55. 55 And if you Have…TIME • Today (status time) – Consumed vs, attended resources – Forecast based upon experience • Mean time (write / fix / execute) • Mean time (code / analyse / fix) • Application up time • Application performance • MTBE, MTBF Good stop criterion… if enough maturity
  56. 56. 56 Ressources metrics • Time span / person (per profile) – # Hrs / test activity • Preparation • Planning • Design • Environment • Execution • Maintenance • Fixing tests • Reporting • Overhead expenditure
  57. 57. 57 Work Product Trend 0 50 100 150 200 250 300 350 400 450 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC planned designed executed passed
  58. 58. 58 Do NOT Overperform  Date 27/08 Sprint 2 Release 3 # days 15/23 15 # Testers Effective Cum. % planned resources for SP2.3 Execution test designs 5,5 Check printed documents 3 Ad hoc tests 5 63% Quotation (Product Management) 0 100% Re-test SP1 + SP2 ad hoc defects 0 100% Traduction 0 100% Total resources coverage Release 3 13,5 62% Availability - Lost time Availability of the application that day 100,0% week 1/5 week 2/5 week 3/5 week 4/5 week 5/5 Average availability for SP2.3 85,7% 80,1% 78,6% 95,6% 85,7% Test Weather Report 58%  Test execution (2849 data sets) # data sets foreseen that date 170 Found defects # data sets tested that date 90 % data set coverage SP2.3 - against planned ds 53% OK NOK Not testable Intermediate # data sets executed SP2.3 2148 1850 233 0 65 86,1% 10,8% 0,0% 3,0% % data set coverage SP2.3 - against total ds 75,4% Docs waiting for check # SP2 defects foreseen to be re-tested during SP2.3 445 # SP2 defects re-tested that date 22 OK NOK Not testable # SP2 defects re-tested while executing ds SP2.3 211 184 26 1 87,2% 12,3% 0,5% % re-test defects coverage 47% Average datasets/p/d foreseen for SP2.3 19,9 *Average test design (15 ds/p/d) and check document (29 ds/p/d) Average datasets/p/d effective that day 16,4 Average datasets/p/d effective in SP2.3 17,3 Re-test SP2 AD HOC defects # SP2 ad hoc defects foreseen to be re-tested during SP2.3 106 # SP2 ad hoc defects foreseen that date 0 NOK + OK + # SP2 ad hoc defects re-tested that date 0 OK NOK New defect New defect # SP2 ad hoc defects re-tested in SP2.3 99 81 16 0 0 81,8% 16,2% 0,0% 0,0% % SP2 ad hoc defects re-test coverage 93% Average defects re-tested/p/d foreseen for SP2.3 10,0 Average defects re-tested/p/d effective that day - Average datasets/p/d effective in SP2.3 - % data sets executed SP2.3 Cumul # SP2 ad hoc defects re-tested in SP2.3 Cumul # SP2 defects re-tested while executing ds SP2.3 # Defects # encoded defects that day 18 Urgent High Medium Low # active defects in Defect Tracker for SP2.3 102 20 51 24 7 102 # active defects in Defect Tracker 411 29 257 110 15 411 Open IT exam Pend. TEST Sub-total Active Whole IT Factory 65 0 21 86 Ready TLF Pend. TLF User Exam Test Factory 2 297 26 325 Closed Deferred Archived Not active defects 454 47 1507 2008 Test exam 0 0 Remarks: Today : Finalised retest of fiches linked to "marques-modèles" has been done Application has been better regarding response time but a lot of instabilities (T000): see daily urgent defects sheet for details. Program for tomorrow : Same as today 2419 411  Fitness Program Data set foreseen / executed 0 500 1000 1500 2000 2500 3000 5/aug 7/aug 9/aug 13/aug 19/aug 21/aug 23/aug 27/aug 29/aug 2/sep 4/sep 6/sep Executed Cumulated DS DS foreseen this day (intial planning) 120 140 0 500 1000 1500 2000 2500 3000 Availability of the application 91%92% 24% 100% 93% 64% 89% 82% 86% 100%100%100% 92% 71% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 5/aug 7/aug 9/aug 11/aug 13/aug 15/aug 17/aug 19/aug 21/aug 23/aug 25/aug 27/aug 29/aug 31/aug 2/sep 4/sep 6/sep Availability Retest defects SP1 foreseen / executed 0 20 40 60 80 100 120 140 5/aug 7/aug 9/aug 13/aug 19/aug 21/aug 23/aug 27/aug 29/aug 2/sep 4/sep 6/sep Re-tested this daySP1 + SP2 ad hoc Foreseen to be retested this day (intial planning)
  59. 59. 59 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
  60. 60. 60 Yes, Sure…However • You are also accountable for Quality • You can set-up and follow what everyone understands – Added value – Business means • You bring the missing link to life
  61. 61. 61 I Can Do Everything – I’m An Expert
  62. 62. 62 Keynote Outline Let’s break the ice! Why do you need good requirements? How can you better structure and what for? Some tips and tricks A TRH! Oh great!...erh remind me what for exactly? Pilot, prioritize, communicate, follow-up Yes, but I’m an Analyst, not a TM or a TA… That’s all folks!
  63. 63. 63

×