Sogeti Webinar Effective Test Process Improvement 220709

2,541 views

Published on

This webinar presentation briefly covered TPI and TMMi and their usefulness as process assessment frameworks. However, the main thrust of the talk was to present a proven and pragmatic way to do process improvement that gets senior management commitment AND buy-in from project practitioners.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,541
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
193
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Sogeti Webinar Effective Test Process Improvement 220709

  1. 1. Sogeti Ireland Effective Test Process Improvement Fran O’Hara fran.ohara@sogeti.com 22nd July 2009 © Sogeti Group 2009 Page 1 Contents • Introduction • TPI • TMMi • Process improvement challenges and sample solutions • Summary • Q&A © Sogeti Group 2009 Page 2
  2. 2. SOGETI Overview: A Worldwide Leader in IT Services • Sogeti Group Part of 20,000 professionals • Leader in consulting, IT & 200 locations in 14 countries outsourcing • Providing worldwide standards • 88,000 staff • Revenue €8.7 billion Copyright TMap® and TPI® brands • Sogeti: Dedicated R&D on new developments • £1.6 billion revenues Test thought leadership • Testing revenues USA £150M (2007), £190M (2008) Sweden Test Centre of Excellence provides for UK Denmark Ireland • Reduced cost (>25% over 2 yrs) NL Germany • Improved quality Belux Switzerland • Improved flexibility France • Reduced time-to-test/market • Reduced risk Spain • Increased capacity India 3 © Sogeti Group 2009 Page 3 Test Process Improvement - learning to work smarter, to test software better, cheaper, faster Process focus improves World Class predictability of performance (reduces variation), as well as performance itself Products/services (difference between desired and actual) Capability People Process Technology © Sogeti Group 2009 Page 4
  3. 3. Triggers • Efficiency/cost? • Effectiveness? • Growing Complexity? • Different levels of Test Maturity? • Time Pressure? • Are we doing the right thing(s)? Development Business Automation Time-to-market COTS CBD Standard. Reuse ty Growin ali I’s Agile g Qu Complex GU ity Cheap ic esXML Error serv free We b User-friendliness Integration Tester Maintainability © Sogeti Group 2009 Page 5 The TPI® Model Key areas Levels Test Maturity Matrix Checkpoints Improvement suggestions © Sogeti Group 2009 Page 6
  4. 4. 20 Key Areas • Test strategy • Commitment and motivation • Life-cycle model • Testing functions and • Moment of involvement training • Estimating and planning • Scope of methodology • Test specification • Communication techniques • Reporting • Static test techniques • Defect management • Metrics • Testware management • Test automation • Test process management • Test environment • Evaluation • Office environment • Low-level testing © Sogeti Group 2009 Page 7 Current Situation (example) Key area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13 1 Test strategy A B C D 2 Life - cycle model A B 3 Moment of involvement A B C D 4 Estimating and planning A B 5 Test specification A B techniques Dependencies! 6 Static test techniques A B 7 Metrics A B C D 8 Test automation A B C 9 Test environment A B C 10 Office environment A 11 Commitment and A B C motivation 12 Test functions and training A B C 13 Scope of methodology A B C 14 Communication A B C 15 Reporting A B C D 16 Defect management A B C 17 Testware management A B C D 18 Test process management A B C 19 Evaluation A B 20 Low - level testing A B C © Sogeti Group 2009 Page 8
  5. 5. Process Improvement …. Key area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13 1 Test strategy A B C D 2 Life - cycle model A B 3 Moment of involvement A B C D 4 Estimating and planning A B 5 Test specification A B techniques 6 Static test techniques A B 7 Metrics A B C D 8 Test automation A B C 9 Test environment A B C 10 Office environment A Opti ControlledB Efficient 11 Commitment and motivation A Cmizing 12 Test functions and training A B C 13 Scope of methodology A B C 14 Communication A B C 15 Reporting A B C D 16 Defect management A B C 17 Testware management A B C D 18 Test process management A B C 19 Evaluation A B 20 Low - level testing A B C © Sogeti Group 2009 Page 9 Testing Maturity Model Integration (TMMi®) • Currently under development by the TMMi Foundation See www.tmmifoundation.org Levels 2 & 3 published • The TMMi® Foundation will provide: A standard TMMi® Model that can be used in isolation or in support of other process improvement models and is either staged or continuous. An independently managed data repository to support TMMi® assessment method accreditation, assessor and assessment certification/validation and validated assessment data and certificates. Assessment Method Accreditation/Audit Framework for TMMi® in accordance with ISO15504 and the process to certify commercial assessment methods against the standard model. Certification and training/examination process, procedures and standards for formal, public accreditation of Assessors and Lead Assessors and the on-going management. © Sogeti Group 2009 Page 10
  6. 6. TMMi® Model Level Process Areas 5 Optimisation Defect Prevention Test Process Optimisation Quality Control 4 Management & Measurement Test Measurement Software Quality Evaluation Advanced Peer Reviews 3 Integration Test Organisation Test Training Program Test Life Cycle and Integration Non-Functional Testing Peer Reviews 2 Managed Test Policy and Strategy Test Planning Test Monitoring and Control Test Design and Execution Test Environment 1 Initial © Sogeti Group 2009 Page 11 Problems with Process Improvement Typical Issues with Process Results from Sogeti ‘Worldwide survey on TPI’ : Improvement: Lack of management commitment ** Lack of management commitment Unrealistic expectations * Lack of prioritised goals to Unclear scope and target (i.e. goals) * focus/prioritise Unclear communication Only 17% had measurable results Excessive emphasis on documenting and assessing process Results unknown for too long a time Difficulty in defining and Lack of change process deploying the process Change team without influence Inadequate buy-in from staff Involvement of people within the test process* Top down approach (‘Thou shalt..’) Lack of process architecture 44% combined with other improvement projects © Sogeti Group 2009 Page 12
  7. 7. Improvement Strategy - 1 Lack of management commitment Problems - Lack of management commitment** Lack of prioritised goals to focus/prioritise - Unrealistic expectations* - Unclear scope and target (i.e. goals)* - Unclear communication - Only 17% had measurable results - NB Solutions Achieve sign-off on documented requirements and constraints and associated improvement (project) plan Align and prioritise improvement efforts with business goals through a goal-problem tree Focus on early and measurable results © Sogeti Group 2009 Page 13 Problem Scenario - 1 Customer satisfaction Reported Production Defects rating dropped 20% in June 80 survey 70 60 compared to 6 50 months 40 previously. 30 # Severity 1 & 2 defects 20 10 Cost reduction 0 1 2 3 4 5 6 of 10% is also Months a department target © Sogeti Group 2009 Page 14
  8. 8. Problem Scenario -2 Origin of Production Defects Data also shows that rework of production User requirements System requirements defects is Architecture costing 40% of Detailed design development Code & Build budget Integration © Sogeti Group 2009 Page 15 Examples of test-related improvement actions • Perform formal peer reviews of System requirements Involve system testers • Use a risk-based approach to system test involving business users in risk workshops • Use a test driven development approach to unit test (unit test cases developed and automated before code written) • Design system tests as soon as system requirements become available • Resource system test with some dedicated test professionals • Perform formal planning of system test • Measure DDP for peer reviews, unit test and system test • Some ways to identify improvement actions: Brainstorm process problems and vote Brainstorm improvement ideas/actions Model gap analysis – assessment results © Sogeti Group 2009 Page 16
  9. 9. Goal Tree & Cost/Benefit Business Goal Dev/Test Test Process Est. Cost Est. Department Goal Goals of Benefit required actions Reduce IT Reduce rework Improve 200k 1m Appn. costs of production effectiveness of (10% of IT Development defects by 50% Peer reviews, budget) costs by 10% Unit test, and System Test by 75% (Currently running at 50% DDP) Improve Reduce number of As above 0 Satisfied Customer production defects customers satisfaction per month by 50% ratings by 20% (based on equivalent appn. throughput) © Sogeti Group 2009 Page 17 Goal Metric Scenario – Sample Results Defect Detection Percentage per Release 100 80 60 DDP 40 20 0 1 2 3 4 5 6 Release number (or subsequent projects) © Sogeti Group 2009 Page 18
  10. 10. Improvement Strategy - 2 Excessive emphasis on documenting and Problems assessing process - Results unknown for too long a time Use a more agile approach to PI - Iterate in Solutions short improvement cycles (2-3 months) – how do you eat an elephant? Focus on lightweight, rapid process modelling for early deployment (80/20 rule for any gap analysis) Perform formal assessments to confirm process maturity results © Sogeti Group 2009 Page 19 Improvement Strategy - 3 Difficulty in defining and deploying the Problems process Lack of change process Change team without influence Inadequate buy-in from staff Involvement of people within the test process* Top down approach (‘Thou shalt..’) Use a facilitated workshop approach to process Solutions modelling/improvement Process owners involved in workshops, ‘change team’ become facilitators Goal–problem driven focus Use a continuously improving well-defined change process © Sogeti Group 2009 Page 20
  11. 11. Example Modeling Session •Picture of process-level modeling (Image2) •session inserted here. © Sogeti Group 2009 Page 21 Example “Mapped” Element © Sogeti Group 2009 Page 22
  12. 12. When a standard framework is required – sample process level template Level of Abstraction of tasks is Activity Controls or ‘Work Aids’ provide options key to a common framework for supporting templates, tools, procedures, etc. Activity Name Activity Purpose Activity Controls Action Artifact Artifact Inputs Entry Tasks Exit Outputs Artifact Condition ActionAction Condition Artifact Artifact Artifact Condition Action Condition Action Artifact Issues Roles Measurements Planning, Name Name Costs, Efforts, Tailoring, … Time, Quality Tailoring can also be used to highlight key differences of approach required © Sogeti Group 2009 Page 23 Sample Test Process Element Define the Test Strategy Entry Criteria Controls / Work Aids Exit Criteria 1. Inputs are under 1. Project Lifecycles with defined Test Stages 1. Test-ware is placed under Configuration Management 2. Test Methodology (identifying Test Methods & Test Configuration Management control Techniques.) control. 3. Review Meeting Minutes Template 4. Risk List Template 5. Test Strategy Template Inputs Tasks Outputs 1. Project Requirements 1. Determine level of Requirements definition for the 1. (Risk Based) Test Strategy. Document(s) [User, project 2. Prioritised Risk List. System, 2. Divide the System under Test into its components, if Functional/Technical] relevant. Prioritise component parts 2. Statement of Work 3. Identify and analyse Product Risks, taking Project 3. Test Project Scope Requirements, Statement of Work, Prioritized Quality Characteristics and System Components into consideration 4. Identify and Define Test Levels. 5. Identify and Define Test Level Attributes - Methods and Test Techniques, exit criteria, coverage, automation approach, etc. 6. Document the Testing Strategy. 7. Review and sign-off the Testing Strategy. Tailorings Roles Metrics 1. Test Manager (T.1-T.7) 1. Planned versus actual effort 2. Test Designer (T.5) 3. Business Analyst (T.3, T.7) 4. System Expert (T.3-T.7) 5. End User (T.3, T.7) © Sogeti Group 2009 Page 24
  13. 13. Process of change Obtain awareness Determine target, area of consideration and approach Define Execute improvement assessment actions Perform Formulate evaluation plan Implement improvement actions © Sogeti Group 2009 Page 25 Suggestion: Use a performance-based ‘agile’ process improvement approach Phase 2: Phase 1: Selected Documented goals and problems baselines of for process current practice improvement and performance Phase 3: Process Phase 4: Revised documents to be process and deployed performance baselines © Sogeti Group 2009 Page 26
  14. 14. Do’s – from TPI® survey • Clear scope and target 16% • Clear communication 9% • Results periodically visible 8% • Champions in change team 8% • Use of a change process 7% • Other ... 5% Biggest DO’s: • Management commitment 24% • Involvement of “own” people 23% © Sogeti Group 2009 Page 27 Summary – key suggestions • Use an agile approach to test process improvement – incremental with 80/20 prioritisation Driven by and linked to measurable business goals Addressing key staff process problems • Involvement and ownership by test staff • Lightweight, usable process definition (80/20 again!) © Sogeti Group 2009 Page 28
  15. 15. Sogeti Support • Capability building test process and how to improve it seminars, workshops and training • Process assessments Model based (e.g. TPI®) and/or benchmarking • Consulting/mentoring during process improvement • fran.ohara@sogeti.com • www.sogeti.ie © Sogeti Group 2009 Page 29

×