Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
TP
Half-day Tutorials
5/6/2014 1:00:00 PM
Root Cause Analysis for
Software Testers
Presented by:
Alon Linetzki
Best-Testin...
Alon Linetzki
Best-Testing
Alon Linetzki, founder and managing director of Best-Testing, has been a coach and consultant i...
Root Cause Analysis in Testing
 Partners in this course are:
Course Partners
March 2014
2
March 20143
 The course materials have been prepared by:
Best – Testing,
 Copyright: © Best – Testing, 2008
 All materi...
March 20144
 Limit of responsibility
 Best-Testing invests significant effort in updating this material, however, Best-T...
Introduction
March 2014
5
• No bullets…should be fun…
Chapter 1
6
Alon Linetzki
 30 years in IT, 20+ years as a test engineer and a test manager
 Certified Scrum Master, Scrum Alliance...
We shall cover…
Introduction
 Presenting participants and trainer
 Workshop expectations
What is Root Cause Analysis?
Ho...
We shall cover…
How to Use RCA for analyzing critical problems?
 Cause Effect graphing, amended
 Case Study #0
 Summary...
 Professional Background
 Years experience in testing/development/both/other
 Responsibilities in current role
 Have y...
 Interactive sessions
 Discussions, skepticism, curiosity
 Examples, demonstration
 Thinking, communicating
 Keeping ...
 There are FAR more slides than we can
cover…
 Say if you have something to say… do not
wait till the end of the worksho...
 Start : 10:15
 Lunch: 11:45 – 12:45
 Finish line: 14:15
But… we may have more
breaks if we feel like it… 
Logistics…
...
What Root Cause Analysis?
March 2014
13
Chapter 2
Root Cause Analysis definition
(My interpretation)
From wiki:
 Root cause analysis (RCA) is a class of problem solving
me...
Root Cause Analysis definition
From Bill Willson’s website:
 Root cause analysis (RCA) is a methodology for
finding and c...
Root Cause Analysis - definition
Root cause analysis (RCA) is a methodology for finding
and correcting the most important ...
Root Cause Analysis definition -
summary
 As a business process improvement tool, RCA seeks
out unnecessary constraints a...
Root Cause Analysis
(My interpretation)
A problem solving method/process designed to search
for the root causes of a probl...
“Cows falling on a road from a mountain”
– is it a problem or a symptom?
 Should we eliminate all cows on
that area?
 Sh...
Root causes are underlying causes
 The investigator’s goal should be to identify
specific underlying causes
 The more sp...
Root causes are those that can reasonably be
identified
 Occurrence investigations must be cost beneficial
 It is not pr...
Root causes are those over which management has control
 Analysts should avoid using general cause classifications such
a...
Root causes are those for which effective recommendations can
be generated (if cannot be done, keep on searching!)
 Recom...
 Root causes are underlying causes
 Root causes are those that can reasonably be
identified
 Root causes are those over...
Time for discussion…25
March 2014
 Root Cause Analysis for Beginners - by James J. Rooney and Lee N. Vanden Heuvel
References – what is Root Cause Analysis...
How to use RCA in Defect Analysis?
March 2014
27
 Escaping defects occur all the time
 We try to contain them, from going to production,
but much efforts should be done ...
Defects RCA guided questions
Data Collection
March 2014
29
Question Who
asks?
Describe circumstances and consequence of th...
 Defect Analysis is the process of analyzing a defect
to determine its root cause.
 Defect Prevention is the process of ...
 Defect Source – Finding defects in the stage they
were introduced and as early in the lifecycle as
possible – Eliminatin...
Defects Root Cause Analysis
32
Canceled Defects Root Cause Analysis
 Cancelled defects are not real defects of the
system-under-test
 They can be the r...
Canceled Defects Root Cause Analysis
34
Code Inspection - Example
March 2014
35
 Background:
 180 people on the project
 3,700 modules interconnecting with eac...
Code Inspection - Example
March 2014
36
Code Inspection - Example
March 2014
37
 Analysis:
 Main concern – Complience with design (10%
Medium+)
 Investigated R...
Code Inspection - Example
March 2014
38
Background:
Another project,
450 people
working, 55 testers,
4,300 modules,
strate...
Code Inspection - Example
March 2014
39
 Analysis:
 Investigated RCA:
 Performance defects were on resource usage, memo...
 Adding defect template fields:
 Injected, Detected, Removed
 Version [build] + Date per field
 [Updating the defect l...
RCA process defined
March 2014
41
Time for discussion…42
March 2014
John Ruberto – Root Cause for Problems Escape,
http://blog.ruberto.com/2008/04/another-post/
References
March 2014
43
How to Use RCA for analyzing critical
problems?
March 2014
44
Challenges the current method could
not solve – using 5whys
45
 ‘5ys’ or ‘5 whys’ technique, and the cause-effect diagram.
 Presenting a problem,
 Asking “why?” it happens, finding t...
‘5ys’ or ‘5 whys’ technique, and the cause-effect
diagram.
Presenting the RCA Technique
47
Cause
Cause Cause Cause
Cause
P...
‘5ys’ or ‘5 whys’ technique, and the cause-effect diagram.
1. There is the assumption that a single cause, at each level
o...
Enhancing the method – case study
49
 Short structured interview with rep’s of management,
development, release, system, testing, product teams.
 Step 1: Dra...
Enhancing the Method:
Example project51
Type Question Cause-Effect
Relevancy What evidence you have that the cause exist? ...
Enhancing the Method:
Example project52
Type Question Cause-Effect
Relevancy What evidence you have that the cause exist? ...
 Step 3: Identify the routes leading to the
problem/s,
 Step 4: Identify the strength and direction (impact)
they have (...
Case Study - implementation
54
 Background:
 company was using a very advanced technology, and
a complex product line,
 Complex product, uses mechanic...
 SQA manager was at a course I gave, and liked
one of the tools,
 He thought automation can solve many of his
problems, ...
 I investigated their automation
needs,
 Followed the steps of the
enhanced method,
 Found out their problems might
be ...
1st drawing – RCA meeting
58
Our way of thinking1
2
The RCA meeting (company exec’s and directors):
 At first, the belief was that the primary problem
was:
Partial Test Plan...
1st drawing – RCA meeting
60
Many clients
ask for
different Sw
of the
product
Many
versions
open in
parallel
Complexity of...
1st drawing – RCA meeting
61
Many clients
ask for
different Sw
of the
product
Many
versions
open in
parallel
Complexity of...
1st drawing – RCA meeting
62
Many clients
ask for
different Sw
of the
product
Many
versions
open in
parallel
Complexity of...
2nd drawing – RCA meeting
63
Many clients
ask for
different Sw
of the
product
Many
versions
open in
parallel
Complexity of...
 After a while, we shifted the focus and agreed that
the real problem was actually:
Poor Product Quality
 Because that w...
3rd drawing – RCA meeting
65
Our way of thinking
3rd drawing – RCA meeting
66
Many clients
ask for
different Sw
of the
product
Many
versions
open in
pa...
 We then defined the relevancy, strength and
impact of each arrow (cause),
 And calculated the grades for the arrows (wh...
4th drawing – RCA meeting
68
5th drawing – RCA meeting
69
Many
clients ask
for
different
Sw of the
product
Many
versions
open in
parallel
SCM - Complex...
 We went back to double check the RCA of the
routes leading to the primary problem, marking
the arrows with their grades ...
6th drawing – RCA meeting
71
Last drawing – RCA meeting
72
Many
clients ask
for
different
Sw of the
product
Many
versions
open in
parallel
SCM - Complexity of version control
manage...
Many
clients ask
for
different
Sw of the
product
Many
versions
open in
parallel
SCM - Complexity of version control
manage...
 5 major Root Topics were Identified, explained and prioritized:
1. Produce requirements from client definitions
2. Requi...
 Major Areas of Concern identified and prioritized:
1. Requirements Management
2. Configuration Management
3. Design Docu...
Differentiating problems from
symptoms
77
 If we follow the questioning and grading
process, than problems shall be the ‘leafs’
of the cause-effect diagram, with:
...
Solving problems rather than
covering symptoms
79
 Cause-effect & RCA may eliminate the problems of the model
when answering the questions and marking them on the diagram
...
 Further enhancing the mode, we must think of the following:
 What about the junctions points (inbound and outbound):
di...
Time for discussion…82
March 2014
RCA and Process Improvement?
March 2014
83
Introduction
Case Study #1
Case Study #2
 Process improvement should use data and RCA
outputs
 CI example… - process was changed, training was
done to relevant p...
Analyzing test environments
utilization downtime
March 2014
85
 Analysis:
 Small, yet impacting, breaks in work, caused huge
impact on the release (big team of testers)
 Breaks were ...
 Improved process:
 Defined measurements for monitoring downtime
of test environments
 Identify trend of areas of impac...
Analyzing development effort
with defects found
March 2014
88
15
26
11
120
65
43
32
187
12
38
210
294
588
2184
504
210
126...
 Analysis:
 Areas of defects clustering mainly in: Rater, E-
Support, E-Care, AR
 Relative % of defects are much higher...
 Improved process:
 Defined Unit test and Integration test process and
strategy guidelines and work instructions
 Train...
Time for discussion…91
March 2014
 Root Cause Analysis techniques can help us in
finding the underlying problems (root causes), and
get to deal with real p...
“It is not the strongest of the species that
survives, nor the most intelligent but the one
that is most responsive to cha...
Or perhaps . . .
94
. . . the one who had anticipated all possible
requirements !
Root Cause Analysis in Testing
Upcoming SlideShare
Loading in …5
×

Root Cause Analysis for Software Testers

10,263 views

Published on

In many cases, we choose solutions to problems without sufficient analysis of the underlying causes. This results in implementing a cover-up of the symptoms rather than a solution to the real underlying problem. When we do this, the problem is likely to resurface in one disguise or another, and we may mishandle it again—just as we did initially. Getting to the root of the problem is the better way to solve the current problem, and save time and money in the future. Alon Linetzki identifies and explains a number of root cause analysis techniques widely used in the industry, gives examples of how to apply them in software testing, demonstrates how to implement them, and discusses how to connect them to our day-to-day testing context. Alon shares how root cause analysis can be an effective tool in defect prevention.

Published in: Technology
  • Discover a WEIRD trick I use to make over $3500 per month taking paid surveys online. read more...  https://tinyurl.com/make2793amonth
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Earn $500 for taking a 1 hour paid survey! read more... ♣♣♣ https://tinyurl.com/make2793amonth
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Root Cause Analysis for Software Testers

  1. 1. TP Half-day Tutorials 5/6/2014 1:00:00 PM Root Cause Analysis for Software Testers Presented by: Alon Linetzki Best-Testing Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  2. 2. Alon Linetzki Best-Testing Alon Linetzki, founder and managing director of Best-Testing, has been a coach and consultant in development, testing, and quality assurance for twenty-eight years. Alon helps organizations enhance the test engineer’s professional and personal skills, training test managers in optimization and test process improvement, and optimizing their test operations. His main areas of expertise are in agile testing and transitioning to agile, exploratory testing, test process Improvement, risk-based testing, and test automation. Alon is a member of the ISTQB® authors and review team for the ISTQB® Agile Tester Add-on certification, cofounder of ISTQB® in Israel, leader of the ISTQB® Partner Program, and founder of the SIGiST Israel conference.
  3. 3. Root Cause Analysis in Testing
  4. 4.  Partners in this course are: Course Partners March 2014 2
  5. 5. March 20143  The course materials have been prepared by: Best – Testing,  Copyright: © Best – Testing, 2008  All materials contained in this Handbook were prepared by Best-Testing. All rights of this material are reserved solely for Best-Testing. The material is intended for personal, noncommercial use. All materials published in this material are protected by copyright, and owned or controlled by Best- Testing, or the party credited as the provider of the content. You may not modify, publish, transmit, participate in the transfer of sale of, reproduce, create new works from, distribute, perform, store on any magnetic device or any other device, display, on in any way exploit, any of the content in whole or in part. You may not alter or remove any trademark, copyright or other notice from copies of the content. You may no use the material in this material for the purpose of training of any kind, internal or for customers, without beforehand a written approval of Best-Testing.  The Use of this material  The material in this material is designed to assist the student during the course. It does not include all of the information that will be referred to during the course and should not be regarded as a replacement for reference manuals or other instructions. Copyrights
  6. 6. March 20144  Limit of responsibility  Best-Testing invests significant effort in updating this material, however, Best-Testing is not responsible of any errors or material which may not meet specific requirements. The user alone is responsible for decisions based on the information contained in this material.  Protected Trademark  In this material, protected trademarks appear that are under copyright. All rights to the trademarks in this material are reserved to the authors. Best–Testing wishes you success in the course! Copyrights
  7. 7. Introduction March 2014 5 • No bullets…should be fun… Chapter 1
  8. 8. 6 Alon Linetzki  30 years in IT, 20+ years as a test engineer and a test manager  Certified Scrum Master, Scrum Alliance, 2008  ISQTB® Partner Program leader, ISTQB® Agile Tester Certification Leader  Specializes in Agile testing test strategy & optimization test process improvement  test management  test design  risk based testing  test automation  Building Smart Teams  A degree in ‘testing’ – B.Sc. statistics and criminology,  International Speaker worldwide, since 1995  Co-founder of the Israeli Testing Certification Board (www.itcb.org.il, 2004)  Founder of the SIGiST Israel (testing forum) in Israel (www.sigist.org.il, 2000)
  9. 9. We shall cover… Introduction  Presenting participants and trainer  Workshop expectations What is Root Cause Analysis? How to use RCA for Defect analysis? 7 March 2014
  10. 10. We shall cover… How to Use RCA for analyzing critical problems?  Cause Effect graphing, amended  Case Study #0  Summary How to use RCA for process improvement?  Case Study #1  Case Study #2 Retrospective  Workshop retrospective 8 March 2014 (*) Bonus topic – only if there is enough time left…
  11. 11.  Professional Background  Years experience in testing/development/both/other  Responsibilities in current role  Have you implemented RCA? Can you share?  Workshop expectations? Introduce participants… 9 March 2014
  12. 12.  Interactive sessions  Discussions, skepticism, curiosity  Examples, demonstration  Thinking, communicating  Keeping an open mind for ideas and concepts  Contribute to the class team  Learn…discuss…review…  Give quick feedback on everything you may think of  Time boxed sessions…breaks…exercises…discussions Introducing workshop dynamics 10 March 2014
  13. 13.  There are FAR more slides than we can cover…  Say if you have something to say… do not wait till the end of the workshop! Introducing workshop dynamics 11 March 2014
  14. 14.  Start : 10:15  Lunch: 11:45 – 12:45  Finish line: 14:15 But… we may have more breaks if we feel like it…  Logistics… March 2014 12
  15. 15. What Root Cause Analysis? March 2014 13 Chapter 2
  16. 16. Root Cause Analysis definition (My interpretation) From wiki:  Root cause analysis (RCA) is a class of problem solving methods aimed at identifying the root causes of problems or events.  The practice of RCA is predicated on the belief that problems are best solved by attempting to correct or eliminate root causes, as opposed to merely addressing the immediately obvious symptoms.  By directing corrective measures at root causes, it is hoped that the likelihood of problem recurrence will be minimized. 14
  17. 17. Root Cause Analysis definition From Bill Willson’s website:  Root cause analysis (RCA) is a methodology for finding and correcting the most important reasons for performance problems.  It differs from troubleshooting and problem-solving in that these disciplines typically seek solutions to specific difficulties, whereas RCA is directed at underlying issues 15
  18. 18. Root Cause Analysis - definition Root cause analysis (RCA) is a methodology for finding and correcting the most important reasons for performance problems. It differs from troubleshooting and problem-solving in that these disciplines typically seek solutions to specific difficulties, whereas RCA is directed at underlying issues. 16
  19. 19. Root Cause Analysis definition - summary  As a business process improvement tool, RCA seeks out unnecessary constraints as well as inadequate controls.  In safety and risk management, it looks for both unrecognized hazards and broken or missing barriers.  It helps target CAPA (corrective action and preventive action) efforts at the points of most leverage.  RCA is an essential ingredient in pointing organizational change efforts in the right direction.  Finally, it is probably the only way to find the core issues contributing to your toughest problems. 17
  20. 20. Root Cause Analysis (My interpretation) A problem solving method/process designed to search for the root causes of a problem using a predefined structural thinking process, identifying the underlying issues, with the expectation that dealing with these issues will dramatically reduce the likelihood of the problem to occur. The process involves data collection, cause charting, root cause identification and recommendation generation and implementation 18
  21. 21. “Cows falling on a road from a mountain” – is it a problem or a symptom?  Should we eliminate all cows on that area?  Should we dig-out the mountain?  Should we rotate the sign?  Should we divert the road elsewhere? It seems that sometimes eliminating the causes is not an easy task, and finding the problems is even harder! 19
  22. 22. Root causes are underlying causes  The investigator’s goal should be to identify specific underlying causes  The more specific the investigator can be about why an event occurred, the easier it will be to arrive at recommendations that will prevent recurrence. Practical RCA – definition guidelines March 2014 20
  23. 23. Root causes are those that can reasonably be identified  Occurrence investigations must be cost beneficial  It is not practical to keep valuable manpower occupied indefinitely searching for the root causes of occurrences…  Structured RCA helps analysts get the most out of the time they have invested in the investigation. Practical RCA – definition guidelines March 2014 21
  24. 24. Root causes are those over which management has control  Analysts should avoid using general cause classifications such as “operator error”, “equipment failure” or “external factor”  Such causes are not specific enough to allow management to make effective changes  Management needs to know exactly why a failure occurred before action can be taken to prevent recurrence  We must also identify a root cause that management can influence:  Identifying “severe weather” as the root cause of parts not being delivered on time to customers is not appropriate  Severe weather is not controlled by management. Practical RCA – definition guidelines March 2014 22
  25. 25. Root causes are those for which effective recommendations can be generated (if cannot be done, keep on searching!)  Recommendations should directly address the root causes identified during the investigation  If the analysts arrive at vague recommendations such as, “Improve adherence to written policies and procedures,” then they probably have not found a basic and specific enough cause and need to expend more effort in the analysis process. Practical RCA – definition guidelines March 2014 23
  26. 26.  Root causes are underlying causes  Root causes are those that can reasonably be identified  Root causes are those over which management has control  Root causes are those for which effective recommendations can be generated (if cannot be done, keep on searching!) Practical RCA – definition guidelines March 2014 24
  27. 27. Time for discussion…25 March 2014
  28. 28.  Root Cause Analysis for Beginners - by James J. Rooney and Lee N. Vanden Heuvel References – what is Root Cause Analysis? 26 March 2014
  29. 29. How to use RCA in Defect Analysis? March 2014 27
  30. 30.  Escaping defects occur all the time  We try to contain them, from going to production, but much efforts should be done in preventing them going from Design to Code  A set of questions may help out leading us to Data Collection on those Defects for performing the RCA – next page… Defects RCA introduction March 2014 28
  31. 31. Defects RCA guided questions Data Collection March 2014 29 Question Who asks? Describe circumstances and consequence of the Problem? Test Team Where was the problem introduced? Dev Team Describe the root cause of the problem Dev Team What reviews were held for the phase? Dev Team How and why did the problem escape testing? Test Team What could be done to prevent this type of error in the future? Both What could be done to find this type of problem in the future? Both John Ruberto – Root Cause for Problems Escape
  32. 32.  Defect Analysis is the process of analyzing a defect to determine its root cause.  Defect Prevention is the process of addressing root causes of defects to prevent their future occurrence. RCA – few definitions March 2014 30
  33. 33.  Defect Source – Finding defects in the stage they were introduced and as early in the lifecycle as possible – Eliminating escaping defects RCA – few definitions March 2014 31
  34. 34. Defects Root Cause Analysis 32
  35. 35. Canceled Defects Root Cause Analysis  Cancelled defects are not real defects of the system-under-test  They can be the result of: environment problem (non product), out of scope, test design or test execution problem, un-reproducible defect, not understanding the specifications, etc. 33
  36. 36. Canceled Defects Root Cause Analysis 34
  37. 37. Code Inspection - Example March 2014 35  Background:  180 people on the project  3,700 modules interconnecting with each other  Complex infrastructure  25 testers  Using Test Director/Quality Center as reporting platform  Strategic customer
  38. 38. Code Inspection - Example March 2014 36
  39. 39. Code Inspection - Example March 2014 37  Analysis:  Main concern – Complience with design (10% Medium+)  Investigated RCA:  A few specs were written by same designer  A few modules were written by 3 programmers  Initiated training for designer and programmers
  40. 40. Code Inspection - Example March 2014 38 Background: Another project, 450 people working, 55 testers, 4,300 modules, strategic customer Focus on:  Performance  Maintainability  Proper algorithm usage
  41. 41. Code Inspection - Example March 2014 39  Analysis:  Investigated RCA:  Performance defects were on resource usage, memory leaks  Implementing improper algorithm was mainly due to specification documents written unclear + implementation complexity chosen by a few programmers  Initiated training for architects and programmers
  42. 42.  Adding defect template fields:  Injected, Detected, Removed  Version [build] + Date per field  [Updating the defect lifecycle process (status)]  Changing field CRUD permissions – Dev, Test, Managers  Defining reports  Updating the Defect Management process  Training the teams  Implementing action plan… Implementation guidelines – defects RCA March 2014 40
  43. 43. RCA process defined March 2014 41
  44. 44. Time for discussion…42 March 2014
  45. 45. John Ruberto – Root Cause for Problems Escape, http://blog.ruberto.com/2008/04/another-post/ References March 2014 43
  46. 46. How to Use RCA for analyzing critical problems? March 2014 44
  47. 47. Challenges the current method could not solve – using 5whys 45
  48. 48.  ‘5ys’ or ‘5 whys’ technique, and the cause-effect diagram.  Presenting a problem,  Asking “why?” it happens, finding the effect that caused it (1 effect),  Presenting the effect on the diagram,  Asking “why?” it happens… [back to previous step, unless we ask it for 5 times already]  Done. Presenting the 5whys Technique 46
  49. 49. ‘5ys’ or ‘5 whys’ technique, and the cause-effect diagram. Presenting the RCA Technique 47 Cause Cause Cause Cause Cause Problem Why #1 Why #2 Why #3 Why #4 Why #5 Thinking path…
  50. 50. ‘5ys’ or ‘5 whys’ technique, and the cause-effect diagram. 1. There is the assumption that a single cause, at each level of "why", is sufficient to explain the effect in question. 2. What if one of the ‘Why’ is answered wrongly? Maybe our answer is possible, but what if the actual cause is something else entirely? 3. When we have found the problem, and draw the route, how ‘strong’ is this solution? Maybe we should prefer one over the other? Challenges: what the method can not solve48
  51. 51. Enhancing the method – case study 49
  52. 52.  Short structured interview with rep’s of management, development, release, system, testing, product teams.  Step 1: Draw a cause-effect diagram & exercise the 5whys  Step 2: Investigate the arrows/causes for:  What evidence you have that the cause exist? (relevancy)  What evidence you have that the cause leads to the effect? (strength)  Is anything else needed together with the cause for the effect to occur? (strength)  Is there evidence that the cause is contributing to the problem I’m looking at? How much it contributes? (Impact: direct/indirect) Enhancing the Method: Example project50
  53. 53. Enhancing the Method: Example project51 Type Question Cause-Effect Relevancy What evidence you have that the cause exist? H/M/L Strength (S or W) What evidence you have that the cause leads to the effect? H/M/L Strength (S or W) Is anything else needed, together with the cause, for the effect to occur? Yes/No Impact (D or I) Is there a evidence that the cause is contributing to the problem I’m looking at? Yes / No Impact (D or I) How much this cause is contributing to a possible resolution? Direct / Indirect Mark
  54. 54. Enhancing the Method: Example project52 Type Question Cause-Effect Relevancy What evidence you have that the cause exist? High (3) Strength (S or W) What evidence you have that the cause leads to the effect? Medium (2) Strength (S or W) Is anything else needed, together with the cause, for the effect to occur? Yes (1) Impact (D or I) Is there a evidence that the cause is contributing to the problem I’m looking at? Yes (1) Impact How much this cause is contributing to a possible resolution? Direct (2) Mark 9  You should mark each arrow using this table.
  55. 55.  Step 3: Identify the routes leading to the problem/s,  Step 4: Identify the strength and direction (impact) they have (calculating the mark for each arrow),  Step 5: Choose the best route to focus on,  [Improve it, and go to the next one]. Enhancing the Method: Example project53
  56. 56. Case Study - implementation 54
  57. 57.  Background:  company was using a very advanced technology, and a complex product line,  Complex product, uses mechanics, electronics, hardware, software, devices, cooling device, has water resistant, has heating resistant, accurate up to 1:1,000,000 cm,  In the last 0.5 year, 50% of released machines returned from the floor (clients) for fixing, Example project – Hi-Tech Company 55
  58. 58.  SQA manager was at a course I gave, and liked one of the tools,  He thought automation can solve many of his problems, because:  A lot more tests running,  Identifying more defects before the clients do,  Less products coming back,  Clients are happy! Example project – Hi-Tech Company 56
  59. 59.  I investigated their automation needs,  Followed the steps of the enhanced method,  Found out their problems might be elsewhere… Example project – Hi-Tech Company 57 Lets see the drawing board from that meeting…
  60. 60. 1st drawing – RCA meeting 58 Our way of thinking1 2
  61. 61. The RCA meeting (company exec’s and directors):  At first, the belief was that the primary problem was: Partial Test Planning (less tests are executed) Example project – Hi-Tech Company 59 Lets see an illustration diagram …
  62. 62. 1st drawing – RCA meeting 60 Many clients ask for different Sw of the product Many versions open in parallel Complexity of version control management is very high Defining req’ not good enough by client Spec Lvl 0 No specs in lvl 1 Spec Lvl 1 not complete or does not fit Spec Lvl 2 not written Good definition of Spec Lvl 0 Spec Lvl 1 fits Spec Lvl 2 fit Spec Lvl 2 does not fit/complete Code written with low match to client req’ Only Partial Test planning and not full coverage Partial test case planning and coverage Partial test execution and low coverage Our way of thinking1 2
  63. 63. 1st drawing – RCA meeting 61 Many clients ask for different Sw of the product Many versions open in parallel Complexity of version control management is very high Defining req’ not good enough by client Spec Lvl 0 No specs in lvl 1 Spec Lvl 1 not complete or does not fit Spec Lvl 2 not written Good definition of Spec Lvl 0 Spec Lvl 1 fits Spec Lvl 2 fit Spec Lvl 2 does not fit/complete Code written with low match to client req’ Only Partial Test planning and not full coverage Partial test case planning and coverage Partial test execution and low coverage Our way of thinking1 2
  64. 64. 1st drawing – RCA meeting 62 Many clients ask for different Sw of the product Many versions open in parallel Complexity of version control management is very high Defining req’ not good enough by client Spec Lvl 0 No specs in lvl 1 Spec Lvl 1 not complete or does not fit Spec Lvl 2 not written Good definition of Spec Lvl 0 Spec Lvl 1 fits Spec Lvl 2 fit Spec Lvl 2 does not fit/complete Code written with low match to client req’ Only Partial Test planning and not full coverage Partial test case planning and coverage Partial test execution and low coverage Our way of thinking 1 2
  65. 65. 2nd drawing – RCA meeting 63 Many clients ask for different Sw of the product Many versions open in parallel Complexity of version control management is very high Defining req’ not good enough by client Spec Lvl 0 No specs in lvl 1 Spec Lvl 1 not complete or does not fit Spec Lvl 2 not written Good definition of Spec Lvl 0 Spec Lvl 1 fits Spec Lvl 2 fit Spec Lvl 2 does not fit/complete Code written with low match to client req’ Only Partial Test planning and not full coverage Partial test case planning and coverage Partial test execution and low coverage Our way of thinking1 2
  66. 66.  After a while, we shifted the focus and agreed that the real problem was actually: Poor Product Quality  Because that was the reason the clients returned their product.  And we started RCA from there.  After a while, we started to see the light – real problems started to crystallize, problems that involved people and processes Example project – Hi-Tech Company 64
  67. 67. 3rd drawing – RCA meeting 65
  68. 68. Our way of thinking 3rd drawing – RCA meeting 66 Many clients ask for different Sw of the product Many versions open in parallel SCM - Complexity of version control management is very high Defining req’ not good enough by client Spec Lvl 0 No specs in lvl 1 Spec Lvl 1 not complete or does not fit Spec Lvl 2 not written Good definition of Spec Lvl 0 Spec Lvl 1 fits Spec Lvl 2 fit Spec Lvl 2 does not fit/complete Code written with low match to client req’ Only Partial Test planning and not full coverage Partial test case planning and coverage Partial test execution and low coverage 1 2 Tight schedule projectPrioritization and compromise on scope to clients Low Quality Product Req’ managemen t not good enough Lack of methods and techniques in testing Low lvl of test identification
  69. 69.  We then defined the relevancy, strength and impact of each arrow (cause),  And calculated the grades for the arrows (which are not seen here), Example project – Hi-Tech Company 67 Back to the board…
  70. 70. 4th drawing – RCA meeting 68
  71. 71. 5th drawing – RCA meeting 69 Many clients ask for different Sw of the product Many versions open in parallel SCM - Complexity of version control management is very high Defining req’ not good enough by client Spec Lvl 0 No specs in lvl 1 Spec Lvl 1 not complete or does not fit Spec Lvl 2 not written Good definition of Spec Lvl 0 Spec Lvl 1 fits Spec Lvl 2 fit Spec Lvl 2 does not fit/complete Code written with low match to client req’ Only Partial Test planning and not full coverage Partial test case planning and coverage Partial test execution and low coverage Our way of thinking1 2 Tight schedule project Prioritization and compromise on scope to clients Low Quality Product Req’ management not good enough Lack of methods and techniques in testing Low lvl of test identification S/D W/D W/I S/I
  72. 72.  We went back to double check the RCA of the routes leading to the primary problem, marking the arrows with their grades (from the table, remember?)  We ended up circling the main causes, that have initiated the strongest routes that are directly impacting our problem, Example project – Hi-Tech Company 70
  73. 73. 6th drawing – RCA meeting 71
  74. 74. Last drawing – RCA meeting 72
  75. 75. Many clients ask for different Sw of the product Many versions open in parallel SCM - Complexity of version control management is very high Defining req’ not good enough by client Spec Lvl 0 No specs in lvl 1 Spec Lvl 1 not complete or does not fit Spec Lvl 2 not written Good definition of Spec Lvl 0 Spec Lvl 1 fits Spec Lvl 2 fit Spec Lvl 2 does not fit/complete Code written with low match to client req’ Only Partial Test planning and not full coverage Partial test case planning and coverage Partial test execution and low coverage Our way of thinking1 2 Tight schedule project Prioritization and compromise on scope to clients Low Quality Product Req’ manageme nt not good enoughLack of methods and techniques in testing Low lvl of test identification S/D W/D W/I S/I Last drawing – RCA meeting 73
  76. 76. Many clients ask for different Sw of the product Many versions open in parallel SCM - Complexity of version control management is very high Defining req’ not good enough by client Spec Lvl 0 No specs in lvl 1 Spec Lvl 1 not complete or does not fit Spec Lvl 2 not written Good definition of Spec Lvl 0 Spec Lvl 1 fits Spec Lvl 2 fit Spec Lvl 2 does not fit/complete Code written with low match to client req’ Only Partial Test planning and not full coverage Partial test case planning and coverage Partial test execution and low coverage Our way of thinking1 2 Tight schedule project Prioritization and compromise on scope to clients Low Quality Product Req’ manageme nt not good enoughLack of methods and techniques in testing Low lvl of test identification S/D W/D W/I S/I 74 4/6 Last drawing – RCA meeting Lets see the routes… 3/3
  77. 77.  5 major Root Topics were Identified, explained and prioritized: 1. Produce requirements from client definitions 2. Requirements management 3. Either ‘No Spec Level 1’, or ‘Spec level 1 not matching requirements’ 4. Lack of methods and techniques in testing for development and testing teams 5. Allot of clients define slightly different requirement for the SW – allot of specials  We defined a pragmatic corrective actions plan, with priority items. Example project – Hi-Tech Company 75
  78. 78.  Major Areas of Concern identified and prioritized: 1. Requirements Management 2. Configuration Management 3. Design Documentation and Flow 4. Testing Methodologies, techniques and tools  Not discussed: - Release Management - Risk Management + Risk Based Testing - Requirements Definition - Project Management - Professional Development Example project – Hi-Tech Company76 Organization Language!
  79. 79. Differentiating problems from symptoms 77
  80. 80.  If we follow the questioning and grading process, than problems shall be the ‘leafs’ of the cause-effect diagram, with:  A positive answer to the questions:  Relevancy – ‘yes’  Highest strength mark (direct/Indirect), and  Direct impact on the undesired outcome (initial hypothesis)  Highest grade per route guideline,  Translate into company language (areas of concern)  Enables an Immediate vs. long term action plan Differentiating Problems from Symptoms78 Just what our office needs..
  81. 81. Solving problems rather than covering symptoms 79
  82. 82.  Cause-effect & RCA may eliminate the problems of the model when answering the questions and marking them on the diagram for the best route(s),  (but) We must deal with symptoms some time, in the short term,  We should estimate effort needed (or money required) for each route, and integrate that into our decision (i.e. 60/40 weight)  The questions (answers) make sure that our analysis is with minimal deviations, and that the route we take is a ‘strongest’ one,  It is a constructive process that leads to Understanding, Clarity, Focus and the right Priority setting Summary 80
  83. 83.  Further enhancing the mode, we must think of the following:  What about the junctions points (inbound and outbound): direct impact of routes with those? Indirect? Impact on speed of performance (bottle-necks)?  What is the ROI of this method within context?  Can we validate a route? Can we tie it to be a successful problem eliminator?  How much the method is context dependant?  Can we hook it to Test Process Improvement methods or other Key Performance/Area Indicators?  Other? Food for Thought… 81
  84. 84. Time for discussion…82 March 2014
  85. 85. RCA and Process Improvement? March 2014 83 Introduction Case Study #1 Case Study #2
  86. 86.  Process improvement should use data and RCA outputs  CI example… - process was changed, training was done to relevant people  Defects example… - lessons learned, training was done to relevant testers, discussions with development made it clear which information is required more on the defect template RCA and Process Improvement? March 2014 84
  87. 87. Analyzing test environments utilization downtime March 2014 85
  88. 88.  Analysis:  Small, yet impacting, breaks in work, caused huge impact on the release (big team of testers)  Breaks were 15-45 min each  Areas of impact were: performance, network, DB Analyzing test environments utilization downtime March 2014 86
  89. 89.  Improved process:  Defined measurements for monitoring downtime of test environments  Identify trend of areas of impact on infrastructure  Used it weekly to establish Status of Downtime, to decrease technical impacts (reached -25% waste on next release). Analyzing test environments utilization downtime March 2014 87
  90. 90. Analyzing development effort with defects found March 2014 88 15 26 11 120 65 43 32 187 12 38 210 294 588 2184 504 210 126 1344 336 378 80% 99% 21% 62% 145% 230% 286% 156% 40% 113% 0% 50% 100% 150% 200% 250% 300% 1 50 SC MAF FBF CSM E-Care E-Support Rater AR Reports BL Interface Investment vs Defects Defects Dev Effort Def % vs Dev Eff % ~28 years development. ~150 people in development team.
  91. 91.  Analysis:  Areas of defects clustering mainly in: Rater, E- Support, E-Care, AR  Relative % of defects are much higher than % of development effort (days)  Investigated UT and Staging (integration test) phases, found missing testing types done, UT & INT regression testing, performance testing, interface testing  Further considerations: Complexity, New/Legacy code, # people touching code Analyzing development effort with defects found March 2014 89
  92. 92.  Improved process:  Defined Unit test and Integration test process and strategy guidelines and work instructions  Trained programmers in UT and INT techniques and new processes  Aligned effort estimation and duration with project management and product management  Defined coverage criteria for UT and INT  Initiated test automation project on UT and INT levels Analyzing development effort with defects found March 2014 90
  93. 93. Time for discussion…91 March 2014
  94. 94.  Root Cause Analysis techniques can help us in finding the underlying problems (root causes), and get to deal with real problems  Structured RCA methods support productive thinking, identifying problems and identify more symptoms  Cause Effect Graphing (amended), Defects RCA  RCA and process improvement  Summary Retrospective March 2014 92
  95. 95. “It is not the strongest of the species that survives, nor the most intelligent but the one that is most responsive to change” Charles Darwin A changing world… 93
  96. 96. Or perhaps . . . 94 . . . the one who had anticipated all possible requirements !
  97. 97. Root Cause Analysis in Testing

×