Production Defects or Incidents are defects found on production environment by end user or reported by actual customer. Incidents negatively impact customer satisfaction and raise concerns about the overall quality and reliability of the product. Fixing the symptom may cause the issue to resurface after some time , fixing the underlying causes will ensure that the actual issues are addressed.

1. ROOT CAUSE ANALYSIS FOR INCIDENTS (OR PRODUCTION
DEFECTS)
VANQ MEETUP

2. AGENDA
 Production defects
 Why do we need to perform Root Cause Analysis
 How : 5 Why Approach with some examples
 Process and accountabilities
 Potential Root Causes
 Reports
 Closing notes

3. PRODUCTION DEFECTS
 Production Defects or Incidents are defects found on production environment by end
user or reported by actual customer
 Incidents negatively impact customer satisfaction
 Incidents raise concerns about the overall quality and reliability of the product

4. WHY DO WE NEED TO PERFORM ROOT CAUSE ANALYSIS
 Fixing the symptom may cause the issue to resurface after
some time – short term fix, frequent reoccurrence can cause
client to get frustrated and loose confidence in the
product/service over time
 Fixing the underlying causes will ensure that the actual issues
are addressed are resolved – long term fix, increased
customer satisfaction

5. HOW: 5 WHY APPROACH FOR DETERMINING ROOT CAUSE
 “5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships
underlying a particular problem” – Wikipedia
 The primary goal of the technique is to determine the root cause of a defect or problem by repeating
the question "Why?" Each answer forms the basis of the next question
 The technique was formally developed by Sakichi Toyoda and was used within the Toyota Motor
Corporation during the evolution of its manufacturing methodologies

6. 5 WHY APPROACH EXAMPLE
If the fuse was replaced, then the
issue would have reoccurred as
soon as the circuit overloaded
However,
If Filter is installed in the pump,
the robot circuit would not be not
be blown

7. 5 WHY APPROACH EXAMPLE
- Time for cleanup now, pick up all your toys
Why?
- Coz your room is dirty
Why?
- Coz you didn’t cleanup
Why do I need to cleanup?
- Because its bedtime, you have to sleep now
Why?
- OMG!!!! Coz you need to go school tomorrow
ROOT CAUSE:
He knows 5 WHY ?? whaaaattt
He is 5
He doesn’t like to sleep
He likes to annoy me
Undetermined

8. 5 WHY APPROACH EXAMPLE
Consider scenario where an incident was reported for a module not directly impacted by the most recent production
release
- No one tested this module - Missed Test scenario
Why?
- Unknown Impacted area
Why?
- Incomplete Requirements
Why?
- Change included mid sprint with limited knowledge of expected functionality
Why?
- Last minute change in roadmap as a result of strategic business plans

9. PROCESS

10. ACCOUNTABILITIES

11. POTENTIAL ROOT CAUSES
 3rd Party Limitation
 Azure Migration
 Coding Issue
 Customer Setup
 Data issue
 EDI Job Failure
 Existing Issue
 Failed Build
 Functionality Issue
 Incorrect Configuration
 Incorrect Logic
 Infrastructure
 Known Defect
 Manual Error
 Missing Configuration
 Missing Requirements
 Missing test scenario
 Others
 Outside Scope
 Performance Issue
 Undetermined
 Unknown Impact area
 User Training
 Working as designed

12. REPORTS
 Determine the Reporting guidelines – frequency and format
 Determine the report audience – External versus internal
 Determine the data that would be included
 Etc.

13. SAMPLE REPORT : QUANTITATIVE ANALYSIS

14. SAMPLE REPORT : QUANTITATIVE ANALYSIS

15. SAMPLE REPORT : QUALITATIVE ANALYSIS

16. SAMPLE REPORT: QUALITATIVEANALYSIS
 Report the action points created
 Report the current status of action points and compliance

17. PRODUCTION INCIDENT REPORTTEMPLATE
 Usually created for critical incidents
 External report – to be circulated to end user
or customer

18. CLOSING NOTES : SOME POINTERS FOR ROOT CAUSE ANALYSIS MEETING
 Accept responsibility for the incident as one team
 Do not get defensive, point fingers or throw anyone under the bus!
 Ask questions till root cause is found !
 Determine effective preventive and corrective action points
 Be accountable for the action point assigned to you
 Document the findings and circulate (if required)

19. CLOSING NOTES
 Defects identified by customers and end users in production (live) environments negatively impact
customer satisfaction and raise concerns about the overall quality and reliability of the product
 Tracking all production defects, determining root cause as a team and taking corrective or
preventative actions to avoid the same issues from happening in the future can contribute towards
increased customer satisfaction, improved product quality and more reliability
 Based on the results of Root Cause Analysis, the teams can determine the problem areas or
components that need improvement.

20. CLOSING NOTES: WHAT’S IN IT FOR ME?
 For Testers, Developers and Test Leads– opportunity to enhance knowledge and understanding of
product and infrastructure, opportunity to suggest actions that can add value and increase product
reliability
 For Quality Managers - opportunity to advocate Quality , enforcing fact that Quality is everyone’s
responsibility
 For Project Managers or Dev Leads – opportunity to use this incident analysis data to determine
problematic code components and prioritize fix for those in roadmap accordingly
 For the end customer – Customer satisfaction, confidence that they are using the products services of
a company that is quick and efficient in responding to incidents and resolving them

21. Q&A About Me
Neha Batta
Quality Assurance Manager at Cymax Group
M.Tech. , CAPM ™