Sonal Singh presented on automating bug analytics. The presentation covered: 1. Categorizing over 500,000 bugs stored in an STLC database into categories like function, degrade, requirement to understand root causes. 2. Analyzing bugs quarter-over-quarter by category and root cause to identify trends. 3. Drilling down into specific tickets to trace the root cause through questioning, like why deployment failed or why a bug wasn't caught in testing. 4. Parts of the analytics process can be automated, like exporting data between databases, but qualitative analysis like tracing root causes requires manual work.

1. 1
Can we automate bug analytics?
Sonal Singh
Booking QA Group
Leisure Product Department
Service Quality Assurance Section
Rakuten Group, Inc.
2022/11/17

2. 2
➢ Who Am I?
➢ Background of bug analytics
➢ What is bug analytics?
➢ Can we automate it?
Agenda

3. 3
➢ My name is Singh Sonal ソナル シング
➢ I’m from India
➢ I have over 10 years of experience into QA and worked with Service as
well as Product based companies.
➢ I’m in Japan since Sep, 2017 and working in Rakuten since Feb, 2018
Who am I?

4. 4
➢ I gave presentation on RTM (Requirement traceability matrix)
in Nov, 2018
Last QUES Presentation

5. 5
➢ Who Am I?
➢ Background of bug analytics
➢ What is bug analytics?
➢ Can we automate it?
Agenda

6. 6
Quality Assurance
QA is responsible for Process and Product both
Process Product
https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/

7. 7
Today’s Theme
Implement process for analytics
Process Product
https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/

8. 8
In STLC Human error = Bug
Plan Test
Develop
In the testing phase, we view human error as a bug.
We had 500K such errors and few years back we had to process this data to make improvement.

9. 9
As a first step we started to record all this data in our STLC
database.
Plan Design Implementation Execution Final report
STLC data base

10. 10
Store
STLC
Database
500k+ results
Step 1 Done
What’s Next

11. 11
REQUIREMENT
NON-FUNCTIONAL
ENVIRONMENT
INVALID
FUNCTION
DEGRADE
EXISTING
LATER
Store Categorize
STLC
Database
500k+ results
Next step: Categorize

12. 12
Category Meaning Example
FUNCTION Do not work as per specification
• Spec says ‘A’ link should be clickable but in app its not
working
DEGRADE Function worker earlier but not working now properly
• In previous R1 execution “Button A” was working fine but
in R2 its not working as expected
REQUIREMENT Not mentioned or wrong explanation in specification
• “A” user should able to purchase only one type of ticket
but actually it should purchase all types of tickets PRD is
wrong.
EXISTING Existing product bug
• QA found difference in PC an SP behavior but its expected
based on usage
UI Usability, Security, Performance issues
• If any page is taking more time to load then its treated as
usability issues
ENVIRONMENT Environment issues which affects for our test execution • Because of wrong deployment of STG we raise the ticket
INVALID QA misunderstood functionality • Spec completely misunderstood by QA
LATER It’s bug but won’t be fixed this release. Future release
• Valid bug wont be fixed in this release and considered for
future releases
NON FUNCTION （userability）（performance（security） • Performance or security related issues
OTHER_SERVICE CWD, Coupon, Salesforce, Dependent on other teams to resolve • Bugs which are dependent on other systems to solve
Categorize and define for common understanding

13. 13
REQUIREMENT
NON-FUNCTIONAL
ENVIRONMENT
INVALID
FUNCTION
DEGRADE
EXISTING
LATER
Store Categorize
STLC
Database
500k+ results
Now what?

14. 14
REQUIREMENT
NON-FUNCTIONAL
ENVIRONMENT
INVALID
FUNCTION
DEGRADE
EXISTING
LATER
Store Categorize Analyze
STLC
Database
500k+ results
Analytics flow in 3 steps

15. 15
➢ Who Am I?
➢ Background of bug analytics
➢ Can we automate it?
➢ What is bug analytics?
Agenda

16. 16
REQUIREMENT
NON-FUNCTIONAL
ENVIRONMENT
INVALID
FUNCTION
DEGRADE
EXISTING
LATER
Store Categorize
STLC
Database
500k+ results
What can be automated?
Can be
automated

17. 17
How to Automate?
Analytics dashboard
Practitest DB
Jira DB
STLC DB

18. 18
BEFORE AFTER
Fill bugs in sheet, search on Ticket by manually Use script to export Ticket tickets with bug
types, root cause
Automation tip – Reduce data collecting time

19. 19
➢ Who Am I?
➢ Background of bug analytics
➢ Can we automate it?
➢ What is bug analytics?
Agenda

20. 20
Bug Analytics flow
Step1
Major
Category and
QoQ
comparison
Step
2
RCA (Root
cause
analysis)
Step
3
Jira ticket
drill down

21. 21
• Divide projects into categories (APP , Web, Front, Back)
• Analyzing individually makes result more accurate
• Android iOS ver xx
• Big Function 01
• Small Function 01
• Backend dashboard
• Middleware Update
Step1-1: Categorize for devices Step 1: Big
category
Step 2: RCA
Step 3: Drill
down

22. 22
UI bug ▼27%
41.5%
2.6%
9.3%
3.2%
6.4%
4.5%
1.6%
28.6%
0.3% 0.6%
44.1%
22.0%
13.6%
6.8% 6.8%
1.7% 1.7% 1.7%
0.0% 0.0%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
Q2 Q3
Functional bug 3%
Step 1: Major
category
Step 2: RCA Step 3: Drill down
QoQ comparison
Step1-2: Categorize by Major category and QoQ comparison

23. 23
Improvement 2021 2022 1H 2022 2H
Data Collection ✅ ✅ ✅
Data Compare ✅ ✅ ✅
Chart Presentation ✅ ✅ ✅
Root Cause Analysis ✅ ✅
Quarter Detail compare ✅ ✅
Improvement Summary ✅ ✅
Analyzing by Categories ✅
QoQ comparison ✅
Ticket ticket detail drilldown ✅
Custom observation and suggestion ✅
Improvement History

24. 24
1
1
2
4
11
16
18
34
DEPLOYMENT ISSUE
TESTING MISTAKE
UNIT TEST MISS
INCORRECT REQUIREMENT
MISSING IMPLEMENTATION
DOW N TIME
EXTERNAL
INCORRECT CODING
Grand Total
Step 1: Major
category
Step 2: RCA Step 3: Drill down
Step2: RCA (Root cause analysis for Major category)

25. 25
Reference:
Final report of each
projects
Genchi-
Genbutsu
Step3-1: Ticket drill down Step 1: Big
category
Step 2: RCA
Step 3: Drill
down
Basic Data:
sum, fluctuation of bug
type, root cause

26. Why
why
why
26
On the top page reservation
calendar icon was not clickable.
Deployment issue
Why no code review
Why couldn’t find in Unit
testing
Source wasn’t deployed
No code review
Missing implementation
and updating sheet
No unit testing
Issue Expected
Why
Step3-2: Ticket drill down on real world Step 1: Big
category
Step 2: RCA
Step 3: Drill
down

27. 27
Judge projects quality by sum of bug, bug ratio: Refer more condition to judge
Condition Good Quality Bad Quality
Bug ration below 3% 〇 ×
Bug ration over 3% × 〇
Can skip R2 〇 ×
Few bug tickets 〇 ×
Condition Good Quality Bad Quality
Bug ration below 3% 〇 ×
Bug ration over 3% × 〇
Can skip R2 〇 ×
Few bug tickets 〇 ×
Release successfully 〇 ×
Roll back even no issue reported
from QA
× 〇
High bug ratio but no hotfix or
prod issue
△ △
Step3.3: Refer PROD ISSUE, counts of Hotfix / Roll back additionally to judge project quality
Earlier Now

28. 28
Only share suggestion but don’t know if it’s
implemented
Add result column to trace action plan
progress
Step 3.4: TRACK result carefully
Previous
quarter
This quarter

29. Can we automate bug analytics: Partially!
Quantitative analysis
By Automation
Qualitative analysis
By Manual
Solution
STLC DB
Jira DB Practitest DB

30. Thank you