SlideShare a Scribd company logo
1 of 73
QA Metrics in Agile
Look at the Quality from different angels. Real appliance.
By Volodymyr Prymakov
www.qaexperts.pro
About Me
Volodymyr Prymakov, Ukraine
Senior QA Manager at Ciklum, TCoE
Head of Performance QA Unit at TCoE
Interim Head of TCoE at Thomas Cook
•14 years in QA
• >45 projects experience
• Certified: ISTQB Advanced Test Analyst &
Manager, ICAgile CP, SAFE
/in/vladimirprimakov/
www.qaexperts.pro
What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
What is quality of Process
Quality Process brings much Business Value with keeping project or program engine safe and efficient
1. Deliver new functionality often
2. Deliver more functionality
3. Delivery quality software
4. Delivery what is planed
5. Deliver with minimal risks
6. Business trusts dev teams, processes, and produced results
www.qaexperts.pro
1. Project vision, goals, roadmap, and other plans are clear to everybody
2. Process is transparent and strait-forward
3. Product quality is always transparent
4. Technical dept is being eliminated
5. Efficient Communication and Interdependency solving
6. Shared decisions making and responsibility
7. Lessons learned and continuous improvement are in place
8. High team sprit and satisfaction
Filters/Bases
for
QA Metrics
Measuring Product
Quality
www.qaexperts.pro
Definition of Done (DoD) defines
Which quality Product should
have?
Which Way and How
thoroughly a product or
functionality should be
tested?
Min Sufficient Max Allowed
www.qaexperts.pro
Definition of Done for Stories, e.g.
• Coded and integrated to main branch,
• Unit tests are written and passing,
• API component tests are written and passing,
• Code reviewed and conforms to coding standards
Passed Dev
Testing:
• Code deployed to QA environment. Automatic API-integration and
EndToEnd smoke testing tests are passing,
• Test Design is completed,
• Positive and negative Acceptance Tests are (automated and)
passed on platforms of priority 1. Exploratory tests are passed on
platform of priority 2.
• Selective Regression is carried out by a necessity and passed.
Passed QA testing:
No (P1/P2)
bugs
outstanding
for STORY
and neighbor
functionality
Story meets INVEST criteria
Story is Signed-Off by Product Owner
www.qaexperts.pro
Definition of Done for Sprint, e.g.
• Stories that meet DoD are considered DONE, the rest goes to
next sprint.
• It should be avoided to merge Risky or buggy code to main
branch at the end of sprint!
Stories meets DoD
• Regression testing is passed with P1 test cases on at least 1
top priority platform/device
• Smoke or exploratory testing is passed on platforms/devices
of priority 1 and 2
Cover Regression
Test
• Main Integration tests for live 3d-party services are passed on
STAGE or other environments3d party Integration
Test
No (P1/P2)
bugs
outstanding
for product
and neighbor
products.
Sprint is Signed-Off by Chief Product Owner
www.qaexperts.pro
Definition of Done for RELEASE, e.g.
• All Stories, which were merged to Main branch, meets DoD for stories.
• Risky or buggy code is not merged to main branch or is toggled off or is
reverted!
Stories meets DoD
• Regression testing is passed for all priority 1 platforms/devices.
• Exploratory testing is passed for priority 2 platforms/devices.
• Smoke testing is passed for priority 3 platforms.
Full Regression Test
• All Integration tests for live 3d-party services are passed on STAGE or
other environments
• There is no other 3rd party release ongoing that may impact release
negatively
3d party Integration Test
and Releases
• UAT is passed on STAGE environment
UAT
• Performance Testing successfully passed
• Security Testing successfully passed
• and other types of testing and preparations to release
Other Types of Testing
No (P1/P2)
bugs
outstanding
for product
and neighbor
products
Release GO Aproval from by Chief Product Owner, QA team, Release Manager and
respective Business representatives
All other unresolved
bugs are tracked,
reviewed, and planned
when to be fixed
www.qaexperts.pro
Definition of Done
dictates you
which Activities and QA metrics
to perform and monitor!
www.qaexperts.pro
QAMetrics
www.qaexperts.pro
Types of Metrics
Quantative - Information that is directly measured as a
number
Qualitative - Information that results from human
observation and analysis
www.qaexperts.pro
Types of Metrics
1. Absolute metrics
2. Relative metrics
3. Derivative metrics
Purpose of Metrics.
1. Goal Metrics: To measure results/goals
2. Actionable metrics: to make decision
3. Predictive metrics: to predict future
www.qaexperts.pro
Stages at which QA and other metrics are needed
Implementation period
• Daily Monitoring of
Quality, product
implementation, and
test progress
Key sprint and release
points
• QA Report Before
sprint Demo
• QA Reports During
stabilization /
regression / UAT
period
• QA report Before
Release
• QA Reminder report
when product has
critical quality state
or low product
implementation or
testing progress
After Release
• Release quality and
process analysis
• Quality and process
Trends and analysis
• Team efficiency
analysis
www.qaexperts.pro
QA Metrics
for
Daily
Monitoring
www.qaexperts.pro
Goals of Daily Quality Monitoring
1. What is the current quality of the
product?
2. How much work is left?
3. Do we have enough time to finish and
stabilize the product?
www.qaexperts.pro
!!!Important:
Breakdown Defect Metrics
Always breakdown your metrics to the following groups:
All Bugs! – shows general product quality state
Critical and Blocker bugs – Shows whether we
meet Readiness criteria!
Lower Priority bugs– Shows the rest of bugs,
general amount of which can negatively influence on the
end-user’s impression about the product.
www.qaexperts.pro
Defects reported by Date
Critical and Blocker bugs Major and lower priority bugs
Jira “Recently Created Chart” Gadget
Allows you track new bugs and immediately react to them!
www.qaexperts.pro
Open Defects reported Today
Jira “Filter Results” Gadget
Allows you to quickly review new bugs
www.qaexperts.pro
Open Defects By Component/Platform
Major and lower priority bugs
Jira “Pie Chart” Gadget
Critical and Blocker bugs
Shows which
component/platform
is not ready for release
or requires bug fixes
www.qaexperts.pro
Defects By Component/Platform for Current
Affected Version (any Status)
Major and lower priority bugs
Jira “Pie Chart” Gadget
Critical and Blocker bugs
Shows which
component/platform
require more thorough
testing,
e.g. (Selective)
Regression testing
www.qaexperts.pro
Critical Open Bugs by Status
Jira “Issue Statistics” Gadget
Allows you to quickly get Status of Open Critical Bugs and understand whether we have enough time to fix them
www.qaexperts.pro
Open Bugs by Priority
Jira “Issue Statistics” Gadget
Shows general quality of the product
www.qaexperts.pro
!!!NOT-assigned Open Critical Defects
Jira “Issue Statistics” Gadget
To Fix Version
To Team
(for big teams only)
To Sprint To Developer
Shows probably lost or forgotten Critical open bugs, which should be processed or escalated immediately!
www.qaexperts.pro
www.qaexperts.pro
Workload:
Assigned Open Tasks and Bugs for Developers/QAs
Jira “Issue Statistics” Gadget
By teams By Developers
For current Fix Version or sprint
By QAs
Stories
By teams By Developers By QAs
Bugs
To assure that there is no bottlenecks and overloads. To Remind about
work assigned!
To spread load equally between teams, Developers and QAs!
Release scope status by developers/QAs/teams
If by the end of sprint or before stabilization period at release a lot of stories are in open status,
it can bring risk of late merge status, not enough time for testing, not ability to finish functionality etc.
Action: Try to spread load equally by teams, QAs and Dev Specialists. Remind about work Not Done.
Postpone certain functionality to next Release!
Stories Bugs
Jira “Two Dimensional Filter Statistics” Gadget
www.qaexperts.pro
Stories/Bug Status By Fix Version
created >= -365d
Stories Bugs
created >= -180d
Allows to understand ability to finish/fix all tasks or bugs in time for a certain sprint/release
Jira “Two Dimensional Filter Statistics” Gadget
www.qaexperts.pro
Current Fix Version Progress
Shows general release/sprint ProgressJira “Release Report”
www.qaexperts.pro
Team Burndown Chart
www.qaexperts.pro
Kanban board for All Open Critical and Blocker Bugs
(For big teams)
Allows to
accumulate
and track all
open Critical
and Blocker
bugs in 1 place
www.qaexperts.pro
Regression testing
Recommended Statuses:
• Failed with Critical and Blocker bugs
• Failed with Major and lower priority bugs
• Failed by automated tests
• Blocked
• Not tested
• In Progress
• Passed
www.qaexperts.pro
Regression Testing Coverage and Results
For Release/Milestone
– General Results
By Cycle/Stages
- Every cycle should has certain purpose, e.g:
1. Initial Feedback about quality,
2. Exploratory testing,
3. Confirmation testing
4. 3rd Party services testing.
www.qaexperts.pro
Regression Testing Coverage and Results
By Product Component
• by platform/device
• by test priority:
• Priority 1
• Priority 2
• Priority 3
By Type of Test:
• Smoke test
• Regression test,
• UAT test
www.qaexperts.pro
QA Metrics
For Regular
Quality Data
Reports
(Snapshots)
www.qaexperts.pro
Sprint Report
1. Story Delivery rate % = Actual Stories Completed vs. Committed Stories, e.g.: 90%
2. Amount of Stories met DoD, e.g.: 25
3. Amount of Stories Moved to a new sprint, e.g.: 4
Because of 1 not implemented, 1 Reopen because of Critical Bugs, 2 Not Tested
Specifies the reasons, risks and recommendations.
4. Regression testing coverage and results: Graph
5. Cross platform testing coverage and results: Graph
6. 3d-party integration testing results: Graph or written explanation.
7. Testing and testing activities NOT done – describe reasons, connected risks, and recommendations.
8. Open Bugs:
All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link to jira)
Open Major bugs opened during the sprint, e.g: 40 Major bugs open during sprint. (link to jira)
9. Point to the buggiest product component and platform: Graph and explanation
10. General Conclusions
www.qaexperts.pro
Regression testing reports, daily or by cycles
Reports about Regression testing, cross platform testing, 3d-party integration testing, 3d party releases testing, etc.
1. Name of the Regression testing stage, e.g.: Regression Cycle 1 (Preliminary feedback)
2. Coverage and results: Graph
3. Open Bugs
New (created today) and All Open Critical and Blocker bugs:
New 4 Critical&Blocker Bugs, (All 5 Critical&Blocker Bugs) (list or links to jira)
New (Created today) and All Open Major bugs opened during the release:
New 10 Major bugs, (All Major bugs created for current release) (links to jira)
4. Specify Time needed to finish regression testing and/or necessity in additional regression and other testing.
5. Point to the buggiest product component, platform, or functionality if needed.
6. Recommendations
www.qaexperts.pro
Release Readiness testing Report (by necessity)
1. Specify functional testing accomplished and its status:
Regression and cross-platform testing,
3dparty integration testing,
UAT
e.g.: Written explanations and Links to Test management system,
or Graphs of testing coverage and results.
2. Testing and testing activities NOT done – describe reasons, connected risks and recommendations.
3. Specify Non-functional testing status, like performance and security testing.
Whether the product is ready or not ready from non-functional point of view.
Describe connected risks and recommendations.
4. Open Bugs.
All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link)
Open Major bugs opened during the release and All Major bugs, e.g: 40 Major bugs open during release.,
70 Major bugs are Open in general
(list or links to jira)
5. Functionality Not ready for release, etc.: (list or link)+ written explanation
6. General Conclusions!
www.qaexperts.pro
Regression and Release Readiness Report example
www.qaexperts.pro
Reminder Reports (Be necessity)
1. Critical and Blocker Bugs in open state.
2. A Lot of Open Major bugs
3. A lot of Bugs and stories Not Done in the middle or later
stage of sprints: In development, in Test, etc.
4. Availability of not tested stories during stabilization period.
5. Inability to finish testing in time because of certain issues.
Reminder emails or chat messages about:
www.qaexperts.pro
Trend-based
QA Metrics
for
for process
and product
quality
analysis www.qaexperts.pro
Milestones for trends
• AffectedVersion – AffectedVersion field in jira
• Time Span - e.g. period between start and end
date of sprint
Time based conditions in jira queries:
a) createdDate >= '2017-08-30' and
createdDate <='2017-09-12 ‘
b) status changed TO 'READY TO TEST' after
'2017-08-30' before '2017-09-12‘
c) assignee was in (‘user’)
• Sprint – Sprint Field in jira, etc.
www.qaexperts.pro
Found on Environment
Critical
and
Blocker
bugs
Shows leaks of
Critical and
Blocker bugs to
later
development
stages
Pivot Report
www.qaexperts.pro
Found on Environment
All bugs
Shows leaks of
bugs to later
development
stages
Pivot Report
www.qaexperts.pro
Bugs By Priority – Any Status
All Bugs
by
Aff. Version
Shows absolute
release quality
history
– decrease in
number of all
and critical bugs
shows process
improvements
Pivot Report
Note: Should be considered carefully when the team size and product increment increase.
Note: Blank version is bad, and it shows that the Affected Version is not specified for
certain bugs and thus they can be lost www.qaexperts.pro
Bugs By Priority – OPEN (not-closed)
All Bugs
by
Aff. Version
Shows still open
bugs for
releases
– Ideally No
critical and
blocker bugs
should be here
Pivot Report Note: If critical bugs are shown here for previous releases, they might be lost, forgotten or
blocked. www.qaexperts.pro
Bugs by RootCause
Typical Root causes of Bugs:
FrontendOthers
Backend
Frontend
+ Backend
Environme
nt
issues
3d party
components
www.qaexperts.pro
Bugs by RootCause
Critical
and Blocker
bugs
Shows Root cause of all bugs
Pivot Report
All Bugs
Shows Root cause of the most
of important bugs
www.qaexperts.pro
Created Vs Resolved - Cumulative
Critical
and Blocker
Bugs
Created should be less than resolved
Jira “Created vs. Resolved Chart” Gadget
Bug found vs. fixed chart is a defect analysis line chart that lets us see defect
removal process patterns and understand defect management effectiveness
www.qaexperts.pro
Created Vs Resolved - Cumulative
All Bugs
Difference should be minimal
and If it is big and no positive
trend is shown then probably
Bagathon sprint should be
planned
Jira “Created vs. Resolved Chart” Gadget
www.qaexperts.pro
Defects Dept
Critical and Blocker Bugs
Note: Should Be always close to Zero for Critical bugs
All Bugs
Spreadsheets + advanced Jira time period based queries
Defects dept is a cumulative difference between Resolved and Open bugs
= (Open in sprint N - Resolved in sprint N)+ (Open in sprint N+1- Resolved in sprint N+1 )+…
Note: Shows accumulation of bugs in time
www.qaexperts.pro
Bugs Opened vs Stories Delivered
There should be correlation between the metrics - more stories you deliver more bugs may appear.
Ideal trend is to deliver more user stories with less bugs.
!!!Pay attention to anomalies and investigate them deeper.
Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
Bugs per Story Point
Critical
and Blocker Bugs
If number
decreases in
time it means
that we
improve QA
process Note: This metric is very good for cases when team size or
product increment is variable or change (increase)
All Bugs
Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
Bugs per Story Point
Other Variations:
1. Bugs per requirement
2. Bugs per developer
3. Bugs per 1000/Code
4. Time lost for bugs h - Time spent for Bugs vs time spent for stories
Note: Very useful to for making future project estimations.
www.qaexperts.pro
Fix Rate
Should be close to 100% or higher
Spreadsheets + Advanced Jira period based queries
Critical and Blocker Bugs
All Bugs
Fix Rate = Closed bugs / Open bugs
Ideally should be close to 100%
www.qaexperts.pro
“Average Age” and “Time in status”
For Critical
and Blocker
Bugs
Jira “Average Age” and “Time in Status” Gadgets
Average Age - Displays the average number of days issues have been unresolved.
For critical and Blocker bugs this time should be very short.
If process improves it should decrease in time.
Average Time in status, e.g. Open status
- Displays the average number of days issues have spent in status.
For critical and Blocker bugs this time should be very short.
If process improves it should decrease in time.
www.qaexperts.pro
Cumulative Flow Chart
Jira Agile Reports
To find process bottlenecks and predict project close time
www.qaexperts.pro
Bugs by Components/Platforms
Pivot or other spreadsheet Report
Bugs trend By Component – Critical and Blocker / All Bugs
Shows weakest component of the product, shows
where additional testing should be performed, shows
where technical Improvements should be made,
Shows whether component quality improves
Bugs trend By Platform – Critical and Blocker / All Bugs.
Shows amount of cross-platform bugs, shows the
weakest platform requiring more testing and
stabilization.
.
www.qaexperts.pro
Sprint and Business Metrics
Estimates Accuracy % = Estimated amount of
effort/Actual amount of effort-1
Story Delivery Rate % = Actual Stories Completed /
Committed Stories
% of Change Requests = Number of change requests /
Number of Initial Backlog stories
Change Requests Costs = Time spent of change
requests * Average Rate of developers on the project
% Costs of Defect = Average Time for defect fix *
Average developer rate on the project
 Trends
 For Sprint
 General value
www.qaexperts.pro
User Satisfaction
Number of hot fixes had to be made in this or that release.
Shows leaks of Critical bugs on prod.
Bugs found by end-users
Critical/Blocker (Shows leaks of Critical bugs on prod.)
Major and Low priority
Customer Complaints and Feedbacks reports. (Net Promoter Score)
A/B Testing results
Shows which functionality is more preferred
Other Marketing investigations and surveys
All these answer the following questions:
a) What are the weakest points of the product?
b) What functionality is most used ?
c) Which focus in product development and testing should be made?
www.qaexperts.pro
Qualitative Metrics about the project state
If you want to gather more metrics about project processes, approaches,
technology, communication, mood in teams, etc.
Just ask people to provide evaluation and feedback
Experts evaluation and/or Audit
Surveys
Retrospectives
1-2-1 and other meeting
Rumors/small talks at Smoking room/kitchen 
www.qaexperts.pro
QA Metrics
for
Team and
Individual
Efficiency
analysis
www.qaexperts.pro
Developers and Dev Team efficiency
REOPENED:
Reopened Bugs Ratio % = Reopened bugs vs Resolved bugs
Reopened Bugs Ratio % = Reopened Stories vs Stories
delivered to testing
STORY POINTS DELIVERED
STORY DEFECTS ISSUED
STORY DEFECTs/STORY POINTs rate %
QUALITATIVE METRICS: Gather qualitative metrics about
developers: approach to coding, complexity of code,
contribution to innovation, communication efficiency, quick
reaction to problems, etc.  Per PROJECT in general
 Per Team
 Per specific Developer
www.qaexperts.pro
QAs and QA team Efficiency
REJECTED
Absolute Number of rejected bugs:
All resolutions,
Not a bug,
Duplicate
Cannot reproduce
Incomplete
Rejected Rate % = rejected vs created
BUGs LEAKS: to PROD other later stages of development.
Break down by
Per product part
Per functionality
So you can correlate it with a certain QA or group of QAs
Separately:
 For critical and blocker
 For Lower priority
defects
 Per PROJECT in general
 Per QA Team
 Per specific QA
www.qaexperts.pro
QAs and QA team Efficiency
WORK DONE:
 Stories Tested
 Created bugs Critical and Blocker (not rejected)
 Created bugs Major and lower priority (not Rejected)
 Test cases Created (passed DoD). Test cases rejected.
 Test cases Executed.
 Automation scripts Created (passed DoD). Test cases
rejected.
 Advanced test artifacts created and activities
accomplished: test suites, plans, release support, etc.
QUALITATIVE METRICS: Gather qualitative metrics about QAs:
Complexity of tasks, Communication efficiency, Proactivity,
Contribution to innovation, Test management skils, Conflict solving,
etc.
 Per PROJECT in general
 Per QA team
 Per specific QA
 For Sprint (Velocity)
 Trends
 General value
www.qaexperts.pro
QA Metrics for
Test
Automation
www.qaexperts.pro
 Unit test coverage & number of test
 API testing coverage & number of test
for product in general
by component
 GUI Automation testing coverage & number of test
 for product in general
 by product part
 by platform/device
 by type of test
end-to-end test
Smoke test
Regression test
UAT test
 by test priority:
Priority 1
Priority 2
Priority 3
Automation Coverage & Results
www.qaexperts.pro
Automation Efficiency
AUTOMATION SPEED:
 Automation tests Frequency
 Duration of tests Running
 Velocity of script creation and maintenance
 Time for script creation and maintenance
AUTOMATION EFFICIENCY:
Stability of automation
False Positives
Root Cause for failure
Bugs found by Automation
Critical and Blocker
Other bugs
Bugs missed with automation.
Critical and Blocker
Other bugs
ROI – Time automation economy, etc.
Cost per automated test
 Per PROJECT in general
 Per QA team
 Per specific QA
 For Sprint (Velocity)
 Trends
 General value
www.qaexperts.pro
QA Metrics
TOOLS
www.qaexperts.pro
Tools
 Jira + Special time-
related JQL
 Confluence + Jira and
other Macros in it
 Spreadsheet processors:
Googlesheets, Excel
 TestTrail
www.qaexperts.pro
JiraReportingInGsheets Tool
Link to test version:
https://chrome.google.com/webstore/detail/jirareportingingsheets/jmjoohlbcigmigokjldgcgnfofcagock
JiraReportingInGSheets add-on
for GoogleSheets
allows to automatically build custom jira
reports in google sheets
Created by QAExperts.pro
www.qaexperts.pro
Thank you
Darya Alymova and Sergey Sosnitskiy
for their contribution to certain metrics definition and
implementation
www.qaexperts.pro
QA Experts.Pro Workshops
2 days workshop
November 2017
http://qaexperts.pro/#signup
www.qaexperts.pro
Thank you
www.qaexperts.pro

More Related Content

What's hot

Istqb intro with question answer for exam preparation
Istqb intro with question answer for exam preparationIstqb intro with question answer for exam preparation
Istqb intro with question answer for exam preparationKevalkumar Shah
 
Test Estimation
Test Estimation Test Estimation
Test Estimation SQALab
 
Basic software-testing-concepts
Basic software-testing-conceptsBasic software-testing-concepts
Basic software-testing-conceptsmedsherb
 
Manual testing interview questions and answers
Manual testing interview questions and answersManual testing interview questions and answers
Manual testing interview questions and answersTestbytes
 
Software testing and quality assurance
Software testing and quality assuranceSoftware testing and quality assurance
Software testing and quality assuranceBenjamin Baumann
 
Lesson 4...Bug Life Cycle
Lesson 4...Bug Life CycleLesson 4...Bug Life Cycle
Lesson 4...Bug Life Cyclebhushan Nehete
 
TDD and Unit Testing in Golang
TDD and Unit Testing in GolangTDD and Unit Testing in Golang
TDD and Unit Testing in GolangSofian Hadiwijaya
 
How to use selenium successfully
How to use selenium successfullyHow to use selenium successfully
How to use selenium successfullyTEST Huddle
 
Test-Driven Development
Test-Driven DevelopmentTest-Driven Development
Test-Driven DevelopmentJohn Blum
 
Unit testing & TDD concepts with best practice guidelines.
Unit testing & TDD concepts with best practice guidelines.Unit testing & TDD concepts with best practice guidelines.
Unit testing & TDD concepts with best practice guidelines.Mohamed Taman
 
Test Driven Development (TDD)
Test Driven Development (TDD)Test Driven Development (TDD)
Test Driven Development (TDD)David Ehringer
 
Software Testing As a Career Path
Software Testing As a Career PathSoftware Testing As a Career Path
Software Testing As a Career PathRock Interview
 
Software testing
Software testingSoftware testing
Software testingBharath K
 
Selenium Testing Project report
Selenium Testing Project reportSelenium Testing Project report
Selenium Testing Project reportKapil Rajpurohit
 
Just Java2007 - Daniel Wildt - Tools For Java Test Automation
Just Java2007 - Daniel Wildt - Tools For Java Test AutomationJust Java2007 - Daniel Wildt - Tools For Java Test Automation
Just Java2007 - Daniel Wildt - Tools For Java Test AutomationDaniel Wildt
 

What's hot (19)

Istqb intro with question answer for exam preparation
Istqb intro with question answer for exam preparationIstqb intro with question answer for exam preparation
Istqb intro with question answer for exam preparation
 
Test Estimation
Test Estimation Test Estimation
Test Estimation
 
Unit Testing Your Application
Unit Testing Your ApplicationUnit Testing Your Application
Unit Testing Your Application
 
Basic software-testing-concepts
Basic software-testing-conceptsBasic software-testing-concepts
Basic software-testing-concepts
 
Manual testing interview questions and answers
Manual testing interview questions and answersManual testing interview questions and answers
Manual testing interview questions and answers
 
Testing in java
Testing in javaTesting in java
Testing in java
 
Software testing and quality assurance
Software testing and quality assuranceSoftware testing and quality assurance
Software testing and quality assurance
 
Lesson 4...Bug Life Cycle
Lesson 4...Bug Life CycleLesson 4...Bug Life Cycle
Lesson 4...Bug Life Cycle
 
TDD and Unit Testing in Golang
TDD and Unit Testing in GolangTDD and Unit Testing in Golang
TDD and Unit Testing in Golang
 
How to use selenium successfully
How to use selenium successfullyHow to use selenium successfully
How to use selenium successfully
 
Formal method
Formal methodFormal method
Formal method
 
Test-Driven Development
Test-Driven DevelopmentTest-Driven Development
Test-Driven Development
 
Unit testing & TDD concepts with best practice guidelines.
Unit testing & TDD concepts with best practice guidelines.Unit testing & TDD concepts with best practice guidelines.
Unit testing & TDD concepts with best practice guidelines.
 
Test Driven Development (TDD)
Test Driven Development (TDD)Test Driven Development (TDD)
Test Driven Development (TDD)
 
Test Automation
Test AutomationTest Automation
Test Automation
 
Software Testing As a Career Path
Software Testing As a Career PathSoftware Testing As a Career Path
Software Testing As a Career Path
 
Software testing
Software testingSoftware testing
Software testing
 
Selenium Testing Project report
Selenium Testing Project reportSelenium Testing Project report
Selenium Testing Project report
 
Just Java2007 - Daniel Wildt - Tools For Java Test Automation
Just Java2007 - Daniel Wildt - Tools For Java Test AutomationJust Java2007 - Daniel Wildt - Tools For Java Test Automation
Just Java2007 - Daniel Wildt - Tools For Java Test Automation
 

Similar to QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных сторон. Реальное применение

QE Presentation.pptx
QE Presentation.pptxQE Presentation.pptx
QE Presentation.pptxAllanJayIsip1
 
Fundamentals of software testing
Fundamentals of software testingFundamentals of software testing
Fundamentals of software testingNoha Gamal
 
SoftwareTesting_Interview_Ques.pptx
SoftwareTesting_Interview_Ques.pptxSoftwareTesting_Interview_Ques.pptx
SoftwareTesting_Interview_Ques.pptxmahadev46
 
Zero touch QA automation platform for DevOps
Zero touch QA automation platform for DevOpsZero touch QA automation platform for DevOps
Zero touch QA automation platform for DevOpsTaUB Solutions
 
Understand release engineering
Understand release engineeringUnderstand release engineering
Understand release engineeringgaoliang641
 
Enhancing Software Quality
Enhancing Software QualityEnhancing Software Quality
Enhancing Software QualityAnand Prabhala
 
Software testing interview Q&A – Part 2
Software testing interview Q&A – Part 2Software testing interview Q&A – Part 2
Software testing interview Q&A – Part 2Khoa Bui
 
Continuous delivery is more than dev ops
Continuous delivery is more than dev opsContinuous delivery is more than dev ops
Continuous delivery is more than dev opsAgile Montréal
 
Project management for qa manager
Project management for qa managerProject management for qa manager
Project management for qa managergaoliang641
 
Performance Testing in Continous Delivery
Performance Testing in Continous DeliveryPerformance Testing in Continous Delivery
Performance Testing in Continous DeliveryPablo Gallego Hermann
 
Process Guidelines V2
Process Guidelines V2Process Guidelines V2
Process Guidelines V2Imaginea
 
Experiment Your Way to Product Success: How User Acceptance Testing Can Save ...
Experiment Your Way to Product Success: How User Acceptance Testing Can Save ...Experiment Your Way to Product Success: How User Acceptance Testing Can Save ...
Experiment Your Way to Product Success: How User Acceptance Testing Can Save ...Aggregage
 
Introduction and Role of a manual testing in a SDLC
Introduction and Role of a manual testing in a SDLC Introduction and Role of a manual testing in a SDLC
Introduction and Role of a manual testing in a SDLC minimini22
 

Similar to QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных сторон. Реальное применение (20)

QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)
 
Qa management in big agile teams
Qa management in big agile teamsQa management in big agile teams
Qa management in big agile teams
 
QE Presentation.pptx
QE Presentation.pptxQE Presentation.pptx
QE Presentation.pptx
 
Fundamentals of software testing
Fundamentals of software testingFundamentals of software testing
Fundamentals of software testing
 
SoftwareTesting_Interview_Ques.pptx
SoftwareTesting_Interview_Ques.pptxSoftwareTesting_Interview_Ques.pptx
SoftwareTesting_Interview_Ques.pptx
 
Zero touch QA automation platform for DevOps
Zero touch QA automation platform for DevOpsZero touch QA automation platform for DevOps
Zero touch QA automation platform for DevOps
 
Alpha & Beta Testing
Alpha & Beta TestingAlpha & Beta Testing
Alpha & Beta Testing
 
MyPresentation2
MyPresentation2MyPresentation2
MyPresentation2
 
Understand release engineering
Understand release engineeringUnderstand release engineering
Understand release engineering
 
Enhancing Software Quality
Enhancing Software QualityEnhancing Software Quality
Enhancing Software Quality
 
Software testing interview Q&A – Part 2
Software testing interview Q&A – Part 2Software testing interview Q&A – Part 2
Software testing interview Q&A – Part 2
 
Why Automate
Why AutomateWhy Automate
Why Automate
 
Continuous delivery is more than dev ops
Continuous delivery is more than dev opsContinuous delivery is more than dev ops
Continuous delivery is more than dev ops
 
Project management for qa manager
Project management for qa managerProject management for qa manager
Project management for qa manager
 
Performance Testing in Continous Delivery
Performance Testing in Continous DeliveryPerformance Testing in Continous Delivery
Performance Testing in Continous Delivery
 
Mobile Quality Assurance
Mobile Quality AssuranceMobile Quality Assurance
Mobile Quality Assurance
 
Process Guidelines V2
Process Guidelines V2Process Guidelines V2
Process Guidelines V2
 
Qulaity strategy template
Qulaity strategy templateQulaity strategy template
Qulaity strategy template
 
Experiment Your Way to Product Success: How User Acceptance Testing Can Save ...
Experiment Your Way to Product Success: How User Acceptance Testing Can Save ...Experiment Your Way to Product Success: How User Acceptance Testing Can Save ...
Experiment Your Way to Product Success: How User Acceptance Testing Can Save ...
 
Introduction and Role of a manual testing in a SDLC
Introduction and Role of a manual testing in a SDLC Introduction and Role of a manual testing in a SDLC
Introduction and Role of a manual testing in a SDLC
 

More from QAFest

QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилинQA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилинQAFest
 
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The FutureQA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The FutureQAFest
 
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...QAFest
 
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...QAFest
 
QA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать большеQA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать большеQAFest
 
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiledQA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiledQAFest
 
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгораниемQA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгораниемQAFest
 
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...QAFest
 
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...QAFest
 
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium ClusterQA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium ClusterQAFest
 
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...QAFest
 
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...QAFest
 
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automationQA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automationQAFest
 
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...QAFest
 
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...QAFest
 
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях ITQA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях ITQAFest
 
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложенииQA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложенииQAFest
 
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...QAFest
 
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...QAFest
 
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22QAFest
 

More from QAFest (20)

QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилинQA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
 
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The FutureQA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
 
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
 
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
 
QA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать большеQA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать больше
 
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiledQA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
 
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгораниемQA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
 
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
 
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
 
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium ClusterQA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
 
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
 
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
 
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automationQA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
 
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
 
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
 
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях ITQA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
 
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложенииQA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
 
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
 
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
 
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
 

Recently uploaded

647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
647291105-Ppt-Arts-10-4th-Quarter-1.pdfi647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
647291105-Ppt-Arts-10-4th-Quarter-1.pdfijoemmbrillantes
 
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...Nguyen Thanh Tu Collection
 
Views in Odoo 17 - Kanban View in odoo 17
Views in Odoo 17 - Kanban View  in odoo 17Views in Odoo 17 - Kanban View  in odoo 17
Views in Odoo 17 - Kanban View in odoo 17Celine George
 
Basic cosmetics prepared by my student Mr. Balamurugan, II Maths, 2023-2024
Basic cosmetics prepared by my student Mr. Balamurugan, II Maths, 2023-2024Basic cosmetics prepared by my student Mr. Balamurugan, II Maths, 2023-2024
Basic cosmetics prepared by my student Mr. Balamurugan, II Maths, 2023-2024St.John's College
 
4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptx4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptxmary850239
 
Self directed Learning - SDL, introduction to SDL
Self directed Learning - SDL, introduction to SDLSelf directed Learning - SDL, introduction to SDL
Self directed Learning - SDL, introduction to SDLspmdoc
 
Shark introduction Morphology and its behaviour characteristics
Shark introduction Morphology and its behaviour characteristicsShark introduction Morphology and its behaviour characteristics
Shark introduction Morphology and its behaviour characteristicsArubSultan
 
What is Property Fields in Odoo 17 ERP Module
What is Property Fields in Odoo 17 ERP ModuleWhat is Property Fields in Odoo 17 ERP Module
What is Property Fields in Odoo 17 ERP ModuleCeline George
 
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC ...
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC ...BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC ...
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC ...Nguyen Thanh Tu Collection
 
4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptxmary850239
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...Nguyen Thanh Tu Collection
 
The Shop Floor Overview in the Odoo 17 ERP
The Shop Floor Overview in the Odoo 17 ERPThe Shop Floor Overview in the Odoo 17 ERP
The Shop Floor Overview in the Odoo 17 ERPCeline George
 
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...Osopher
 
Sarah Lahm In Media Res Media Component
Sarah Lahm  In Media Res Media ComponentSarah Lahm  In Media Res Media Component
Sarah Lahm In Media Res Media ComponentInMediaRes1
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...Nguyen Thanh Tu Collection
 
DBMSArchitecture_QueryProcessingandOptimization.pdf
DBMSArchitecture_QueryProcessingandOptimization.pdfDBMSArchitecture_QueryProcessingandOptimization.pdf
DBMSArchitecture_QueryProcessingandOptimization.pdfChristalin Nelson
 
Supply agency market - aiming high 2.pdf
Supply agency market - aiming high 2.pdfSupply agency market - aiming high 2.pdf
Supply agency market - aiming high 2.pdftomeskell101
 
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdfMJDuyan
 

Recently uploaded (20)

647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
647291105-Ppt-Arts-10-4th-Quarter-1.pdfi647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
 
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
 
Views in Odoo 17 - Kanban View in odoo 17
Views in Odoo 17 - Kanban View  in odoo 17Views in Odoo 17 - Kanban View  in odoo 17
Views in Odoo 17 - Kanban View in odoo 17
 
Basic cosmetics prepared by my student Mr. Balamurugan, II Maths, 2023-2024
Basic cosmetics prepared by my student Mr. Balamurugan, II Maths, 2023-2024Basic cosmetics prepared by my student Mr. Balamurugan, II Maths, 2023-2024
Basic cosmetics prepared by my student Mr. Balamurugan, II Maths, 2023-2024
 
4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptx4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptx
 
Self directed Learning - SDL, introduction to SDL
Self directed Learning - SDL, introduction to SDLSelf directed Learning - SDL, introduction to SDL
Self directed Learning - SDL, introduction to SDL
 
Shark introduction Morphology and its behaviour characteristics
Shark introduction Morphology and its behaviour characteristicsShark introduction Morphology and its behaviour characteristics
Shark introduction Morphology and its behaviour characteristics
 
What is Property Fields in Odoo 17 ERP Module
What is Property Fields in Odoo 17 ERP ModuleWhat is Property Fields in Odoo 17 ERP Module
What is Property Fields in Odoo 17 ERP Module
 
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC ...
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC ...BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC ...
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC ...
 
4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
 
The Shop Floor Overview in the Odoo 17 ERP
The Shop Floor Overview in the Odoo 17 ERPThe Shop Floor Overview in the Odoo 17 ERP
The Shop Floor Overview in the Odoo 17 ERP
 
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
Healthy Minds, Flourishing Lives: A Philosophical Approach to Mental Health a...
 
Israel Genealogy Research Assoc. April 2024 Database Release
Israel Genealogy Research Assoc. April 2024 Database ReleaseIsrael Genealogy Research Assoc. April 2024 Database Release
Israel Genealogy Research Assoc. April 2024 Database Release
 
Chi-Square Test Non Parametric Test Categorical Variable
Chi-Square Test Non Parametric Test Categorical VariableChi-Square Test Non Parametric Test Categorical Variable
Chi-Square Test Non Parametric Test Categorical Variable
 
Sarah Lahm In Media Res Media Component
Sarah Lahm  In Media Res Media ComponentSarah Lahm  In Media Res Media Component
Sarah Lahm In Media Res Media Component
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
 
DBMSArchitecture_QueryProcessingandOptimization.pdf
DBMSArchitecture_QueryProcessingandOptimization.pdfDBMSArchitecture_QueryProcessingandOptimization.pdf
DBMSArchitecture_QueryProcessingandOptimization.pdf
 
Supply agency market - aiming high 2.pdf
Supply agency market - aiming high 2.pdfSupply agency market - aiming high 2.pdf
Supply agency market - aiming high 2.pdf
 
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
 

QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных сторон. Реальное применение

  • 1. QA Metrics in Agile Look at the Quality from different angels. Real appliance. By Volodymyr Prymakov www.qaexperts.pro
  • 2. About Me Volodymyr Prymakov, Ukraine Senior QA Manager at Ciklum, TCoE Head of Performance QA Unit at TCoE Interim Head of TCoE at Thomas Cook •14 years in QA • >45 projects experience • Certified: ISTQB Advanced Test Analyst & Manager, ICAgile CP, SAFE /in/vladimirprimakov/ www.qaexperts.pro
  • 3. What is quality of an application Requirements fit business needs Requirements Has high quality Product corresponds to requirements Product No Outstanding Defects Product satisfies real life conditions Product Meets user needs Minimal QA Influence Medium QA Influence High QA Influence High QA Influence High QA Influence Guessing Verification Verification Verification Validation Validation Minimal QA Influence www.qaexperts.pro
  • 4. What is quality of Process Quality Process brings much Business Value with keeping project or program engine safe and efficient 1. Deliver new functionality often 2. Deliver more functionality 3. Delivery quality software 4. Delivery what is planed 5. Deliver with minimal risks 6. Business trusts dev teams, processes, and produced results www.qaexperts.pro 1. Project vision, goals, roadmap, and other plans are clear to everybody 2. Process is transparent and strait-forward 3. Product quality is always transparent 4. Technical dept is being eliminated 5. Efficient Communication and Interdependency solving 6. Shared decisions making and responsibility 7. Lessons learned and continuous improvement are in place 8. High team sprit and satisfaction
  • 6. Definition of Done (DoD) defines Which quality Product should have? Which Way and How thoroughly a product or functionality should be tested? Min Sufficient Max Allowed www.qaexperts.pro
  • 7. Definition of Done for Stories, e.g. • Coded and integrated to main branch, • Unit tests are written and passing, • API component tests are written and passing, • Code reviewed and conforms to coding standards Passed Dev Testing: • Code deployed to QA environment. Automatic API-integration and EndToEnd smoke testing tests are passing, • Test Design is completed, • Positive and negative Acceptance Tests are (automated and) passed on platforms of priority 1. Exploratory tests are passed on platform of priority 2. • Selective Regression is carried out by a necessity and passed. Passed QA testing: No (P1/P2) bugs outstanding for STORY and neighbor functionality Story meets INVEST criteria Story is Signed-Off by Product Owner www.qaexperts.pro
  • 8. Definition of Done for Sprint, e.g. • Stories that meet DoD are considered DONE, the rest goes to next sprint. • It should be avoided to merge Risky or buggy code to main branch at the end of sprint! Stories meets DoD • Regression testing is passed with P1 test cases on at least 1 top priority platform/device • Smoke or exploratory testing is passed on platforms/devices of priority 1 and 2 Cover Regression Test • Main Integration tests for live 3d-party services are passed on STAGE or other environments3d party Integration Test No (P1/P2) bugs outstanding for product and neighbor products. Sprint is Signed-Off by Chief Product Owner www.qaexperts.pro
  • 9. Definition of Done for RELEASE, e.g. • All Stories, which were merged to Main branch, meets DoD for stories. • Risky or buggy code is not merged to main branch or is toggled off or is reverted! Stories meets DoD • Regression testing is passed for all priority 1 platforms/devices. • Exploratory testing is passed for priority 2 platforms/devices. • Smoke testing is passed for priority 3 platforms. Full Regression Test • All Integration tests for live 3d-party services are passed on STAGE or other environments • There is no other 3rd party release ongoing that may impact release negatively 3d party Integration Test and Releases • UAT is passed on STAGE environment UAT • Performance Testing successfully passed • Security Testing successfully passed • and other types of testing and preparations to release Other Types of Testing No (P1/P2) bugs outstanding for product and neighbor products Release GO Aproval from by Chief Product Owner, QA team, Release Manager and respective Business representatives All other unresolved bugs are tracked, reviewed, and planned when to be fixed www.qaexperts.pro
  • 10. Definition of Done dictates you which Activities and QA metrics to perform and monitor! www.qaexperts.pro
  • 12. Types of Metrics Quantative - Information that is directly measured as a number Qualitative - Information that results from human observation and analysis www.qaexperts.pro
  • 13. Types of Metrics 1. Absolute metrics 2. Relative metrics 3. Derivative metrics Purpose of Metrics. 1. Goal Metrics: To measure results/goals 2. Actionable metrics: to make decision 3. Predictive metrics: to predict future www.qaexperts.pro
  • 14. Stages at which QA and other metrics are needed Implementation period • Daily Monitoring of Quality, product implementation, and test progress Key sprint and release points • QA Report Before sprint Demo • QA Reports During stabilization / regression / UAT period • QA report Before Release • QA Reminder report when product has critical quality state or low product implementation or testing progress After Release • Release quality and process analysis • Quality and process Trends and analysis • Team efficiency analysis www.qaexperts.pro
  • 16. Goals of Daily Quality Monitoring 1. What is the current quality of the product? 2. How much work is left? 3. Do we have enough time to finish and stabilize the product? www.qaexperts.pro
  • 17. !!!Important: Breakdown Defect Metrics Always breakdown your metrics to the following groups: All Bugs! – shows general product quality state Critical and Blocker bugs – Shows whether we meet Readiness criteria! Lower Priority bugs– Shows the rest of bugs, general amount of which can negatively influence on the end-user’s impression about the product. www.qaexperts.pro
  • 18. Defects reported by Date Critical and Blocker bugs Major and lower priority bugs Jira “Recently Created Chart” Gadget Allows you track new bugs and immediately react to them! www.qaexperts.pro
  • 19. Open Defects reported Today Jira “Filter Results” Gadget Allows you to quickly review new bugs www.qaexperts.pro
  • 20. Open Defects By Component/Platform Major and lower priority bugs Jira “Pie Chart” Gadget Critical and Blocker bugs Shows which component/platform is not ready for release or requires bug fixes www.qaexperts.pro
  • 21. Defects By Component/Platform for Current Affected Version (any Status) Major and lower priority bugs Jira “Pie Chart” Gadget Critical and Blocker bugs Shows which component/platform require more thorough testing, e.g. (Selective) Regression testing www.qaexperts.pro
  • 22. Critical Open Bugs by Status Jira “Issue Statistics” Gadget Allows you to quickly get Status of Open Critical Bugs and understand whether we have enough time to fix them www.qaexperts.pro
  • 23. Open Bugs by Priority Jira “Issue Statistics” Gadget Shows general quality of the product www.qaexperts.pro
  • 24. !!!NOT-assigned Open Critical Defects Jira “Issue Statistics” Gadget To Fix Version To Team (for big teams only) To Sprint To Developer Shows probably lost or forgotten Critical open bugs, which should be processed or escalated immediately! www.qaexperts.pro
  • 25. www.qaexperts.pro Workload: Assigned Open Tasks and Bugs for Developers/QAs Jira “Issue Statistics” Gadget By teams By Developers For current Fix Version or sprint By QAs Stories By teams By Developers By QAs Bugs To assure that there is no bottlenecks and overloads. To Remind about work assigned! To spread load equally between teams, Developers and QAs!
  • 26. Release scope status by developers/QAs/teams If by the end of sprint or before stabilization period at release a lot of stories are in open status, it can bring risk of late merge status, not enough time for testing, not ability to finish functionality etc. Action: Try to spread load equally by teams, QAs and Dev Specialists. Remind about work Not Done. Postpone certain functionality to next Release! Stories Bugs Jira “Two Dimensional Filter Statistics” Gadget www.qaexperts.pro
  • 27. Stories/Bug Status By Fix Version created >= -365d Stories Bugs created >= -180d Allows to understand ability to finish/fix all tasks or bugs in time for a certain sprint/release Jira “Two Dimensional Filter Statistics” Gadget www.qaexperts.pro
  • 28. Current Fix Version Progress Shows general release/sprint ProgressJira “Release Report” www.qaexperts.pro
  • 30. Kanban board for All Open Critical and Blocker Bugs (For big teams) Allows to accumulate and track all open Critical and Blocker bugs in 1 place www.qaexperts.pro
  • 31. Regression testing Recommended Statuses: • Failed with Critical and Blocker bugs • Failed with Major and lower priority bugs • Failed by automated tests • Blocked • Not tested • In Progress • Passed www.qaexperts.pro
  • 32. Regression Testing Coverage and Results For Release/Milestone – General Results By Cycle/Stages - Every cycle should has certain purpose, e.g: 1. Initial Feedback about quality, 2. Exploratory testing, 3. Confirmation testing 4. 3rd Party services testing. www.qaexperts.pro
  • 33. Regression Testing Coverage and Results By Product Component • by platform/device • by test priority: • Priority 1 • Priority 2 • Priority 3 By Type of Test: • Smoke test • Regression test, • UAT test www.qaexperts.pro
  • 34. QA Metrics For Regular Quality Data Reports (Snapshots) www.qaexperts.pro
  • 35. Sprint Report 1. Story Delivery rate % = Actual Stories Completed vs. Committed Stories, e.g.: 90% 2. Amount of Stories met DoD, e.g.: 25 3. Amount of Stories Moved to a new sprint, e.g.: 4 Because of 1 not implemented, 1 Reopen because of Critical Bugs, 2 Not Tested Specifies the reasons, risks and recommendations. 4. Regression testing coverage and results: Graph 5. Cross platform testing coverage and results: Graph 6. 3d-party integration testing results: Graph or written explanation. 7. Testing and testing activities NOT done – describe reasons, connected risks, and recommendations. 8. Open Bugs: All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link to jira) Open Major bugs opened during the sprint, e.g: 40 Major bugs open during sprint. (link to jira) 9. Point to the buggiest product component and platform: Graph and explanation 10. General Conclusions www.qaexperts.pro
  • 36. Regression testing reports, daily or by cycles Reports about Regression testing, cross platform testing, 3d-party integration testing, 3d party releases testing, etc. 1. Name of the Regression testing stage, e.g.: Regression Cycle 1 (Preliminary feedback) 2. Coverage and results: Graph 3. Open Bugs New (created today) and All Open Critical and Blocker bugs: New 4 Critical&Blocker Bugs, (All 5 Critical&Blocker Bugs) (list or links to jira) New (Created today) and All Open Major bugs opened during the release: New 10 Major bugs, (All Major bugs created for current release) (links to jira) 4. Specify Time needed to finish regression testing and/or necessity in additional regression and other testing. 5. Point to the buggiest product component, platform, or functionality if needed. 6. Recommendations www.qaexperts.pro
  • 37. Release Readiness testing Report (by necessity) 1. Specify functional testing accomplished and its status: Regression and cross-platform testing, 3dparty integration testing, UAT e.g.: Written explanations and Links to Test management system, or Graphs of testing coverage and results. 2. Testing and testing activities NOT done – describe reasons, connected risks and recommendations. 3. Specify Non-functional testing status, like performance and security testing. Whether the product is ready or not ready from non-functional point of view. Describe connected risks and recommendations. 4. Open Bugs. All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link) Open Major bugs opened during the release and All Major bugs, e.g: 40 Major bugs open during release., 70 Major bugs are Open in general (list or links to jira) 5. Functionality Not ready for release, etc.: (list or link)+ written explanation 6. General Conclusions! www.qaexperts.pro
  • 38. Regression and Release Readiness Report example www.qaexperts.pro
  • 39. Reminder Reports (Be necessity) 1. Critical and Blocker Bugs in open state. 2. A Lot of Open Major bugs 3. A lot of Bugs and stories Not Done in the middle or later stage of sprints: In development, in Test, etc. 4. Availability of not tested stories during stabilization period. 5. Inability to finish testing in time because of certain issues. Reminder emails or chat messages about: www.qaexperts.pro
  • 40. Trend-based QA Metrics for for process and product quality analysis www.qaexperts.pro
  • 41. Milestones for trends • AffectedVersion – AffectedVersion field in jira • Time Span - e.g. period between start and end date of sprint Time based conditions in jira queries: a) createdDate >= '2017-08-30' and createdDate <='2017-09-12 ‘ b) status changed TO 'READY TO TEST' after '2017-08-30' before '2017-09-12‘ c) assignee was in (‘user’) • Sprint – Sprint Field in jira, etc. www.qaexperts.pro
  • 42. Found on Environment Critical and Blocker bugs Shows leaks of Critical and Blocker bugs to later development stages Pivot Report www.qaexperts.pro
  • 43. Found on Environment All bugs Shows leaks of bugs to later development stages Pivot Report www.qaexperts.pro
  • 44. Bugs By Priority – Any Status All Bugs by Aff. Version Shows absolute release quality history – decrease in number of all and critical bugs shows process improvements Pivot Report Note: Should be considered carefully when the team size and product increment increase. Note: Blank version is bad, and it shows that the Affected Version is not specified for certain bugs and thus they can be lost www.qaexperts.pro
  • 45. Bugs By Priority – OPEN (not-closed) All Bugs by Aff. Version Shows still open bugs for releases – Ideally No critical and blocker bugs should be here Pivot Report Note: If critical bugs are shown here for previous releases, they might be lost, forgotten or blocked. www.qaexperts.pro
  • 46. Bugs by RootCause Typical Root causes of Bugs: FrontendOthers Backend Frontend + Backend Environme nt issues 3d party components www.qaexperts.pro
  • 47. Bugs by RootCause Critical and Blocker bugs Shows Root cause of all bugs Pivot Report All Bugs Shows Root cause of the most of important bugs www.qaexperts.pro
  • 48. Created Vs Resolved - Cumulative Critical and Blocker Bugs Created should be less than resolved Jira “Created vs. Resolved Chart” Gadget Bug found vs. fixed chart is a defect analysis line chart that lets us see defect removal process patterns and understand defect management effectiveness www.qaexperts.pro
  • 49. Created Vs Resolved - Cumulative All Bugs Difference should be minimal and If it is big and no positive trend is shown then probably Bagathon sprint should be planned Jira “Created vs. Resolved Chart” Gadget www.qaexperts.pro
  • 50. Defects Dept Critical and Blocker Bugs Note: Should Be always close to Zero for Critical bugs All Bugs Spreadsheets + advanced Jira time period based queries Defects dept is a cumulative difference between Resolved and Open bugs = (Open in sprint N - Resolved in sprint N)+ (Open in sprint N+1- Resolved in sprint N+1 )+… Note: Shows accumulation of bugs in time www.qaexperts.pro
  • 51. Bugs Opened vs Stories Delivered There should be correlation between the metrics - more stories you deliver more bugs may appear. Ideal trend is to deliver more user stories with less bugs. !!!Pay attention to anomalies and investigate them deeper. Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
  • 52. Bugs per Story Point Critical and Blocker Bugs If number decreases in time it means that we improve QA process Note: This metric is very good for cases when team size or product increment is variable or change (increase) All Bugs Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
  • 53. Bugs per Story Point Other Variations: 1. Bugs per requirement 2. Bugs per developer 3. Bugs per 1000/Code 4. Time lost for bugs h - Time spent for Bugs vs time spent for stories Note: Very useful to for making future project estimations. www.qaexperts.pro
  • 54. Fix Rate Should be close to 100% or higher Spreadsheets + Advanced Jira period based queries Critical and Blocker Bugs All Bugs Fix Rate = Closed bugs / Open bugs Ideally should be close to 100% www.qaexperts.pro
  • 55. “Average Age” and “Time in status” For Critical and Blocker Bugs Jira “Average Age” and “Time in Status” Gadgets Average Age - Displays the average number of days issues have been unresolved. For critical and Blocker bugs this time should be very short. If process improves it should decrease in time. Average Time in status, e.g. Open status - Displays the average number of days issues have spent in status. For critical and Blocker bugs this time should be very short. If process improves it should decrease in time. www.qaexperts.pro
  • 56. Cumulative Flow Chart Jira Agile Reports To find process bottlenecks and predict project close time www.qaexperts.pro
  • 57. Bugs by Components/Platforms Pivot or other spreadsheet Report Bugs trend By Component – Critical and Blocker / All Bugs Shows weakest component of the product, shows where additional testing should be performed, shows where technical Improvements should be made, Shows whether component quality improves Bugs trend By Platform – Critical and Blocker / All Bugs. Shows amount of cross-platform bugs, shows the weakest platform requiring more testing and stabilization. . www.qaexperts.pro
  • 58. Sprint and Business Metrics Estimates Accuracy % = Estimated amount of effort/Actual amount of effort-1 Story Delivery Rate % = Actual Stories Completed / Committed Stories % of Change Requests = Number of change requests / Number of Initial Backlog stories Change Requests Costs = Time spent of change requests * Average Rate of developers on the project % Costs of Defect = Average Time for defect fix * Average developer rate on the project  Trends  For Sprint  General value www.qaexperts.pro
  • 59. User Satisfaction Number of hot fixes had to be made in this or that release. Shows leaks of Critical bugs on prod. Bugs found by end-users Critical/Blocker (Shows leaks of Critical bugs on prod.) Major and Low priority Customer Complaints and Feedbacks reports. (Net Promoter Score) A/B Testing results Shows which functionality is more preferred Other Marketing investigations and surveys All these answer the following questions: a) What are the weakest points of the product? b) What functionality is most used ? c) Which focus in product development and testing should be made? www.qaexperts.pro
  • 60. Qualitative Metrics about the project state If you want to gather more metrics about project processes, approaches, technology, communication, mood in teams, etc. Just ask people to provide evaluation and feedback Experts evaluation and/or Audit Surveys Retrospectives 1-2-1 and other meeting Rumors/small talks at Smoking room/kitchen  www.qaexperts.pro
  • 62. Developers and Dev Team efficiency REOPENED: Reopened Bugs Ratio % = Reopened bugs vs Resolved bugs Reopened Bugs Ratio % = Reopened Stories vs Stories delivered to testing STORY POINTS DELIVERED STORY DEFECTS ISSUED STORY DEFECTs/STORY POINTs rate % QUALITATIVE METRICS: Gather qualitative metrics about developers: approach to coding, complexity of code, contribution to innovation, communication efficiency, quick reaction to problems, etc.  Per PROJECT in general  Per Team  Per specific Developer www.qaexperts.pro
  • 63. QAs and QA team Efficiency REJECTED Absolute Number of rejected bugs: All resolutions, Not a bug, Duplicate Cannot reproduce Incomplete Rejected Rate % = rejected vs created BUGs LEAKS: to PROD other later stages of development. Break down by Per product part Per functionality So you can correlate it with a certain QA or group of QAs Separately:  For critical and blocker  For Lower priority defects  Per PROJECT in general  Per QA Team  Per specific QA www.qaexperts.pro
  • 64. QAs and QA team Efficiency WORK DONE:  Stories Tested  Created bugs Critical and Blocker (not rejected)  Created bugs Major and lower priority (not Rejected)  Test cases Created (passed DoD). Test cases rejected.  Test cases Executed.  Automation scripts Created (passed DoD). Test cases rejected.  Advanced test artifacts created and activities accomplished: test suites, plans, release support, etc. QUALITATIVE METRICS: Gather qualitative metrics about QAs: Complexity of tasks, Communication efficiency, Proactivity, Contribution to innovation, Test management skils, Conflict solving, etc.  Per PROJECT in general  Per QA team  Per specific QA  For Sprint (Velocity)  Trends  General value www.qaexperts.pro
  • 66.  Unit test coverage & number of test  API testing coverage & number of test for product in general by component  GUI Automation testing coverage & number of test  for product in general  by product part  by platform/device  by type of test end-to-end test Smoke test Regression test UAT test  by test priority: Priority 1 Priority 2 Priority 3 Automation Coverage & Results www.qaexperts.pro
  • 67. Automation Efficiency AUTOMATION SPEED:  Automation tests Frequency  Duration of tests Running  Velocity of script creation and maintenance  Time for script creation and maintenance AUTOMATION EFFICIENCY: Stability of automation False Positives Root Cause for failure Bugs found by Automation Critical and Blocker Other bugs Bugs missed with automation. Critical and Blocker Other bugs ROI – Time automation economy, etc. Cost per automated test  Per PROJECT in general  Per QA team  Per specific QA  For Sprint (Velocity)  Trends  General value www.qaexperts.pro
  • 69. Tools  Jira + Special time- related JQL  Confluence + Jira and other Macros in it  Spreadsheet processors: Googlesheets, Excel  TestTrail www.qaexperts.pro
  • 70. JiraReportingInGsheets Tool Link to test version: https://chrome.google.com/webstore/detail/jirareportingingsheets/jmjoohlbcigmigokjldgcgnfofcagock JiraReportingInGSheets add-on for GoogleSheets allows to automatically build custom jira reports in google sheets Created by QAExperts.pro www.qaexperts.pro
  • 71. Thank you Darya Alymova and Sergey Sosnitskiy for their contribution to certain metrics definition and implementation www.qaexperts.pro
  • 72. QA Experts.Pro Workshops 2 days workshop November 2017 http://qaexperts.pro/#signup www.qaexperts.pro

Editor's Notes

  1. What is quality of the product and development process. How to measure it. Which metrics assures product quality and which ones are crucial for making release readiness conclusions. How quality trends help us to understand that we are improving product and development process quality. Absolute vs relative metrics. Tools.