SlideShare a Scribd company logo
1 of 41
Project quality and test
process metrics
Zbigniew Moćkun, Tomasz Rękawek
© 2011 Cognifide Limited. In commercial confidence only.
Who are we?

© 2011 Cognifide Limited. In commercial confidence only.
Agenda
• Introduction to metrics
• Code quality metrics
− Metrics
− Sonar as example

• Project quality and testing process metrics
− Metrics
− Example quality report

© 2011 Cognifide Limited. In commercial confidence only.
Metrics
What is it about?

© 2011 Cognifide Limited. In commercial confidence only.
Why do we use metrics?
•
•
•
•
•
•

Project overview
Control a process
Control risks
Force the use of good practices
Project audits
Projects / methodologies / ... comparison
Lorem ipsum dolor sit
amet, consectetur adipisicing
elit, sed do eiusmod tempor
incididunt ut labore

© 2011 Cognifide Limited. In commercial confidence only.
I realized that metrics...

Encourage me to
ask questions

© 2011 Cognifide Limited. In commercial confidence only.
Be careful with metrics!!

Processes
Lorem ipsum dolor sit
amet, consectetur adipisicing
elit, sed do eiusmod tempor
incididunt ut labore

People

© 2011 Cognifide Limited. In commercial confidence only.
Type of measurements
• Trend
• KPI (Key Performance Indicator)
− Lower threshold
− Higher threshold
− Value

• How to set correct KPI?
Lorem ipsum dolor sit amet,
consectetur adipisicing elit,
sed do eiusmod tempor
incididunt ut labore

© 2011 Cognifide Limited. In commercial confidence only.
Not only about numbers

Lorem ipsum dolor sit
amet, consectetur adipisicing
elit, sed do eiusmod tempor
incididunt ut labore

© 2011 Cognifide Limited. In commercial confidence only.
Qualitative vs. Quantitative
- case study

Lorem ipsum dolor sit
amet, consectetur adipisicing
elit, sed do eiusmod tempor
incididunt ut labore

© 2011 Cognifide Limited. In commercial confidence only.
Level of details
•
•
•
•

You cann’t measure everything
Choose the most important metrics
Monitor and ask questions
Looking for details when you found a problem

Lorem ipsum dolor sit amet,
consectetur adipisicing elit,
sed do eiusmod tempor
incididunt ut labore

© 2011 Cognifide Limited. In commercial confidence only.
Metrics gathering
•
•
•
•

Analysis of live data (on request)
Analisis based on daily values (can be run as cron job)
Data vary in time manner
Froze data for analysis

© 2011 Cognifide Limited. In commercial confidence only.
Visualize metrics
•
•
•
•

Use charts if possible
Use colors as status (green, amber, red)
Add description
Remember about recommendation (your feeling)
− Subjective assessment is important too
Lorem ipsum dolor sit
amet, consectetur adipisicing
elit, sed do eiusmod tempor
incididunt ut labore

© 2011 Cognifide Limited. In commercial confidence only.
Project Quality metrics
Quality is everywhere

© 2011 Cognifide Limited. In commercial confidence only.
Code quality metrics
Code quaility metrics are covered by Tomek Rękawek
presentation which can be found here:
http://www.slideshare.net/TomaszRkawek/code-metrics

© 2011 Cognifide Limited. In commercial confidence only.
QA vs. code quality metrics
• Part of assurance
• Define KPI with Technical Lead

• Understand what you measure and why
• Discuss trends, exception with TL
• Motivate developers to keep high coding standards

© 2011 Cognifide Limited. In commercial confidence only.
Project quality metrics

© 2011 Cognifide Limited. In commercial confidence only.
Why do we test?

Software testing is an
investigation conducted to provide
stakeholders with information
about the quality of the product or
service under test

© 2011 Cognifide Limited. In commercial confidence only.
Testing = measuring

Software testing is an
investigation conducted to provide
stakeholders with information
about the quality of the product or
service under test
Source: wikipedia

© 2011 Cognifide Limited. In commercial confidence only.
Dev – QA cooperation
© 2011 Cognifide Limited. In commercial confidence only.
Metric by actions

© 2011 Cognifide Limited. In commercial confidence only.
One measurement,
two metrics
• Acceptance rate / Internal Quality
− Acceptance actions/all qa actions
− All QA actions = accept or reject
− Measure quality of software send to QA

• QA velocity
− Sum off all actions
− Divide by number of QA Engineers assigned to project
− QA velocity depend on software quality that comes to qa
− Why testing takes so long?

© 2011 Cognifide Limited. In commercial confidence only.
Test Case Execution metrics
• Planned test cases vs. written test case (Preparation, UAT)
• Number / percentage of executed test cases
(Acceptance, regression, UAT)
• Number of passed/failed test cases
(Acceptance, regression, UAT)
• Test Case metrics mean nothing

© 2011 Cognifide Limited. In commercial confidence only.
Requirement coverage
• Requirement coverage
− How many requirements have test cases?

− How many requirements were tested?
− How many requirements passed?
− Can we measure coverage if we use exploratory approach?

• Requirement traceability
− Defect – test case - requirement

© 2011 Cognifide Limited. In commercial confidence only.
Bugs metrics – S curve
• S Curve
• Open against Resolved

• Wages
− Blocker: 10
− Critical: 8

− Major: 6

• No Major or higher rule
− 5 Trivials = 1 Minor
− 5 Minors = 1 Major , ...

© 2011 Cognifide Limited. In commercial confidence only.
External quality

© 2011 Cognifide Limited. In commercial confidence only.
The importance of metrics
• Example A
−
−
−
−
−
−
−

Story comes to QA
QA found 1 major issue and reject (1 day)
Dev fix it (1 day)
QA test it again, 1 major issue found and reject(1 day)
Dev fix it (1 day)
QA accept (1 day)
Sum: 5 days, 2, rejects, 2 issues only

• Example B
−
−
−
−
−

Story comes to QA
QA found 1 critical, 1 major and 3 minor issues and reject story (1 day)
Dev fix it (2 day)
QA test it again and accept (1 day)
Sum: 4 days, 1 reject, 5 issues

• Acceptance rate against raised issues metric

© 2011 Cognifide Limited. In commercial confidence only.
Bug Taxonomy

© 2011 Cognifide Limited. In commercial confidence only.
Bug Taxonomy
• Phases
− Project: Discovery, Development, UAT, Live

• Test Case lifecycle
− Requirements/Documentation, Test Scenario, Test Execution

• Risk areas

• Application modules/components
• Functional/non-functional
• Product specific areas
− CMS: author, publish, frontend, backend

© 2011 Cognifide Limited. In commercial confidence only.
Defect Density - components

© 2011 Cognifide Limited. In commercial confidence only.
Defect Density - Acceptance vs.
Regression (automation vs. manual)

© 2011 Cognifide Limited. In commercial confidence only.
Defect Density
Application specific

© 2011 Cognifide Limited. In commercial confidence only.
Defect Density – won’t fix
Do we test in right way?

© 2011 Cognifide Limited. In commercial confidence only.
Automation - coverage
• Defects found (manual vs. automated)
• Coverage
− Manual vs. automated test cases
− High priority
− Application specific
 Author vs. publish

© 2011 Cognifide Limited. In commercial confidence only.
Other metrics
• Performance monitoring trend as example
• Client side grades (webpagetest.org as example)

• QA queue
• Agile metrics
− Lead Time (Open to Resolved time)

− Idle Time (time spend in queues)

© 2011 Cognifide Limited. In commercial confidence only.
Qualitative metrics
• Client feeling
− Surveys

− Demo
− Just try to talk

• Tester / team subjective opinion
− Recommendation
− Feeling
− Usability

© 2011 Cognifide Limited. In commercial confidence only.
Agile vs. Waterfall
• Quality metrics are independent against:
− Methodology

− Approach

• Two main test approach can be used in both
− Exploratory vs. scripting

• Agile put more emphasis on qualitative metrics

© 2011 Cognifide Limited. In commercial confidence only.
Example report

© 2011 Cognifide Limited. In commercial confidence only.
References
• http://www.kaner.com/pdfs/BugTaxonomies.pdf

• ISTQB Advance Test Manager Syllabus
− http://www.istqb.org/downloads/viewcategory/46.html

• Michael Bolton blog post
− http://www.developsense.com/blog/2009/01/meaningful-metrics/

• Douglas Hoffman
− http://www.softwarequalitymethods.com/Papers/DarkMets%20Pape
r.pdf

© 2011 Cognifide Limited. In commercial confidence only.
Poznań Testing and Quality Group

• Local group

• Next meeting: 28th of November
− Bogdan Bereza-Jarociński: Quality in Agile – quality at all
− Jakub Bryl: How to organize security testing

© 2011 Cognifide Limited. In commercial confidence only.
Q&A

© 2011 Cognifide Limited. In commercial confidence only.

More Related Content

What's hot

Project Quality Management
 Project Quality Management Project Quality Management
Project Quality Management
Serdar Temiz
 
Key performance indicators 2010
Key performance indicators 2010Key performance indicators 2010
Key performance indicators 2010
Hj Arriffin Mansor
 
Jurnal an example of using key performance indicators for software development
Jurnal   an example of using key performance indicators for software developmentJurnal   an example of using key performance indicators for software development
Jurnal an example of using key performance indicators for software development
Universitas Putera Batam
 
software testing metrics do's - don'ts-XBOSoft-QAI Webinar
software testing metrics do's - don'ts-XBOSoft-QAI Webinarsoftware testing metrics do's - don'ts-XBOSoft-QAI Webinar
software testing metrics do's - don'ts-XBOSoft-QAI Webinar
XBOSoft
 

What's hot (20)

Lean Agile Metrics And KPIs
Lean Agile Metrics And KPIsLean Agile Metrics And KPIs
Lean Agile Metrics And KPIs
 
Project Quality Management
 Project Quality Management Project Quality Management
Project Quality Management
 
Key performance indicators 2010
Key performance indicators 2010Key performance indicators 2010
Key performance indicators 2010
 
Project Quality Planning and KickOff
Project Quality Planning and KickOffProject Quality Planning and KickOff
Project Quality Planning and KickOff
 
Project Metrics & Measures
Project Metrics & MeasuresProject Metrics & Measures
Project Metrics & Measures
 
Project Quality Management Plan Checklist PowerPoint Presentation Slides
Project Quality Management Plan Checklist PowerPoint Presentation SlidesProject Quality Management Plan Checklist PowerPoint Presentation Slides
Project Quality Management Plan Checklist PowerPoint Presentation Slides
 
The Metrics of Project Management Performance and PMBOK
The Metrics of Project Management Performance and PMBOKThe Metrics of Project Management Performance and PMBOK
The Metrics of Project Management Performance and PMBOK
 
Pmp quality chapter 8
Pmp quality chapter 8Pmp quality chapter 8
Pmp quality chapter 8
 
Measure Customer and Business Feedback to Drive Improvement
Measure Customer and Business Feedback to Drive ImprovementMeasure Customer and Business Feedback to Drive Improvement
Measure Customer and Business Feedback to Drive Improvement
 
Testing Centre Of Excellence From AppLabs
Testing Centre Of Excellence From AppLabsTesting Centre Of Excellence From AppLabs
Testing Centre Of Excellence From AppLabs
 
Jurnal an example of using key performance indicators for software development
Jurnal   an example of using key performance indicators for software developmentJurnal   an example of using key performance indicators for software development
Jurnal an example of using key performance indicators for software development
 
PMP Key-exam-concepts-QA-QC-VS-VS
PMP Key-exam-concepts-QA-QC-VS-VSPMP Key-exam-concepts-QA-QC-VS-VS
PMP Key-exam-concepts-QA-QC-VS-VS
 
Ch13 apqp
Ch13 apqpCh13 apqp
Ch13 apqp
 
Six sigma green belt project roadmap in 10 deliverables 2013 07
Six sigma green belt project roadmap in 10 deliverables 2013 07Six sigma green belt project roadmap in 10 deliverables 2013 07
Six sigma green belt project roadmap in 10 deliverables 2013 07
 
Controlling Project Performance by Using a Defect Model - SEPG NA 2008 - Ben ...
Controlling Project Performance by Using a Defect Model - SEPG NA 2008 - Ben ...Controlling Project Performance by Using a Defect Model - SEPG NA 2008 - Ben ...
Controlling Project Performance by Using a Defect Model - SEPG NA 2008 - Ben ...
 
Integrated project management and solution delivery process 10082017 dt
Integrated project management and solution delivery process 10082017 dtIntegrated project management and solution delivery process 10082017 dt
Integrated project management and solution delivery process 10082017 dt
 
10. Project Quality Management
10. Project Quality Management 10. Project Quality Management
10. Project Quality Management
 
Testing as a Managed Service using SLAs and KPIs
Testing as a Managed Service using SLAs and KPIsTesting as a Managed Service using SLAs and KPIs
Testing as a Managed Service using SLAs and KPIs
 
Essam lotffy this project is not in line with organizational strategic plans
Essam lotffy   this project is not in line with organizational strategic plansEssam lotffy   this project is not in line with organizational strategic plans
Essam lotffy this project is not in line with organizational strategic plans
 
software testing metrics do's - don'ts-XBOSoft-QAI Webinar
software testing metrics do's - don'ts-XBOSoft-QAI Webinarsoftware testing metrics do's - don'ts-XBOSoft-QAI Webinar
software testing metrics do's - don'ts-XBOSoft-QAI Webinar
 

Viewers also liked

Zen QA Presentation
Zen QA PresentationZen QA Presentation
Zen QA Presentation
baapaa
 
Overview of Multidimensional Quality Metrics (QTLaunchPad)
Overview of Multidimensional Quality Metrics (QTLaunchPad)Overview of Multidimensional Quality Metrics (QTLaunchPad)
Overview of Multidimensional Quality Metrics (QTLaunchPad)
Arle Lommel
 
Agile Scrum in 60 minutes
Agile Scrum in 60 minutesAgile Scrum in 60 minutes
Agile Scrum in 60 minutes
Syed Arh
 
Presentation -Quality Metrics For Agile Development
Presentation -Quality Metrics For Agile DevelopmentPresentation -Quality Metrics For Agile Development
Presentation -Quality Metrics For Agile Development
Nabilahmed Patel
 
Quality metrics project management
Quality metrics project managementQuality metrics project management
Quality metrics project management
selinasimpson1501
 

Viewers also liked (20)

Code metrics
Code metricsCode metrics
Code metrics
 
Test Metrics and KPIs
Test Metrics and KPIsTest Metrics and KPIs
Test Metrics and KPIs
 
Zen QA Presentation
Zen QA PresentationZen QA Presentation
Zen QA Presentation
 
SlingQuery
SlingQuerySlingQuery
SlingQuery
 
Knowledge sharing for QA teams
Knowledge sharing for QA teamsKnowledge sharing for QA teams
Knowledge sharing for QA teams
 
Overview of Multidimensional Quality Metrics (QTLaunchPad)
Overview of Multidimensional Quality Metrics (QTLaunchPad)Overview of Multidimensional Quality Metrics (QTLaunchPad)
Overview of Multidimensional Quality Metrics (QTLaunchPad)
 
Defect Density matrix SQA
Defect Density  matrix   SQADefect Density  matrix   SQA
Defect Density matrix SQA
 
Wind Power in Maine
Wind Power in MaineWind Power in Maine
Wind Power in Maine
 
CRX2Oak - all the secrets of repository migration
CRX2Oak - all the secrets of repository migrationCRX2Oak - all the secrets of repository migration
CRX2Oak - all the secrets of repository migration
 
Agile Scrum in 60 minutes
Agile Scrum in 60 minutesAgile Scrum in 60 minutes
Agile Scrum in 60 minutes
 
Agile metrics - Measure and Improve
Agile metrics - Measure and ImproveAgile metrics - Measure and Improve
Agile metrics - Measure and Improve
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
 
GBlackard USCPPD512B - KPI Development
GBlackard USCPPD512B - KPI DevelopmentGBlackard USCPPD512B - KPI Development
GBlackard USCPPD512B - KPI Development
 
Key Performance Indicators, a spy in the camp or a useful tool?
Key Performance Indicators, a spy in the camp or a useful tool?Key Performance Indicators, a spy in the camp or a useful tool?
Key Performance Indicators, a spy in the camp or a useful tool?
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
 
Top 10 Agile Metrics
Top 10 Agile MetricsTop 10 Agile Metrics
Top 10 Agile Metrics
 
Presentation -Quality Metrics For Agile Development
Presentation -Quality Metrics For Agile DevelopmentPresentation -Quality Metrics For Agile Development
Presentation -Quality Metrics For Agile Development
 
Agile Base Camp - Agile metrics
Agile Base Camp - Agile metricsAgile Base Camp - Agile metrics
Agile Base Camp - Agile metrics
 
Quality metrics project management
Quality metrics project managementQuality metrics project management
Quality metrics project management
 
5 of 5 Quality Metrics & Science by Obaid Ali
5 of 5  Quality Metrics & Science by Obaid Ali5 of 5  Quality Metrics & Science by Obaid Ali
5 of 5 Quality Metrics & Science by Obaid Ali
 

Similar to Project quality (and test process) metrics

Sdec10 lean package implementation
Sdec10 lean package implementationSdec10 lean package implementation
Sdec10 lean package implementation
Terry Bunio
 
Sivasubramani_N_Testing
Sivasubramani_N_TestingSivasubramani_N_Testing
Sivasubramani_N_Testing
nsivamsit
 
Resume bernadette yousif-sr. qa
Resume   bernadette yousif-sr. qaResume   bernadette yousif-sr. qa
Resume bernadette yousif-sr. qa
Bernadette Yousif
 

Similar to Project quality (and test process) metrics (20)

Resume .Thangam
Resume .ThangamResume .Thangam
Resume .Thangam
 
Quality Assurance vs. Quality Control, Future of Software Quality
Quality Assurance vs. Quality Control, Future of Software Quality Quality Assurance vs. Quality Control, Future of Software Quality
Quality Assurance vs. Quality Control, Future of Software Quality
 
Between Scrum and Kanban - define a test process for Agile methodologies
Between Scrum and Kanban - define a test process for Agile methodologiesBetween Scrum and Kanban - define a test process for Agile methodologies
Between Scrum and Kanban - define a test process for Agile methodologies
 
Between Scrum and Kanban - define test process for Agile methodologies
Between Scrum and Kanban - define test process for Agile methodologiesBetween Scrum and Kanban - define test process for Agile methodologies
Between Scrum and Kanban - define test process for Agile methodologies
 
Automation as a project - lessons learnt
Automation as a project - lessons learntAutomation as a project - lessons learnt
Automation as a project - lessons learnt
 
Resume
ResumeResume
Resume
 
Nitin Chourasia
Nitin ChourasiaNitin Chourasia
Nitin Chourasia
 
Sdec10 lean package implementation
Sdec10 lean package implementationSdec10 lean package implementation
Sdec10 lean package implementation
 
Independent verification & validation presented by Maneat v02
Independent verification & validation presented by Maneat v02Independent verification & validation presented by Maneat v02
Independent verification & validation presented by Maneat v02
 
Process Guidelines V2
Process Guidelines V2Process Guidelines V2
Process Guidelines V2
 
Quality Index: A Composite Metric for the Voice of Testing
Quality Index: A Composite Metric for the Voice of TestingQuality Index: A Composite Metric for the Voice of Testing
Quality Index: A Composite Metric for the Voice of Testing
 
Requirements and Acceptance Tests: Yes, They Go Together
Requirements and Acceptance Tests: Yes, They Go TogetherRequirements and Acceptance Tests: Yes, They Go Together
Requirements and Acceptance Tests: Yes, They Go Together
 
Test Policy and Practices
Test Policy and PracticesTest Policy and Practices
Test Policy and Practices
 
Agile Metrics - ASTQB Workshop by Philip Lew - XBOSoft
Agile Metrics - ASTQB Workshop by Philip Lew - XBOSoftAgile Metrics - ASTQB Workshop by Philip Lew - XBOSoft
Agile Metrics - ASTQB Workshop by Philip Lew - XBOSoft
 
Sivasubramani_N_Testing
Sivasubramani_N_TestingSivasubramani_N_Testing
Sivasubramani_N_Testing
 
Stevan Zovinovic - Testers Agile Pocketbook - EuroSTAR 2012
Stevan Zovinovic - Testers Agile Pocketbook - EuroSTAR 2012Stevan Zovinovic - Testers Agile Pocketbook - EuroSTAR 2012
Stevan Zovinovic - Testers Agile Pocketbook - EuroSTAR 2012
 
Methodology: IT test
Methodology: IT testMethodology: IT test
Methodology: IT test
 
Quality for DevOps teams - Quality engineering in the DevOps culture
Quality for DevOps teams - Quality engineering in the DevOps cultureQuality for DevOps teams - Quality engineering in the DevOps culture
Quality for DevOps teams - Quality engineering in the DevOps culture
 
Debalina_Datta_Resume
Debalina_Datta_ResumeDebalina_Datta_Resume
Debalina_Datta_Resume
 
Resume bernadette yousif-sr. qa
Resume   bernadette yousif-sr. qaResume   bernadette yousif-sr. qa
Resume bernadette yousif-sr. qa
 

More from Zbyszek Mockun

Selenium - The Way Of Success
Selenium - The Way Of SuccessSelenium - The Way Of Success
Selenium - The Way Of Success
Zbyszek Mockun
 

More from Zbyszek Mockun (20)

Freestyle your leadership
Freestyle your leadershipFreestyle your leadership
Freestyle your leadership
 
How to develop diverse team? v2.0
How to develop diverse team? v2.0How to develop diverse team? v2.0
How to develop diverse team? v2.0
 
How to develop diverse team?
How to develop diverse team?How to develop diverse team?
How to develop diverse team?
 
Automation in the world of project
Automation  in the world of projectAutomation  in the world of project
Automation in the world of project
 
The little thing
The little thingThe little thing
The little thing
 
Ptaq day one intro presentation
Ptaq day one intro presentationPtaq day one intro presentation
Ptaq day one intro presentation
 
We are all engineers
We are all engineers We are all engineers
We are all engineers
 
Automation in the world of projects - A few thoughts from business perspective
Automation in the world of projects - A few thoughts from business perspectiveAutomation in the world of projects - A few thoughts from business perspective
Automation in the world of projects - A few thoughts from business perspective
 
PTaQ in numbers after 50 meetups.
PTaQ in numbers after 50 meetups.PTaQ in numbers after 50 meetups.
PTaQ in numbers after 50 meetups.
 
Shift left as first transformation step into Quality Assurance
Shift left as first transformation step into Quality AssuranceShift left as first transformation step into Quality Assurance
Shift left as first transformation step into Quality Assurance
 
There and back again, Our journey with QA Reports and metrics
There and back again, Our journey with QA Reports and metricsThere and back again, Our journey with QA Reports and metrics
There and back again, Our journey with QA Reports and metrics
 
Exploratory testing in practice, short story how approach influenced on strategy
Exploratory testing in practice, short story how approach influenced on strategyExploratory testing in practice, short story how approach influenced on strategy
Exploratory testing in practice, short story how approach influenced on strategy
 
User groups - share your knowledge
User groups - share your knowledgeUser groups - share your knowledge
User groups - share your knowledge
 
Test Strategy
Test StrategyTest Strategy
Test Strategy
 
Automated Exploratory Tests
Automated Exploratory TestsAutomated Exploratory Tests
Automated Exploratory Tests
 
Automated Exploratory Tests
Automated Exploratory TestsAutomated Exploratory Tests
Automated Exploratory Tests
 
Selenium - The Way Of Success
Selenium - The Way Of SuccessSelenium - The Way Of Success
Selenium - The Way Of Success
 
Automation Responsibility
Automation ResponsibilityAutomation Responsibility
Automation Responsibility
 
Continuous Integration testing based on Selenium and Hudson
Continuous Integration testing based on Selenium and HudsonContinuous Integration testing based on Selenium and Hudson
Continuous Integration testing based on Selenium and Hudson
 
Agile testing
Agile testingAgile testing
Agile testing
 

Recently uploaded

Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
UK Journal
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Recently uploaded (20)

TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptxBT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 

Project quality (and test process) metrics

  • 1. Project quality and test process metrics Zbigniew Moćkun, Tomasz Rękawek © 2011 Cognifide Limited. In commercial confidence only.
  • 2. Who are we? © 2011 Cognifide Limited. In commercial confidence only.
  • 3. Agenda • Introduction to metrics • Code quality metrics − Metrics − Sonar as example • Project quality and testing process metrics − Metrics − Example quality report © 2011 Cognifide Limited. In commercial confidence only.
  • 4. Metrics What is it about? © 2011 Cognifide Limited. In commercial confidence only.
  • 5. Why do we use metrics? • • • • • • Project overview Control a process Control risks Force the use of good practices Project audits Projects / methodologies / ... comparison Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  • 6. I realized that metrics... Encourage me to ask questions © 2011 Cognifide Limited. In commercial confidence only.
  • 7. Be careful with metrics!! Processes Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore People © 2011 Cognifide Limited. In commercial confidence only.
  • 8. Type of measurements • Trend • KPI (Key Performance Indicator) − Lower threshold − Higher threshold − Value • How to set correct KPI? Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  • 9. Not only about numbers Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  • 10. Qualitative vs. Quantitative - case study Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  • 11. Level of details • • • • You cann’t measure everything Choose the most important metrics Monitor and ask questions Looking for details when you found a problem Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  • 12. Metrics gathering • • • • Analysis of live data (on request) Analisis based on daily values (can be run as cron job) Data vary in time manner Froze data for analysis © 2011 Cognifide Limited. In commercial confidence only.
  • 13. Visualize metrics • • • • Use charts if possible Use colors as status (green, amber, red) Add description Remember about recommendation (your feeling) − Subjective assessment is important too Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  • 14. Project Quality metrics Quality is everywhere © 2011 Cognifide Limited. In commercial confidence only.
  • 15. Code quality metrics Code quaility metrics are covered by Tomek Rękawek presentation which can be found here: http://www.slideshare.net/TomaszRkawek/code-metrics © 2011 Cognifide Limited. In commercial confidence only.
  • 16. QA vs. code quality metrics • Part of assurance • Define KPI with Technical Lead • Understand what you measure and why • Discuss trends, exception with TL • Motivate developers to keep high coding standards © 2011 Cognifide Limited. In commercial confidence only.
  • 17. Project quality metrics © 2011 Cognifide Limited. In commercial confidence only.
  • 18. Why do we test? Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test © 2011 Cognifide Limited. In commercial confidence only.
  • 19. Testing = measuring Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test Source: wikipedia © 2011 Cognifide Limited. In commercial confidence only.
  • 20. Dev – QA cooperation © 2011 Cognifide Limited. In commercial confidence only.
  • 21. Metric by actions © 2011 Cognifide Limited. In commercial confidence only.
  • 22. One measurement, two metrics • Acceptance rate / Internal Quality − Acceptance actions/all qa actions − All QA actions = accept or reject − Measure quality of software send to QA • QA velocity − Sum off all actions − Divide by number of QA Engineers assigned to project − QA velocity depend on software quality that comes to qa − Why testing takes so long? © 2011 Cognifide Limited. In commercial confidence only.
  • 23. Test Case Execution metrics • Planned test cases vs. written test case (Preparation, UAT) • Number / percentage of executed test cases (Acceptance, regression, UAT) • Number of passed/failed test cases (Acceptance, regression, UAT) • Test Case metrics mean nothing © 2011 Cognifide Limited. In commercial confidence only.
  • 24. Requirement coverage • Requirement coverage − How many requirements have test cases? − How many requirements were tested? − How many requirements passed? − Can we measure coverage if we use exploratory approach? • Requirement traceability − Defect – test case - requirement © 2011 Cognifide Limited. In commercial confidence only.
  • 25. Bugs metrics – S curve • S Curve • Open against Resolved • Wages − Blocker: 10 − Critical: 8 − Major: 6 • No Major or higher rule − 5 Trivials = 1 Minor − 5 Minors = 1 Major , ... © 2011 Cognifide Limited. In commercial confidence only.
  • 26. External quality © 2011 Cognifide Limited. In commercial confidence only.
  • 27. The importance of metrics • Example A − − − − − − − Story comes to QA QA found 1 major issue and reject (1 day) Dev fix it (1 day) QA test it again, 1 major issue found and reject(1 day) Dev fix it (1 day) QA accept (1 day) Sum: 5 days, 2, rejects, 2 issues only • Example B − − − − − Story comes to QA QA found 1 critical, 1 major and 3 minor issues and reject story (1 day) Dev fix it (2 day) QA test it again and accept (1 day) Sum: 4 days, 1 reject, 5 issues • Acceptance rate against raised issues metric © 2011 Cognifide Limited. In commercial confidence only.
  • 28. Bug Taxonomy © 2011 Cognifide Limited. In commercial confidence only.
  • 29. Bug Taxonomy • Phases − Project: Discovery, Development, UAT, Live • Test Case lifecycle − Requirements/Documentation, Test Scenario, Test Execution • Risk areas • Application modules/components • Functional/non-functional • Product specific areas − CMS: author, publish, frontend, backend © 2011 Cognifide Limited. In commercial confidence only.
  • 30. Defect Density - components © 2011 Cognifide Limited. In commercial confidence only.
  • 31. Defect Density - Acceptance vs. Regression (automation vs. manual) © 2011 Cognifide Limited. In commercial confidence only.
  • 32. Defect Density Application specific © 2011 Cognifide Limited. In commercial confidence only.
  • 33. Defect Density – won’t fix Do we test in right way? © 2011 Cognifide Limited. In commercial confidence only.
  • 34. Automation - coverage • Defects found (manual vs. automated) • Coverage − Manual vs. automated test cases − High priority − Application specific  Author vs. publish © 2011 Cognifide Limited. In commercial confidence only.
  • 35. Other metrics • Performance monitoring trend as example • Client side grades (webpagetest.org as example) • QA queue • Agile metrics − Lead Time (Open to Resolved time) − Idle Time (time spend in queues) © 2011 Cognifide Limited. In commercial confidence only.
  • 36. Qualitative metrics • Client feeling − Surveys − Demo − Just try to talk • Tester / team subjective opinion − Recommendation − Feeling − Usability © 2011 Cognifide Limited. In commercial confidence only.
  • 37. Agile vs. Waterfall • Quality metrics are independent against: − Methodology − Approach • Two main test approach can be used in both − Exploratory vs. scripting • Agile put more emphasis on qualitative metrics © 2011 Cognifide Limited. In commercial confidence only.
  • 38. Example report © 2011 Cognifide Limited. In commercial confidence only.
  • 39. References • http://www.kaner.com/pdfs/BugTaxonomies.pdf • ISTQB Advance Test Manager Syllabus − http://www.istqb.org/downloads/viewcategory/46.html • Michael Bolton blog post − http://www.developsense.com/blog/2009/01/meaningful-metrics/ • Douglas Hoffman − http://www.softwarequalitymethods.com/Papers/DarkMets%20Pape r.pdf © 2011 Cognifide Limited. In commercial confidence only.
  • 40. Poznań Testing and Quality Group • Local group • Next meeting: 28th of November − Bogdan Bereza-Jarociński: Quality in Agile – quality at all − Jakub Bryl: How to organize security testing © 2011 Cognifide Limited. In commercial confidence only.
  • 41. Q&A © 2011 Cognifide Limited. In commercial confidence only.

Editor's Notes

  1. Ask about six sigma. Marta asked me. Which company developed six sigma.
  2. Who use metrics so far?
  3. Analyze
  4. People are smart. They can start working to improve metrics
  5. Trends – short story how we set KPI. Do not set it at the begining. Obeserve trends, analysis it. Talk about it, ask which release/ sprint went fine, we are happy, which not and make a decision based on it. Do not set it orbitrary. Of course at the begining you can set base KPI but you have to be carefull. KPI can change as project phase/ requirement / .. Changes.
  6. QuantitativeQualitativeAgile – stawia na qualitative. Demo ask for client feedback after it, talk with your client, clolse cooperation with client, delivery what client need, rather than what client wrote in requirement/specificationStory baout JLP
  7. WE were like a Gilbert. Some timie ago there wer quite big project for very important client. Agile We were quite happy about project quality, everything was smooth, progressive UAT (yes, clent test features after each sprint and accept). But once there was a message from client. That our code quality is very low. When we start asking why, we didn’t get clear answer. But after more detailed question we realize that issue is somewhere else. Migration/content team has an issue with using delivered and it cased delay, client development team weren’t able o take responsibility for maintenence, ... It cause that client management team had feeling that we deliver low quality code. We have verify our metrics and they were fine. Almost on the same level throught during the project. External company did code/architecture review and UAT tests. All went well and we showed that problem were somwher else on client side. But we learnt that we have to carefull on more on qualitative metrics / client feelings and only demo and short feedback after whcih almost claimt that’s fine is not enough
  8. Measure basic things, that shows you were is something wrong. Than looking for details. So do not use metrics for detailed information / do not expect it. Problems can be different, so you metrics to found that you have a problem and then try to identify what caused it.
  9. JIRA and other tools change values for issues. So if you debug after few days you can see totally different values. SO, most of defect can be fixed, ...It great to froze data for debuging
  10. WE have a lot of question about metrics – are they ok? Descriptions are fine, but if you need to go throught several report and read details it takes too much time. Visualisation allow you to understand it better
  11. So we as QA monitor values that are agreed with TL for example or just good practice. WE can monitor, ask, require to add task to new sprint, ... or just raise that good practice are not keep in this project
  12. Questions to students – why does we test?
  13. Testing doesn’t improve, it measure, when we fix defect, we improve quality, but there is a chance to introduce new ones
  14. How testing process looks in Cognifide – as it’s the most important metrics for us are connected with our specific process. And remember metrics are against process not
  15. Metric
  16. Screen z metryka – szybkosc qa zalezy od jakosci jaka dostajemhy. Jest to tak naprade suma wszelkich testow jak i retestow. Czyli jesli cos dostaniemy zlego, odrzucimy to kilka razy, to nasza efektywnosc jest roznie mierzonaManager – przetestowaliscie tylko jedno storyQA przetestowalismy 4 story (za jazdtm razem to samo), ale za kazdym razem dostawalismy je z bledem
  17. Number passed test cases says nothing. We always add one more metric from bug tracker. I have never been in project where there were full mapping between test cases and bugs. Always thre were some bugs reported agains application which are not cover by any test case. There is one more rule in QA world – that number of test cases means nothing. So I would rather be very carefull with metrics against test cases – unfrotunatelly most of companies/test leads believ in test case world – I am not.
  18. Screen z metryka – szybkosc qa zalezy od jakosci jaka dostajemhy. Jest to tak naprade suma wszelkich testow jak i retestow. Czyli jesli cos dostaniemy zlego, odrzucimy to kilka razy, to nasza efektywnosc jest roznie mierzonaManager – przetestowaliscie tylko jedno storyQA przetestowalismy 4 story (za jazdtm razem to samo), ale za kazdym razem dostawalismy je z bledem
  19. Screen z metryka – szybkosc qa zalezy od jakosci jaka dostajemhy. Jest to tak naprade suma wszelkich testow jak i retestow. Czyli jesli cos dostaniemy zlego, odrzucimy to kilka razy, to nasza efektywnosc jest roznie mierzonaManager – przetestowaliscie tylko jedno storyQA przetestowalismy 4 story (za jazdtm razem to samo), ale za kazdym razem dostawalismy je z bledem
  20. Screen z metryka – szybkosc qa zalezy od jakosci jaka dostajemhy. Jest to tak naprade suma wszelkich testow jak i retestow. Czyli jesli cos dostaniemy zlego, odrzucimy to kilka razy, to nasza efektywnosc jest roznie mierzonaManager – przetestowaliscie tylko jedno storyQA przetestowalismy 4 story (za jazdtm razem to samo), ale za kazdym razem dostawalismy je z bledem
  21. What is it?
  22. Screen z metryka – szybkosc qa zalezy od jakosci jaka dostajemhy. Jest to tak naprade suma wszelkich testow jak i retestow. Czyli jesli cos dostaniemy zlego, odrzucimy to kilka razy, to nasza efektywnosc jest roznie mierzonaManager – przetestowaliscie tylko jedno storyQA przetestowalismy 4 story (za jazdtm razem to samo), ale za kazdym razem dostawalismy je z bledem
  23. Screen z metryka – szybkosc qa zalezy od jakosci jaka dostajemhy. Jest to tak naprade suma wszelkich testow jak i retestow. Czyli jesli cos dostaniemy zlego, odrzucimy to kilka razy, to nasza efektywnosc jest roznie mierzonaManager – przetestowaliscie tylko jedno storyQA przetestowalismy 4 story (za jazdtm razem to samo), ale za kazdym razem dostawalismy je z bledem
  24. Please look at automation results. We have quite good coverage in this project – about 20%, but automated tests found not too many. Do you know why?Pesticide effect – you still need to work an update your automation. But it doesn’t mean that automation do not work. It minimalize our test effort. Automated tests can be run on dev env too, and if found issue they are not reported – so please be carefull. The same during acceptance tests, we do not raise all defect. AS mostly dev-tester are paired and if issue is found and can be fixed immidiatelly we do not raise as it takes more time . Why we do not care – if effect not delay development – we can leave with them.
  25. Screen z metryka – szybkosc qa zalezy od jakosci jaka dostajemhy. Jest to tak naprade suma wszelkich testow jak i retestow. Czyli jesli cos dostaniemy zlego, odrzucimy to kilka razy, to nasza efektywnosc jest roznie mierzonaManager – przetestowaliscie tylko jedno storyQA przetestowalismy 4 story (za jazdtm razem to samo), ale za kazdym razem dostawalismy je z bledem
  26. Quite important metric. Do we test write thing. Do we raise right defect? It’s not only about tester time, but client who prioritize it, developers who are trying to fix/reproduce, .... Do we undersand what is the most important for client and businessThe same for external compoany – UAT. WE shows that they raising a lot of issues but not correct ones. We lost time only for debugging not adding any value to project
  27. Quite important metric. Do we test write thing. Do we raise right defect? It’s not only about tester time, but client who prioritize it, developers who are trying to fix/reproduce, .... Do we undersand what is the most important for client and business
  28. Queues are very important metrics in agile env – especially in Kanabn methodology.
  29. Quality merics are universal. Quality doesn’t ask about methodology but measure what you deliver. The same wioth testing approach. There are very small difference sometimes in measurement like requirement covereage but still tit’s the same metric.
  30. Screen z metryka – szybkosc qa zalezy od jakosci jaka dostajemhy. Jest to tak naprade suma wszelkich testow jak i retestow. Czyli jesli cos dostaniemy zlego, odrzucimy to kilka razy, to nasza efektywnosc jest roznie mierzonaManager – przetestowaliscie tylko jedno storyQA przetestowalismy 4 story (za jazdtm razem to samo), ale za kazdym razem dostawalismy je z bledem