The document discusses test metrics that can be used in agile software development to support frequent changes. It proposes measuring test coverage, defects found during testing versus after release, issues reported by customers, time spent by users during testing, and regression test suite duration. These key performance indicators (KPIs) provide visibility into test results and quality. The document outlines how to configure tools like Jira to calculate and visualize these KPIs to help make data-driven decisions.
Quality Engineering Approaches by Yotam Sharan
A lecture given by Yotam on October 8, 2013 at eBay Israel Development Center, as part of eBay's QA Conference.
“ЕРАМ у Південному регіоні та можливості розвитку для QA спеціалістів” Online...QADay
Online QADay 2021
“ЕРАМ у Південному регіоні та можливості розвитку для QA спеціалістів”
telegram: wwww.t.me/goqameetup
fb: www.fb.com/goqaevent
fb: www.fb.com/qaday.org
linkedin: https://www.linkedin.com/company/goqa/
Сайт: www.qaday.org
Achieving CI Excellence with Quality EngineeringGreg Sypolt
Quality Engineering roles continue to evolve and will be entirely different in the future. At Gannett | USA Today Network, the change has started by blurring the lines between Test Automation and DevOps daily tasks with Quality Engineering owning continuous integration (CI), defining CI best practices, building the CI pipeline, and being the quality gatekeeper of product releases.
- Setting expectations for CI-
- CI ownership as a community activity, not an individual one
- Defining a continuous testing strategy
- Designing repeatable and disposable CI architecture
- Setting CI standards
- Quality Engineering roles and responsibilities
Quality Engineering Approaches by Yotam Sharan
A lecture given by Yotam on October 8, 2013 at eBay Israel Development Center, as part of eBay's QA Conference.
“ЕРАМ у Південному регіоні та можливості розвитку для QA спеціалістів” Online...QADay
Online QADay 2021
“ЕРАМ у Південному регіоні та можливості розвитку для QA спеціалістів”
telegram: wwww.t.me/goqameetup
fb: www.fb.com/goqaevent
fb: www.fb.com/qaday.org
linkedin: https://www.linkedin.com/company/goqa/
Сайт: www.qaday.org
Achieving CI Excellence with Quality EngineeringGreg Sypolt
Quality Engineering roles continue to evolve and will be entirely different in the future. At Gannett | USA Today Network, the change has started by blurring the lines between Test Automation and DevOps daily tasks with Quality Engineering owning continuous integration (CI), defining CI best practices, building the CI pipeline, and being the quality gatekeeper of product releases.
- Setting expectations for CI-
- CI ownership as a community activity, not an individual one
- Defining a continuous testing strategy
- Designing repeatable and disposable CI architecture
- Setting CI standards
- Quality Engineering roles and responsibilities
- Understand the principles behind the agile approach to software development
- Differentiate between the testing role in agile projects compared with the role of testers in non-agile projects
- Positively contribute as an agile team member focused on testing
- Appreciate the challenges and difficulties associated with the non-testing activities performed in an agile team
- Demonstrate a range of soft skills required by agile team members
Do you have to learn code to create test automation? The answer is no. The latest technology enables business and QA teams to ensure enterprise application quality using easy to maintain, plain English tests. Powerful automated business process discovery, risk based analysis and lights-out regression tests can then be used to extend automation and create additional value – all without code!
This is a free module from my course ISTQB CTFL Agile Tester revised to 2014 syllabus. If you need full training feel free to contact me by email (amraldo@hotmail.com) or by mobile (+201223600207).
Guarda il webinar su Youtube! http://youtu.be/_GlXZAuQrJc
A CHI E' RIVOLTO:
Quality Assurance Managers, Test Managers, Testers, Senior Developers, Project Managers and Compliance Managers
DESCRIZIONE:
La tecnologia avanza rapidamente, impattando quasi tutti gli aspetti delle pratiche di test: il test agile, il test basato sui requisiti, test as a service e il testing crowdsource.
In questo webinar verrà illustrato un processo in 5 step, in grado di migliorare i processi di test, riducendo i costi e i tempi dei progetti.
Questo webinar metterà in evidenza i limiti e le problematiche dello sviluppo software e le metodologie per un suo immediato miglioramento.
AGENDA:
- Creare e gestire facilmente e velocemente test cases, defects, enhancement requests e test specifications - con Polarion LiveDocs
- Definire ed eseguire test runs con the Polarion Testing Framework
- Risparmiare tempo e denaro riutilizzando tutti i testing artifacts
- Integrare nei propri processi i test cases importandoli da MS Office
- Integrare completamente la gestione dei requisiti ai test e al defect management
- Gestire con Polarion le attività con tutti gli stakeholder.
Zero touch QA automation platform for DevOpsTaUB Solutions
Presentation based on the Award Winning Paper by Varadarajan Srinivasan on Zero Touch QA Automation Platform at STC Nov 2018.
Companies that are in DevOps Journey go through a rough weather in QA Automation initiatives. With the industry transformation to Digital Trend, there is always a need for an efficient streamlined Continuous Delivery.
This Webinar would talk about Zero Touch QA Automation Platform. It deals with the Solution on Integrated DevOps & Quality Approach. This would consist of Artificial Intelligence in Reporting and Data Visualization. The Complete QA Cycle is automated from Test Design to Test Closure with Continuous Monitoring, involving no manual intervention. The Webinar would also give insights on Business benefits/outcome on adopting this approach with streamlined Governance.
Continuous Testing for CTOs (Webinar Slides)Rainforest QA
Watch on-demand: http://info.rainforestqa.com/webinar-continuous-testing-for-ctos
Slides from the Continuous Testing for CTOs webinar with Laks Srini (CTO, Zenefits) and Russell Smith (CTO, Rainforest QA).
QA testing is often a bottleneck to true continuous deployment, and traditional processes are ill-suited for CI/CD environments. As your company grows, the cost of scaling traditional QA practices to meet the needs of a larger audience with more complex products can overextend bandwidth. With continuous testing, you can keep your growing organization's QA costs down and empower your team to test more effectively.
- How Zenefits uses continuous testing to speed up their QA cycle to match their CD environment
- How continuous testing minimizes the cost of scaling your QA team
- How to plug continuous testing into your existing Agile or CD workflow for minimal disruption
- How to speed up manual QA activities to ensure fast but comprehensive testing
Test Automation Strategies and Frameworks: What Should Your Team Do?TechWell
Agile practices have done a magnificent job of speeding up the software development process. Unfortunately, simply applying agile practices to testing isn't enough to keep testers at the same pace. Test automation is necessary to support agile delivery. Max Saperstone explores popular test automation frameworks and shares the benefits of applying these frameworks, their implementation strategies, and best usage practices. Focusing on the pros and cons of each framework, Max discusses data-driven, keyword-driven, and action-driven approaches. Find out which framework and automation strategy are most beneficial for specific situations. Although this presentation is tool agnostic, Max demonstrates automation with examples from current tooling options. If you are new to test automation or trying to optimize your current automation strategy, this session is for you.
We have some great new enhancements that were added to qTest on Mach 28th 2016 that will help make your testing process much more efficient. We are also introducing a beta version of our new Atlassian HipChat add-on that will help your team improve collaboration while using qTest.
View the slides for the On-Demand webinar that aired on April 6th, to learn about all the new qTest features:
- Copy and paste across projects
- Data query enhancements
- Filtering tree structure
- New! Hip-Chat Add-On
View the On-Demand Webinar here: http://pi.qasymphony.com/qtest-7.4-release-webinar-lp056
It can be confusing for everyone in an agile team to understand when or what to test, when there isn't a test phase or any formal documented requirements. Whatever your agile methodology, projects require a change in the way QA and development work together. The use of technology and automation are much more difficult and finding a practical approach to testing is critical for successful agile projects.
In the Webinar presentation, George Wilson explored how testing in agile is different and revealed pragmatic advice to ensure that application quality, within an agile environment, isn't compromised. Listen to the archive Webinar discussion on the techniques for quickly getting control of manual testing and progressing to automated testing in agile, which will leave you with fresh thinking to resolve or prevent any testing dysfunctions in your agile teams.
- See more at: http://www.origsoft.com/webinars/agile_testing/
'Quality Engineering: Build It Right The First Time' by Allan Woodcock, Shoba...TEST Huddle
It's no secret, that Quality Assurance (QA) plays a very vital role in assuring customer experience. Then, why wait till the end of the development phase to validate the business requirements?
Also, the cost of defect multiplies in the later stages of the development lifecycle. Quality Engineering (QE) is an innovative approach which focusses on upstream assurance of software through early involvement of QA in the lifecycle.
Techniques like architecture validation, design validation, quality profiling and static testing are defining the QE landscape. Industry leading tools and point solutions also enable effective and efficient QE approach for quality assurance teams.
As the IT focus moves to engineering quality with a lower Cost of Quality (CoQ), QE will play a vital role in achieving the end business objectives.
Key Takeaways
- Business drivers requiring the paradigm shift of Building it Right the First Time
- Tools, techniques and methodologies for embracing Quality Engineering
- Predictive test design through analytics powered Quality Intelligence
- Importance of QE in the sphere of Agile, Continuous Integration and DevOps environment
Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness in meeting some performance goal. Various goals are possible; one involves rate of fault detection | a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test, and let software engineers begin correcting faults earlier than might otherwise be possible.
Agile testing: from Quality Assurance to Quality AssistanceLuca Giovenzana
The history of the CHILI Agile transition from Quality Assurance to Quality Assistance.
Presented during a meetup:
https://www.youtube.com/watch?v=ZewEdNgo_oI&t=19s
and at the Working Software 2021:
https://www.agilemovement.it/workingsoftware/index.html
Velocity is one of the most common metrics used—and one of the most commonly misused—on agile projects. Velocity is simply a measurement of speed in a given direction—the rate at which a team is delivering toward a product release. As with a vehicle en route to a particular destination, increasing the speed may appear to ensure a timely arrival. However, that assumption is dangerous because it ignores the risks with higher speeds. And while it’s easy to increase a vehicle’s speed, where exactly is the accelerator on a software team? This presentation covers Hawthorne Effect and Goodhart’s Law to explain why setting goals for velocity can actually hurt a project's chances. Take a look at what can negatively impact velocity, ways to stabilize fluctuating velocity, and methods to improve velocity without the risks.
- Understand the principles behind the agile approach to software development
- Differentiate between the testing role in agile projects compared with the role of testers in non-agile projects
- Positively contribute as an agile team member focused on testing
- Appreciate the challenges and difficulties associated with the non-testing activities performed in an agile team
- Demonstrate a range of soft skills required by agile team members
Do you have to learn code to create test automation? The answer is no. The latest technology enables business and QA teams to ensure enterprise application quality using easy to maintain, plain English tests. Powerful automated business process discovery, risk based analysis and lights-out regression tests can then be used to extend automation and create additional value – all without code!
This is a free module from my course ISTQB CTFL Agile Tester revised to 2014 syllabus. If you need full training feel free to contact me by email (amraldo@hotmail.com) or by mobile (+201223600207).
Guarda il webinar su Youtube! http://youtu.be/_GlXZAuQrJc
A CHI E' RIVOLTO:
Quality Assurance Managers, Test Managers, Testers, Senior Developers, Project Managers and Compliance Managers
DESCRIZIONE:
La tecnologia avanza rapidamente, impattando quasi tutti gli aspetti delle pratiche di test: il test agile, il test basato sui requisiti, test as a service e il testing crowdsource.
In questo webinar verrà illustrato un processo in 5 step, in grado di migliorare i processi di test, riducendo i costi e i tempi dei progetti.
Questo webinar metterà in evidenza i limiti e le problematiche dello sviluppo software e le metodologie per un suo immediato miglioramento.
AGENDA:
- Creare e gestire facilmente e velocemente test cases, defects, enhancement requests e test specifications - con Polarion LiveDocs
- Definire ed eseguire test runs con the Polarion Testing Framework
- Risparmiare tempo e denaro riutilizzando tutti i testing artifacts
- Integrare nei propri processi i test cases importandoli da MS Office
- Integrare completamente la gestione dei requisiti ai test e al defect management
- Gestire con Polarion le attività con tutti gli stakeholder.
Zero touch QA automation platform for DevOpsTaUB Solutions
Presentation based on the Award Winning Paper by Varadarajan Srinivasan on Zero Touch QA Automation Platform at STC Nov 2018.
Companies that are in DevOps Journey go through a rough weather in QA Automation initiatives. With the industry transformation to Digital Trend, there is always a need for an efficient streamlined Continuous Delivery.
This Webinar would talk about Zero Touch QA Automation Platform. It deals with the Solution on Integrated DevOps & Quality Approach. This would consist of Artificial Intelligence in Reporting and Data Visualization. The Complete QA Cycle is automated from Test Design to Test Closure with Continuous Monitoring, involving no manual intervention. The Webinar would also give insights on Business benefits/outcome on adopting this approach with streamlined Governance.
Continuous Testing for CTOs (Webinar Slides)Rainforest QA
Watch on-demand: http://info.rainforestqa.com/webinar-continuous-testing-for-ctos
Slides from the Continuous Testing for CTOs webinar with Laks Srini (CTO, Zenefits) and Russell Smith (CTO, Rainforest QA).
QA testing is often a bottleneck to true continuous deployment, and traditional processes are ill-suited for CI/CD environments. As your company grows, the cost of scaling traditional QA practices to meet the needs of a larger audience with more complex products can overextend bandwidth. With continuous testing, you can keep your growing organization's QA costs down and empower your team to test more effectively.
- How Zenefits uses continuous testing to speed up their QA cycle to match their CD environment
- How continuous testing minimizes the cost of scaling your QA team
- How to plug continuous testing into your existing Agile or CD workflow for minimal disruption
- How to speed up manual QA activities to ensure fast but comprehensive testing
Test Automation Strategies and Frameworks: What Should Your Team Do?TechWell
Agile practices have done a magnificent job of speeding up the software development process. Unfortunately, simply applying agile practices to testing isn't enough to keep testers at the same pace. Test automation is necessary to support agile delivery. Max Saperstone explores popular test automation frameworks and shares the benefits of applying these frameworks, their implementation strategies, and best usage practices. Focusing on the pros and cons of each framework, Max discusses data-driven, keyword-driven, and action-driven approaches. Find out which framework and automation strategy are most beneficial for specific situations. Although this presentation is tool agnostic, Max demonstrates automation with examples from current tooling options. If you are new to test automation or trying to optimize your current automation strategy, this session is for you.
We have some great new enhancements that were added to qTest on Mach 28th 2016 that will help make your testing process much more efficient. We are also introducing a beta version of our new Atlassian HipChat add-on that will help your team improve collaboration while using qTest.
View the slides for the On-Demand webinar that aired on April 6th, to learn about all the new qTest features:
- Copy and paste across projects
- Data query enhancements
- Filtering tree structure
- New! Hip-Chat Add-On
View the On-Demand Webinar here: http://pi.qasymphony.com/qtest-7.4-release-webinar-lp056
It can be confusing for everyone in an agile team to understand when or what to test, when there isn't a test phase or any formal documented requirements. Whatever your agile methodology, projects require a change in the way QA and development work together. The use of technology and automation are much more difficult and finding a practical approach to testing is critical for successful agile projects.
In the Webinar presentation, George Wilson explored how testing in agile is different and revealed pragmatic advice to ensure that application quality, within an agile environment, isn't compromised. Listen to the archive Webinar discussion on the techniques for quickly getting control of manual testing and progressing to automated testing in agile, which will leave you with fresh thinking to resolve or prevent any testing dysfunctions in your agile teams.
- See more at: http://www.origsoft.com/webinars/agile_testing/
'Quality Engineering: Build It Right The First Time' by Allan Woodcock, Shoba...TEST Huddle
It's no secret, that Quality Assurance (QA) plays a very vital role in assuring customer experience. Then, why wait till the end of the development phase to validate the business requirements?
Also, the cost of defect multiplies in the later stages of the development lifecycle. Quality Engineering (QE) is an innovative approach which focusses on upstream assurance of software through early involvement of QA in the lifecycle.
Techniques like architecture validation, design validation, quality profiling and static testing are defining the QE landscape. Industry leading tools and point solutions also enable effective and efficient QE approach for quality assurance teams.
As the IT focus moves to engineering quality with a lower Cost of Quality (CoQ), QE will play a vital role in achieving the end business objectives.
Key Takeaways
- Business drivers requiring the paradigm shift of Building it Right the First Time
- Tools, techniques and methodologies for embracing Quality Engineering
- Predictive test design through analytics powered Quality Intelligence
- Importance of QE in the sphere of Agile, Continuous Integration and DevOps environment
Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness in meeting some performance goal. Various goals are possible; one involves rate of fault detection | a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test, and let software engineers begin correcting faults earlier than might otherwise be possible.
Agile testing: from Quality Assurance to Quality AssistanceLuca Giovenzana
The history of the CHILI Agile transition from Quality Assurance to Quality Assistance.
Presented during a meetup:
https://www.youtube.com/watch?v=ZewEdNgo_oI&t=19s
and at the Working Software 2021:
https://www.agilemovement.it/workingsoftware/index.html
Velocity is one of the most common metrics used—and one of the most commonly misused—on agile projects. Velocity is simply a measurement of speed in a given direction—the rate at which a team is delivering toward a product release. As with a vehicle en route to a particular destination, increasing the speed may appear to ensure a timely arrival. However, that assumption is dangerous because it ignores the risks with higher speeds. And while it’s easy to increase a vehicle’s speed, where exactly is the accelerator on a software team? This presentation covers Hawthorne Effect and Goodhart’s Law to explain why setting goals for velocity can actually hurt a project's chances. Take a look at what can negatively impact velocity, ways to stabilize fluctuating velocity, and methods to improve velocity without the risks.
Team Foundation Server - Tracking & ReportingSteve Lange
Comprehensive presentation detailing reporting and tracking capabilities of Team Foundation Server. Focuses on Excel workbooks and Reporting Services, but touches on other technologies as well.
Test Metrics in Agile: A Powerful Tool to Demonstrate ValueTechWell
Most understand that an agile development and testing approach improves quality and reduces risks in our projects. In some companies and culture however, there are skeptics. Is the move to agile—and therefore agile testing—really beneficial? Join Iuliia Zavertailo for a closer look at a Scandinavian insurance company that started with one manual tester and within three years moved toward opening a large test center in the Baltic. Behind this story were many small steps of demonstrating testing's value to the client through a well-defined set of agile metrics which quantitatively supported the importance and value of testing. Iullia gives examples of key performance indicators—test coverage, defect open and close rates, issues reported by customers, and regression test suite duration—and provides a roadmap for building a test metrics framework. She then discusses tools that support the agile test framework, provides guidance on how to analyze test statistics, and offers ways to present the facts that interest clients most.
Use Automation to Assist—Not Replace—Manual TestingTechWell
Automation is a powerful tool to help testing but too often it is used to replicate existing manual tests. This leads organizations to spend large amounts of time and money constantly updating flaky automated tests and test teams to suffer frustration from having to focus on activities that are not truly testing. This cost and frustration can be avoided by using automation as a tool to assist testing—not to replace tests. Jeffrey Martin shares some real-world examples of using automation to supplement testing by leveraging its true value—the replication and repetition of tasks instead of tests. Examples are drawn from several testing teams, as well as his own. Jeffrey explores what kinds of tasks are the best fit for automation, identifies which tasks are better left to testers, and provides examples of melding task automation and manual tests together. Jeffrey discusses how organizations have introduced these concepts to maximize adoption and team buy-in. Leave with a different view of automation and ideas on how to best use this powerful tool to supplement actual tests seamlessly in your own team.
For a company like Blue Apron that is radically transforming the way we buy, prepare and eat meals, experimentation is mission critical for delivering a great customer experience. Blue Apron doesn’t just think about experimenting to improve short term conversion, they focus on ways to impact longer term metrics like retention, referrals, and lifetime value.
Join John Cline, engineering manager at Blue Apron, to learn how his team has built their experimentation program on Optimizely’s platform.
Attend this webinar to learn:
-How Blue Apron built their experimentation program on top of Optimizely Full Stack
-How developers play a critical role in experimentation
-The key considerations for developers when thinking about experimentation
This is a practical guide for sprint development based on the OutSystems Delivery Method.
It helps you focus some of the main challenges found when using Agile in the field:
- Your sprints often start not being ready?
- Delivering at sprint end is always struggle?
Then you should take a look!
Target audience: Agile Project Managers (including Engagement and Delivery Managers)
Use the Windshield, Not the Mirror Predictive Metrics that Drive Successful ...Seapine Software
Sharon Niemi, Practice Director of SQA, talks about how the right combination of predictive and reactive metrics can help you build a measurement portfolio that improves product quality and release consistency. You’ll learn how to build a measurement system that incorporates leading and lagging indicators to improve your team’s consistency in delivering quality products on time and within budget.
Continuous Performance Testing: The New StandardTechWell
In the past several years the software development lifecycle has changed significantly with high-speed software releases, shared application services, and platform virtualization. The traditional performance assurance approach of pre-release testing does not address these innovations. To maintain confidence in acceptable performance in production, pre-release testing must be augmented with in-production performance monitoring. Obbie Pet describes three types of monitors—performance, resource, and VM platform—and three critical metrics fundamental to isolating performance problems—response time, transaction rate, and error rate. Obbie reviews techniques to acquire and interpret these metrics, and describes how to develop a continuous performance monitoring process. In conjunction with pre-release testing, this monitoring can be woven into a single integrated process, offering a best bet in assuring performance in today’s development world. Take away this integrated process for consideration in your own shop.
Test Driven Development – What Works And What Doesn’t Synerzip
This webinar covers the role of TDD and CI and what pitfalls to watch out for.
Stay tuned for Synerzip's upcoming webinars that you might be interested in https://www.synerzip.com/webinars/
An Agile Testing Dashboard: Metrics that MatterTechWell
Has your organization moved to agile but your metrics have not? Do you spend multiple hours each week generating separate sets of metrics for your agile squad, management, and C-level executives? Do you find yourself questioning the things you measure and actions you take as a result of those measurements? Prachi Maini introduces you to a set of lean but comprehensive metrics that show the efficiency and effectiveness of the squads, the primary activities of the squad, and that of the overall project. Learn the key evaluators for measuring quality of code, quality of quality assurance (testing), project progress, automation ROI, and team sentiments. Take back an agile dashboard that can easily be generated to provide measurable data to both squad members and key executives, to help quantify risk and process improvements, and to allow for more effective and efficient project planning, forecasting, and estimating.
This is collection of question & answer in software testing interview job. Part 2 with 10 questions and answers.
This is designed by Khoa Bui, which owner of http://www.testing.com.vn site
Implementing a Test Dashboard to Boost QualityTechWell
You are responsible for addressing quality problems that are plaguing your product and having an adverse impact on the business. Have you been challenged to provide a simple mechanism for quantifying and tracking key performance indicators selected by your organization. The ultimate goal is an approach that will enable the cross-functional team to identify problem areas so they can take corrective action. Where do you start? Attend this session to learn how you can develop a quantifiable approach to assessing testing effectiveness and addressing quality. Scott Acker shows you a solution he developed, deployed, and managed to effectively leverage various types of data to support analyzing, tracking, and reporting changes in testing and quality over time. Discover how to drive communication and collaboration improvements across the entire cross-functional team and boost quality efforts.
Similar to Test Metrics in Agile - powerful tool to support changes - Zavertailo Iuliia (20)
Test Metrics in Agile - powerful tool to support changes - Zavertailo Iuliia
1. TEST METRICS IN AGILE
TEST MANAGEMENT
A POWERFUL TOOL TO SUPPORT CHANGES
Yulia Zavertailo, Senior Test manager
2. Agenda
− About me.
− Introduction to the client’s case.
− What needs to be improved?
− How do we visualize our results?
− A closer look at the KPIs and how to gather them.
− Conclusions.
03.04.2017 / 3
3. About me
− 10 years of work experience in the field of
Testing and Quality Assurance, 7.5 years in
Itera.
− Moved to Oslo in November 2014.
− Key competences:
− test management,
− test advisory on process establishment
in a project and entire organization.
− Very passionate about the ISO
9000/ISO29119 standards.
− Mentor and coach for junior testers.
− Love running, skiing and oil painting.
03.04.2017 / 4
4. IF Skadeforsikring – general
03.04.2017 / 5
− Waypoint is the largest digital solution at IF built on
modern technology platform with lots of interfaces
for clients and internal users (1000+).
− Methodology – Agile (Scrum, Kanban).
− 13 Agile teams.
− Each team has both business and IT competence.
− IT development is done in headquarters in Oslo.
− Large Scandinavian insurance company present in the
Nordic countries and Baltics with 3,6 millions of clients
and 6 800 employees.
5. – System is not covered by tests.
– No professional IT testing is done in the project.
– Business users are involved to do the functional acceptance and
regression testing.
– Long production cycle: releases are 3-4 times a year.
03.04.2017 / 6
IF Skadeforsikring - as a test challenge
− Critical issues found in production after release.
6. IF Skadeforsikring - in a result
− IT test team has grown from 1 to 17 IT testers in Riga, Latvia.
− Frequency of releases is every 5 weeks.
− Response on the entire system quality and stability within a few hours.
− Project discovers and fixes 136 bugs in average every release .
− There are still functional modules which do not have enough test
coverage = facing high risk of defects.
03.04.2017 / 7
7. Specifically…
what are we going to improve?
1. Increase frequency of releases so that IT delivers functionality to the
business every 5 weeks by:
– Automating the regression test coverage.
– Discovering and fixing bugs earlier in the cycle.
– Improving development teams efficiency by doing continuous IT testing.
03.04.2017 / 8
8. Specifically…
what are we going to improve?
2. Decrease amount of bugs in the production by:
– Running automated regression suits regularly.
– Improving the quality of testing itself.
– Introducing continuous manual testing while developing.
– Focus business resources doing acceptance testing only.
03.04.2017 / 9
9. Hmmm….
but how to prove the succeeded result?
Present test results our client is interested in:
– Test coverage.
– # of defects found in production.
– # of defects found in sprints.
– Time spent by end users.
– Time for feedback about system’s quality.
03.04.2017 / 10
10. KPI #1 – Test Coverage
What do we measure?
− Percentage of requirements covered by at least one test.
Why do we gather it in our Agile projects?
– Visualize “black spots” of our application.
– Visualize that the larger test coverage is, the less risk of missing
defects is.
– Visualize the functionality that needs more attention since it is less
covered by tests.
03.04.2017 / 11
11. Application test coverage - example
03.04.2017 / 12
1075
739
430
800
230
419
310
244
130
465
70
141
54
13 8
69
4 10
0
200
400
600
800
1000
1200
Module 1 Module 2 Module 3 Module 4 Module 5 Module 6
Total user stories US covered manual US covered automated
12. How to gather Test Coverage KPI
1. Configure your Agile TM tool in order to build a traceability matrix in
TM tool (TFS, Jira, QC).
2. Use the graphs module of your TM tool or export to Pivot tables in
Excel.
3. Go for the Excel-option if you do not have any TM tool.
03.04.2017 / 13
13. KPI #2 - Defect open and close rates
What Do we measure?
Defect Open and Close rate = Defects found during testing in a release *100
(Defects found during testing in a release + Defects found after release)
03.04.2017 / 14
Why do we gather it in our Agile projects?
− Categorizing defects as either open or closed.
− Certain defects could slip through the cracks and show up in the finalized
release.
− Testers and developers to work together to identify and address software
issues.
15. How to gather defect open and close rates KPI
To get a rate:
− Get the number of defects found during a release/sprint.
− Get the number of bugs reported by the business after release to
production.
− Calculate rate based on a formula.
To build a trend graph:
− Jira + Zephyr dashboards.
− TFS + pivot tables in Excel.
03.04.2017 / 16
16. KPI #3 - Issues reported by customers
What Do we measure?
% of customer reported issues = Total # of issues reported by customers *100
total number of issues reported
Why do we gather it in our Agile projects?
− Does the product meet the needs of a customer/end user?
− Effectiveness of test team.
− Types and number of defects lingering in released products.
− Critical issues with current QA processes?
03.04.2017 / 17
19. How to gather issues reported by customers
KPI
To get a rate:
− Get the number of defects found by customers.
− Get the total number of defects found in a sprint.
− Calculate the rate based on a formula.
To build a trend graph:
− TFS + pivot tables in Excel.
03.04.2017 / 20
20. KPI #4 – Efficiency of end users’ testing
03.04.2017 / 21
What do we measure?
− Time spent by end users.
− # of bugs found by end users during acceptance testing of release vs
total.
Why do we gather it in our Agile projects?
− Feedback about the quality of requirement/acceptance criteria.
− Feedback about the quality of the functional and regression testing.
− To see the cost (time spent vs amount of bugs found).
21. Time spent by end users – example of trend
03.04.2017 / 22
97
83
135.5
161.5
124 127 123
237
152.5
135.5
288.5
112
131.5
147.5
159
110
30 30 29 28.5 30 31 31 26
15.5 16.5 21.5
15 14.5 16 12 13.5
80 75
62
54
34
47 48
40 40
12 10 7.5 6.5 8 9 5 7 6 5 8 5 10 9.5 5 6.5
30 29
79
85
69 69 69
96
39
48
193
54 54 54
77
42
0
50
100
150
200
250
300
350
Total
Country Domain 1
Country Domain 2
Country Domain 3
Country Domain 4
Country Domain 5
22. KPI #5 - Regression test suite duration
What do we measure?
− Time for feedback about system’s stability and quality – TA.
− Time for feedback about system’s stability and quality – Manual.
Why do we gather it in our Agile projects?
− Get a clue about time needed to regression-test a hot fix or feature.
− Decision-making input for further planning.
− Visualized benefit and efficiency of test automation.
03.04.2017 / 23
23. Regression test suite duration - example
03.04.2017 / 24
Test suite
# of test cases in
a suite
Time to run test suite
with TA, hrs
Time to run suite
manually, hrs
Smoke test suite 6 0,2 1,5
Regression suite 1 191 5,5 47,75
Regression suite 2 204 5,5 51
Regression suite 3 194 5,5 48,5
Total 595 16,7 149
24. KPIs framework – in a few simple steps
1. Decide which KPIs are important for your client (focus on KPIs 1-3).
2. Configure your test management tool and visualize.
3. Continuously use KPIs at all levels of communication.
03.04.2017 / 25
25. Summary
− KPIs framework is a powerful tool to be an Agile test leader!
− Control the status of quality in a measurable way.
− Control the general healthiness of the test solution and strategy in your
project.
− Lead and make decisions in your project.
− Empower yourself to prove that IT- testing is a must in your project!
03.04.2017 / 26
Comments:
Waypoint system has core engine and core modules unified for everyone, though it s customized to the legal tax rules and insurance product specifics per country and domain market
Сomments:
System is not covered by tests -> test coverage is undefined
No professional IT testing is done in the project -> neither manual nor automated
Business users are involved to do the functional acceptance and regression testing –> up to 5-6 people in each country.
Long production cycle: releases are 3-4 times a year -> with the system test 3-4 weeks each time (long production cycle).
Comments:
Increase the amount of releases so that IT delivers functionality to the business every 5 weeks -> instead of 3-4 times a year
Automating the regression test coverage -> getting faster feedback about the system status and quality.
Improving development teams efficiency by doing continuous IT testing -> instead of business testing at the end of the release
Focus on presenting results interesting to the client:
Test coverage increased -> risk areas are covered by tests regularly
# of defects found in production -> reduced =less amount of critical bugs found in production
# of defects found in sprints -> increased and fixed before going to production.
Time spent by end users -> reduced
Сomments:
This is one of the most common metrics we can gather in our project! Metric number1.
Visualize “black spots” of our application -> shows that there are X functions/features in our application and that the risk of bugs in the system=% of system which is not covered by tests, and in which areas it may occur).
Comment to the example on the graph:
Total features for module 1 – 1075
Covered by tests – 364
% of module 1 test coverage = 33
Comments:
Configure your Agile TM tool in order to build a traceability matrix in TM tool (TFS, Jira, QC etc):
Make sure Epics/User stories ->linked to test cases (manual or TA)
Use a query based filter to show Epics/User stories ->linked to test cases.
Check how many Epics/User stories have test cases, how many have not.
Use the graphs module of your TM tool or export to Pivot tables in Excel to visualize the filtered data from tractability matrix:
Connect your Excel to TFS.
Use the filters you configured to display the traceability matrix.
Build the graph of required shape and colour.
This is the most common metric number2 which we can gather in our project.
Why do we gather it in our Agile projects?
While there are numerous defect statuses which indicate the current state of an identified flaw, they can typically be categorized as either open or closed.
If team members are not diligent about measuring the current status of their program’s flaws, certain defects could slip through the cracks and show up in the finalized release
comparing the frequency of open defects with close rates will also provide insight into the ability of testers and developers to work together to identify and address software issues
Comment to the example:
Release 1 has 196 bugs reported in total, where defects open and close rate is 86 %
Shows total number of defects (sprint +production)
Shows sprint trend
Shows production trend
Comments:
Issues found by customer after IT testing
We count on acceptance testing issues
Comments:
Graph shows total defects found in a sprint (grey bar) and issues found by customers (in red)
Comment:
Use defects tracking tool - either Jira or TFS, or HP QC
Comments:
Metric is relevant for Norway, where it s still lots of testing is done by end users without involvement of IT testers.
Comments:
This metric doesnt have to be updated every release, it s one-time analysis
Gives an overview of time to regression-test a hot fix or feature -> how long does it take the QA to actually test a feature and evaluate related risks.
Gives us a decision-making input for further planning -> how frequently releases can be made.
Visualized benefit and efficiency of test automation comparing to manual testing in out project.
Comments:
Decide which KPIs are important for your client -> Start from test coverage and defects open and close rates
Use Confluence for graphs visualization!
Continuously use KPIs at all levels of communication:
Talks with developers or your test team.
Support go-no go meetings.
Steering committee meetings with budget owners, decision makers.
Comments:
KPIs let u control the status of quality in a measurable way, with its numbers.
KPIs let u control the general healthiness of the test solution and strategy in your project giving you trends.
KPIs let u lead and make decisions in your project with facts.
KPIs empower you to prove that IT- testing is a must in your project!
KPIs empower you to be a good leader and help you to adjust your Agile ship’s course before it crashes.