The document discusses quality assurance (QA) metrics in agile development. It begins by defining quality for both products and processes, noting that QA influence increases as development moves from requirements to validation. It then covers the types of metrics that can be used as a foundation for measuring product quality, including quantitative, qualitative, absolute, relative, and derivative metrics. Finally, it provides examples of QA metrics that can be used for daily monitoring of quality, as well as metrics that can be included in regular quality reports for sprints and releases.
Real case studies of QA management in big teams (60-100 people). How to setup robust QA processes and approaches in them. Main impediments and problems, how to solve them. SAFe.
Agile Testing – embedding testing into agile software development lifecycle Kari Kakkonen
My presentation on Agile Testing, including a tuning concept and a case study of agile testing choices in a project, held 16 of June, 2014 at a customer internal seminar.
A test automation framework defines an organization's way of doing things. It is a set of assumptions, concepts and tools that provide support for automated software testing.
Building a Test Automation Strategy for SuccessLee Barnes
Choosing an appropriate tool and building the right framework are typically thought of as the main challenges in implementing successful test automation. However, long term success requires that other key questions must be answered including:
- What are our objectives?
- How should we be organized?
- Will our processes need to change?
- Will our test environment support test automation?
- What skills will we need?
- How and when should we implement?
In this workshop, Lee will discuss how to assess your test automation readiness and build a strategy for long term success. You will interactively walk through the assessment process and build a test automation strategy based on input from the group. Attend this workshop and you will take away a blue print and best practices for building an effective test automation strategy in your organization.
• Understand the key aspects of a successful test automation function
• Learn how to assess your test automation readiness
• Develop a test automation strategy specific to your organization
Real case studies of QA management in big teams (60-100 people). How to setup robust QA processes and approaches in them. Main impediments and problems, how to solve them. SAFe.
Agile Testing – embedding testing into agile software development lifecycle Kari Kakkonen
My presentation on Agile Testing, including a tuning concept and a case study of agile testing choices in a project, held 16 of June, 2014 at a customer internal seminar.
A test automation framework defines an organization's way of doing things. It is a set of assumptions, concepts and tools that provide support for automated software testing.
Building a Test Automation Strategy for SuccessLee Barnes
Choosing an appropriate tool and building the right framework are typically thought of as the main challenges in implementing successful test automation. However, long term success requires that other key questions must be answered including:
- What are our objectives?
- How should we be organized?
- Will our processes need to change?
- Will our test environment support test automation?
- What skills will we need?
- How and when should we implement?
In this workshop, Lee will discuss how to assess your test automation readiness and build a strategy for long term success. You will interactively walk through the assessment process and build a test automation strategy based on input from the group. Attend this workshop and you will take away a blue print and best practices for building an effective test automation strategy in your organization.
• Understand the key aspects of a successful test automation function
• Learn how to assess your test automation readiness
• Develop a test automation strategy specific to your organization
HCMC Software Testing Club - The 1st Meetup
By Thao Vo
Selecting a most suitable automated testing tool is one of big challenges in software test automation. Choosing a test tool is as complicated as getting married to a person. If you marry with an inappropriate person, you tend to break up sooner or later. Similarly, without a suitable test tool, we will deadly end up with failed test automation effort. There are a variety of automated testing tools with different testing types and technologies. How to define a set of criteria of requirements to meet our goal, and making a right tool will help us prevent later problems from the executing of successful testing project. This topic is intended to share steps and criteria to select an appropriate automated testing tool.
In this session, we would discuss what "Agile Testing" is, what are the well known methods and models of Agile Testing and what to expect on the future of Agile Testing.
Agile Testing Framework - The Art of Automated TestingDimitri Ponomareff
Once your organization has successfully implemented Agile methodologies, there are two major areas that will require improvements: Continuous Integration and Automated Testing.
This presentation illustrates why it's important to invest in an Automated Testing Framework (ATF) to reduce technical debt, increase quality and accelerate time to market.
Learn more at www.agiletestingframework.com.
Slides from a session presented by Fadi Stephan from Kaizenko at the 2019 Global Scrum Gathering in Austin, TX on 05/20/2019 DC. Also see the blog series on Agile Testing at https://www.kaizenko.com/agile-testing/
Abstract:
Many teams struggle with fitting in testing activities inside of a Sprint. They end up doing primarily development activities in a Sprint and push testing activities to run in dedicated testing Sprints following the coding Sprints or have a coding and testing Sprint running in parallel. However, in Scrum, the output of every Sprint is a potentially shippable product increment. This means the product increment should be well tested within the Sprint and ready to be delivered. Come to this presentation to learn how to tackle testing on an Agile team, what kind of tests to execute, what to automate and what not to automate, the different test responsibilities, and when to run which tests. Leave with a testing strategy that you can start applying the next day to gradually get a team to start testing from day 1 of the Sprint and deliver a true product increment at the end of each Sprint.
Software testing concepts is going to enhance from manual testing to automated testing. Due to rapidly increasing complexity in software development process, the present manual testing technique is not sufficient for effective testing. Hence, here arises the requirement of test automation process. The automation testing is the software testing process in which the tester use advance automation tools for testing software applications areas, which needs to be tested again-and-again due to frequent changes.
Introduction to Agile software testing - The 5th seminar in public seminar series from KMS Technology which have been delivering from 2011 in every two months
Agile testing principles and practices - Anil KaradeIndicThreads
Traditional test processes are not adaptive to extensive changes in software. Agile process emphasizes on ability to adapt to changing business needs, customer collaboration, integrated teams and frequent delivery of business values. Agile is an umbrella term that describes a variety of methods including XP and Scrum.
The talk will discuss pitfalls of the traditional testing process. Traditional testing process happens very late in the SDLC Where as Agile process focuses on test-first approach. The talk will explain benefits of going agile. Principles and practices of agile process will be discussed and agile methodologies Scrum and Extreme Programming will be discussed in detail. Purpose of Scrum, its effectiveness, timings and managing the scrum will be discussed. Some of the practices for XP like Pair Programming, Test Driven Development will be discussed. The Talk will also cover the QA role in agile world. The talk will cover the implementation issues while shifting from traditional to agile process. Talk will also include an interactive game for illustration of concepts.
What are the Key drivers for automation? What are the Challenges in Agile automation and How to deal with them? How to automate? Who will automate? Which tool to select? Commercial or open source? What to automate? Which features? Here is what our experience says
QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных стор...QAFest
Что такое качество продукта и процесса разработки. Как его измерять. Какие метрики гарантируют качество продукта, а какие важны для принятии решения о готовности продукта к релизу. Тренды качества, их польза в понимании улучшения качества продукты и процесса разработки. Абсолютные и относительные метрики. Инструменты.
HCMC Software Testing Club - The 1st Meetup
By Thao Vo
Selecting a most suitable automated testing tool is one of big challenges in software test automation. Choosing a test tool is as complicated as getting married to a person. If you marry with an inappropriate person, you tend to break up sooner or later. Similarly, without a suitable test tool, we will deadly end up with failed test automation effort. There are a variety of automated testing tools with different testing types and technologies. How to define a set of criteria of requirements to meet our goal, and making a right tool will help us prevent later problems from the executing of successful testing project. This topic is intended to share steps and criteria to select an appropriate automated testing tool.
In this session, we would discuss what "Agile Testing" is, what are the well known methods and models of Agile Testing and what to expect on the future of Agile Testing.
Agile Testing Framework - The Art of Automated TestingDimitri Ponomareff
Once your organization has successfully implemented Agile methodologies, there are two major areas that will require improvements: Continuous Integration and Automated Testing.
This presentation illustrates why it's important to invest in an Automated Testing Framework (ATF) to reduce technical debt, increase quality and accelerate time to market.
Learn more at www.agiletestingframework.com.
Slides from a session presented by Fadi Stephan from Kaizenko at the 2019 Global Scrum Gathering in Austin, TX on 05/20/2019 DC. Also see the blog series on Agile Testing at https://www.kaizenko.com/agile-testing/
Abstract:
Many teams struggle with fitting in testing activities inside of a Sprint. They end up doing primarily development activities in a Sprint and push testing activities to run in dedicated testing Sprints following the coding Sprints or have a coding and testing Sprint running in parallel. However, in Scrum, the output of every Sprint is a potentially shippable product increment. This means the product increment should be well tested within the Sprint and ready to be delivered. Come to this presentation to learn how to tackle testing on an Agile team, what kind of tests to execute, what to automate and what not to automate, the different test responsibilities, and when to run which tests. Leave with a testing strategy that you can start applying the next day to gradually get a team to start testing from day 1 of the Sprint and deliver a true product increment at the end of each Sprint.
Software testing concepts is going to enhance from manual testing to automated testing. Due to rapidly increasing complexity in software development process, the present manual testing technique is not sufficient for effective testing. Hence, here arises the requirement of test automation process. The automation testing is the software testing process in which the tester use advance automation tools for testing software applications areas, which needs to be tested again-and-again due to frequent changes.
Introduction to Agile software testing - The 5th seminar in public seminar series from KMS Technology which have been delivering from 2011 in every two months
Agile testing principles and practices - Anil KaradeIndicThreads
Traditional test processes are not adaptive to extensive changes in software. Agile process emphasizes on ability to adapt to changing business needs, customer collaboration, integrated teams and frequent delivery of business values. Agile is an umbrella term that describes a variety of methods including XP and Scrum.
The talk will discuss pitfalls of the traditional testing process. Traditional testing process happens very late in the SDLC Where as Agile process focuses on test-first approach. The talk will explain benefits of going agile. Principles and practices of agile process will be discussed and agile methodologies Scrum and Extreme Programming will be discussed in detail. Purpose of Scrum, its effectiveness, timings and managing the scrum will be discussed. Some of the practices for XP like Pair Programming, Test Driven Development will be discussed. The Talk will also cover the QA role in agile world. The talk will cover the implementation issues while shifting from traditional to agile process. Talk will also include an interactive game for illustration of concepts.
What are the Key drivers for automation? What are the Challenges in Agile automation and How to deal with them? How to automate? Who will automate? Which tool to select? Commercial or open source? What to automate? Which features? Here is what our experience says
QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных стор...QAFest
Что такое качество продукта и процесса разработки. Как его измерять. Какие метрики гарантируют качество продукта, а какие важны для принятии решения о готовности продукта к релизу. Тренды качества, их польза в понимании улучшения качества продукты и процесса разработки. Абсолютные и относительные метрики. Инструменты.
Zero touch QA automation platform for DevOpsTaUB Solutions
Presentation based on the Award Winning Paper by Varadarajan Srinivasan on Zero Touch QA Automation Platform at STC Nov 2018.
Companies that are in DevOps Journey go through a rough weather in QA Automation initiatives. With the industry transformation to Digital Trend, there is always a need for an efficient streamlined Continuous Delivery.
This Webinar would talk about Zero Touch QA Automation Platform. It deals with the Solution on Integrated DevOps & Quality Approach. This would consist of Artificial Intelligence in Reporting and Data Visualization. The Complete QA Cycle is automated from Test Design to Test Closure with Continuous Monitoring, involving no manual intervention. The Webinar would also give insights on Business benefits/outcome on adopting this approach with streamlined Governance.
The quality assurance checklist for progressive testingMaitrikpaida
Quality assurance (QA) is a strategic way of preventing mistakes and defects in developed products and avoiding problems when delivering products or services to customers. This defect prevention in quality assurance differs subtly from defect detection and rejection in quality control and has been referred to as a shift left since it focuses on quality earlier in the process
The Quality Assurance Checklist for Progressive TestingCygnet Infotech
Quality assurance (QA) is a strategic way of preventing mistakes and defects for engineering solutions that are ready-to-market. At Cygnet we adopt the progressive testing approach to deliver high performing solutions. This presentation highlights key factors we consider when creating QA strategies.
Learn how to establish a greater sense of confidence in your release cycle, along with the practices and processes to create a high-performing engineering culture within your team.
Imaginea's Test engineering shares its process guideliness, best practices and recommedations for effective Product testing. Ensures software products behave the way they are supposed to.
End-to-End Quality Approach: 14 Levels of TestingJosiah Renaudin
In 2015, the Standard & Poor’s Ratings IT team set out an ambitious objective—to tighten the process and controls around the quality of code deployed to production. Based on internal cost of quality assessments, and supporting agile and waterfall internal engineering processes, distinct testing levels were identified to help push quality left and root out the underlying causes of defects as early as possible. The ‘14 Levels of Testing’ were defined to collaboratively span organizational functions, establish quality expectations, and help track towards the goal of eliminating defects. Adrian Thibodeau and Chintan Pandya review their 14 Levels of Testing and focus specifically on sharing the processes and tools employed to help govern the delivery of quality. Adrian and Chintan discuss metrics and dashboards, defect lifecycle management, their home-grown QA Workflow Portal, testing vendor SLAs and contracts, and facilitating UAT best-practices.
Load and Performance Testing Services for Mobile Applications | Calidad InfotechCalidad Infotech
Validate the performance of a mobile application during high workloads. Performance testing allows product owners to measure the performance of the app.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Strategies for Successful Data Migration Tools.pptxvarshanayak241
Data migration is a complex but essential task for organizations aiming to modernize their IT infrastructure and leverage new technologies. By understanding common challenges and implementing these strategies, businesses can achieve a successful migration with minimal disruption. Data Migration Tool like Ask On Data play a pivotal role in this journey, offering features that streamline the process, ensure data integrity, and maintain security. With the right approach and tools, organizations can turn the challenge of data migration into an opportunity for growth and innovation.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Hivelance Technology
Cryptocurrency trading bots are computer programs designed to automate buying, selling, and managing cryptocurrency transactions. These bots utilize advanced algorithms and machine learning techniques to analyze market data, identify trading opportunities, and execute trades on behalf of their users. By automating the decision-making process, crypto trading bots can react to market changes faster than human traders
Hivelance, a leading provider of cryptocurrency trading bot development services, stands out as the premier choice for crypto traders and developers. Hivelance boasts a team of seasoned cryptocurrency experts and software engineers who deeply understand the crypto market and the latest trends in automated trading, Hivelance leverages the latest technologies and tools in the industry, including advanced AI and machine learning algorithms, to create highly efficient and adaptable crypto trading bots
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Your Digital Assistant.
Making complex approach simple. Straightforward process saves time. No more waiting to connect with people that matter to you. Safety first is not a cliché - Securely protect information in cloud storage to prevent any third party from accessing data.
Would you rather make your visitors feel burdened by making them wait? Or choose VizMan for a stress-free experience? VizMan is an automated visitor management system that works for any industries not limited to factories, societies, government institutes, and warehouses. A new age contactless way of logging information of visitors, employees, packages, and vehicles. VizMan is a digital logbook so it deters unnecessary use of paper or space since there is no requirement of bundles of registers that is left to collect dust in a corner of a room. Visitor’s essential details, helps in scheduling meetings for visitors and employees, and assists in supervising the attendance of the employees. With VizMan, visitors don’t need to wait for hours in long queues. VizMan handles visitors with the value they deserve because we know time is important to you.
Feasible Features
One Subscription, Four Modules – Admin, Employee, Receptionist, and Gatekeeper ensures confidentiality and prevents data from being manipulated
User Friendly – can be easily used on Android, iOS, and Web Interface
Multiple Accessibility – Log in through any device from any place at any time
One app for all industries – a Visitor Management System that works for any organisation.
Stress-free Sign-up
Visitor is registered and checked-in by the Receptionist
Host gets a notification, where they opt to Approve the meeting
Host notifies the Receptionist of the end of the meeting
Visitor is checked-out by the Receptionist
Host enters notes and remarks of the meeting
Customizable Components
Scheduling Meetings – Host can invite visitors for meetings and also approve, reject and reschedule meetings
Single/Bulk invites – Invitations can be sent individually to a visitor or collectively to many visitors
VIP Visitors – Additional security of data for VIP visitors to avoid misuse of information
Courier Management – Keeps a check on deliveries like commodities being delivered in and out of establishments
Alerts & Notifications – Get notified on SMS, email, and application
Parking Management – Manage availability of parking space
Individual log-in – Every user has their own log-in id
Visitor/Meeting Analytics – Evaluate notes and remarks of the meeting stored in the system
Visitor Management System is a secure and user friendly database manager that records, filters, tracks the visitors to your organization.
"Secure Your Premises with VizMan (VMS) – Get It Now"
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
1. QA Metrics in Agile
Look at the Quality from different angels. Real appliance.
By Volodymyr Prymakov
www.qaexperts.pro
2. About Me
Volodymyr Prymakov, Ukraine
Head of Performance QA Unit at TCoE,
Interim Head of TCoE at Thomas Cook,
Co-owner at QAExpert.pro
•14.5 years in QA
• >45 projects experience
• Certified: ISTQB Advanced Test Manager/Test
analyst, ICAgile CP, SAFE, CSPO
/in/vladimirprimakov/
www.qaexperts.pro
5. Presentation Plan
1. What is quality of product and process
2. What is foundation for metrics
3. Types of Metrics
4. QA Metrics for Daily Monitoring
5. Regular Quality Reports
6. QA trend Metrics
7. User Satisfaction and Qualitative metrics
8. Team Efficiency Metrics
9. Automation Metrics
10. QA Metrics tools
www.qaexperts.pro
7. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
8. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
9. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
10. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
11. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
12. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
13. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
14. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Bad
Product
Deviation in Quality
17. What is quality of Process
Quality Process brings much Business Value with keeping project or program engine safe and efficient
1. Deliver new functionality often
2. Deliver more functionality
3. Delivery quality software
4. Delivery what is planed
5. Deliver with minimal risks
6. Business trusts dev teams, processes, and produced results
www.qaexperts.pro
1. Project vision, goals, roadmap, and other plans are clear to everybody
2. Process is transparent and strait-forward
3. Product quality is always transparent
4. Technical dept is being eliminated
5. Efficient Communication and Interdependency solving
6. Shared decisions making and responsibility
7. Lessons learned and continuous improvement are in place
8. High team sprit and satisfaction
18. What is quality of Process
Quality Process brings much Business Value with keeping project or program engine safe and efficient
1. Deliver new functionality often
2. Deliver more functionality
3. Delivery quality software
4. Delivery what is planed
5. Deliver with minimal risks
6. Business trusts dev teams, processes, and produced results
www.qaexperts.pro
1. Project vision, goals, roadmap, and other plans are clear to everybody
2. Process is transparent and strait-forward
3. Product quality is always transparent
4. Technical dept is being eliminated
5. Efficient Communication and Interdependency solving
6. Shared decisions making and responsibility
7. Lessons learned and continuous improvement are in place
8. High team sprit and satisfaction
19. What is quality of Process
Quality Process brings much Business Value with keeping project or program engine safe and efficient
1. Deliver new functionality often
2. Deliver more functionality
3. Delivery quality software
4. Delivery what is planed
5. Deliver with minimal risks
6. Business trusts dev teams, processes, and produced results
www.qaexperts.pro
1. Project vision, goals, roadmap, and other plans are clear to everybody
2. Process is transparent and strait-forward
3. Product quality is always transparent
4. Technical dept is being eliminated
5. Efficient Communication and Interdependency solving
6. Shared decisions making and responsibility
7. Lessons learned and continuous improvement are in place
8. High team sprit and satisfaction
23. Definition of Done (DoD) defines
Which quality Product should
have?
Which Way and How
thoroughly a product or
functionality should be
tested?
Min Sufficient Max Allowed
www.qaexperts.pro
24. Definition of Done for Stories, e.g.
• Coded and integrated to main branch,
• Unit tests are written and passing,
• API component tests are written and passing,
• Code reviewed and conforms to coding standards
Passed Dev
Testing:
• Code deployed to QA environment. Automatic API-integration and
EndToEnd smoke testing tests are passing,
• Test Design is completed,
• Positive and negative Acceptance Tests are (automated and)
passed on platforms of priority 1. Exploratory tests are passed on
platform of priority 2.
• Selective Regression is carried out by a necessity and passed.
Passed QA testing:
No (P1/P2)
bugs
outstanding
for STORY
and neighbor
functionality
Story meets INVEST criteria
Story is Signed-Off by Product Owner
www.qaexperts.pro
25. Definition of Done for Sprint, e.g.
• Stories that meet DoD are considered DONE, the rest goes to
next sprint.
• It should be avoided to merge Risky or buggy code to main
branch at the end of sprint!
Stories meets DoD
• Regression testing is passed with P1 test cases on at least 1
top priority platform/device
• Smoke or exploratory testing is passed on platforms/devices
of priority 1 and 2
Cover Regression
Test
• Main Integration tests for live 3d-party services are passed on
STAGE or other environments3d party Integration
Test
No (P1/P2)
bugs
outstanding
for product
and neighbor
products.
Sprint is Signed-Off by Chief Product Owner
www.qaexperts.pro
26. Definition of Done for RELEASE, e.g.
• All Stories, which were merged to Main branch, meets DoD for stories.
• Risky or buggy code is not merged to main branch or is toggled off or is
reverted!
Stories meets DoD
• Regression testing is passed for all priority 1 platforms/devices.
• Exploratory testing is passed for priority 2 platforms/devices.
• Smoke testing is passed for priority 3 platforms.
Full Regression Test
• All Integration tests for live 3d-party services are passed on STAGE or
other environments
• There is no other 3rd party release ongoing that may impact release
negatively
3d party Integration Test
and Releases
• UAT is passed on STAGE environment
UAT
• Performance Testing successfully passed
• Security Testing successfully passed
• and other types of testing and preparations to release
Other Types of Testing
No (P1/P2)
bugs
outstanding
for product
and neighbor
products
Release GO Aproval from by Chief Product Owner, QA team, Release Manager and
respective Business representatives
All other unresolved
bugs are tracked,
reviewed, and planned
when to be fixed
www.qaexperts.pro
30. Types of Metrics
Quantative - Information that is directly measured as a
number
Qualitative - Information that results from human
observation and analysis
www.qaexperts.pro
31. Types of Metrics
1. Absolute metrics
2. Relative metrics
3. Derivative metrics
Purpose of Metrics.
1. Goal Metrics: To measure results/goals
2. Actionable metrics: to make decision
3. Predictive metrics: to predict future
www.qaexperts.pro
32. Stages at which QA and other metrics are needed
Implementation period
• Daily Monitoring of
Quality, product
implementation, and
test progress
Key sprint and release
points
• QA Report Before
sprint Demo
• QA Reports During
stabilization /
regression / UAT
period
• QA report Before
Release
• QA Reminder report
when product has
critical quality state
or low product
implementation or
testing progress
After Release
• Release quality and
process analysis
• Quality and process
Trends and analysis
• Team efficiency
analysis
www.qaexperts.pro
34. Goals of Daily Quality Monitoring
1. What is the current quality of the
product?
2. How much work is left?
3. Do we have enough time to finish and
stabilize the product?
www.qaexperts.pro
35. !!!Important:
Breakdown Defect Metrics
Always breakdown your metrics to the following groups:
All Bugs! – shows general product quality state
Critical and Blocker bugs – Shows whether we
meet Readiness criteria!
Lower Priority bugs– Shows the rest of bugs,
general amount of which can negatively influence on the
end-user’s impression about the product.
www.qaexperts.pro
36. Defects reported by Date
Critical and Blocker bugs Major and lower priority bugs
Jira “Recently Created Chart” Gadget
Allows you track new bugs and immediately react to them!
www.qaexperts.pro
37. Open Defects reported Today
Jira “Filter Results” Gadget
Allows you to quickly review new bugs
www.qaexperts.pro
38. Open Defects By Component/Platform
Major and lower priority bugs
Jira “Pie Chart” Gadget
Critical and Blocker bugs
Shows which
component/platform
is not ready for release
or requires bug fixes
www.qaexperts.pro
39. Defects By Component/Platform for Current
Affected Version (any Status)
Major and lower priority bugs
Jira “Pie Chart” Gadget
Critical and Blocker bugs
Shows which
component/platform
require more thorough
testing,
e.g. (Selective)
Regression testing
www.qaexperts.pro
40. Critical Open Bugs by Status
Jira “Issue Statistics” Gadget
Allows you to quickly get Status of Open Critical Bugs and understand whether we have enough time to fix them
www.qaexperts.pro
41. Open Bugs by Priority
Jira “Issue Statistics” Gadget
Shows general quality of the product
www.qaexperts.pro
42. !!!NOT-assigned Open Critical Defects
Jira “Issue Statistics” Gadget
To Fix Version
To Team
(for big teams only)
To Sprint To Developer
Shows probably lost or forgotten Critical open bugs, which should be processed or escalated immediately!
www.qaexperts.pro
43. www.qaexperts.pro
Workload:
Assigned Open Tasks and Bugs for Developers/QAs
Jira “Issue Statistics” Gadget
By teams By Developers
For current Fix Version or sprint
By QAs
Stories
By teams By Developers By QAs
Bugs
To assure that there is no bottlenecks and overloads. To Remind about
work assigned!
To spread load equally between teams, Developers and QAs!
44. Release scope status by developers/QAs/teams
If by the end of sprint or before stabilization period at release a lot of stories are in open status,
it can bring risk of late merge status, not enough time for testing, not ability to finish functionality etc.
Action: Try to spread load equally by teams, QAs and Dev Specialists. Remind about work Not Done.
Postpone certain functionality to next Release!
Stories Bugs
Jira “Two Dimensional Filter Statistics” Gadget
www.qaexperts.pro
45. Stories/Bug Status By Fix Version
created >= -365d
Stories Bugs
created >= -180d
Allows to understand ability to finish/fix all tasks or bugs in time for a certain sprint/release
Jira “Two Dimensional Filter Statistics” Gadget
www.qaexperts.pro
46. Current Fix Version Progress
Shows general release/sprint ProgressJira “Release Report”
www.qaexperts.pro
48. Kanban board for All Open Critical and Blocker Bugs
(For big teams)
Allows to
accumulate
and track all
open Critical
and Blocker
bugs in 1 place
www.qaexperts.pro
49. Regression testing
Recommended Statuses:
• Failed with Critical and Blocker bugs
• Failed with Major and lower priority bugs
• Failed by automated tests
• Blocked
• Not tested
• In Progress
• Passed
www.qaexperts.pro
50. Regression Testing Coverage and Results
For Release/Milestone
– General Results
By Cycle/Stages
- Every cycle should has certain purpose, e.g:
1. Initial Feedback about quality,
2. Exploratory testing,
3. Confirmation testing
4. 3rd Party services testing.
www.qaexperts.pro
51. Regression Testing Coverage and Results
By Product Component
• by platform/device
• by test priority:
• Priority 1
• Priority 2
• Priority 3
By Type of Test:
• Smoke test
• Regression test,
• UAT test
www.qaexperts.pro
54. Sprint Report
1. Story Delivery rate % = Actual Stories Completed vs. Committed Stories, e.g.: 90%
2. Amount of Stories met DoD, e.g.: 25
3. Amount of Stories Moved to a new sprint, e.g.: 4
Because of 1 not implemented, 1 Reopen because of Critical Bugs, 2 Not Tested
Specifies the reasons, risks and recommendations.
4. Regression testing coverage and results: Graph
5. Cross platform testing coverage and results: Graph
6. 3d-party integration testing results: Graph or written explanation.
7. Testing and testing activities NOT done – describe reasons, connected risks, and recommendations.
8. Open Bugs:
All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link to jira)
Open Major bugs opened during the sprint, e.g: 40 Major bugs open during sprint. (link to jira)
9. Point to the buggiest product component and platform: Graph and explanation
10. General Conclusions
www.qaexperts.pro
55. Regression testing reports, daily or by cycles
Reports about Regression testing, cross platform testing, 3d-party integration testing, 3d party releases testing, etc.
1. Name of the Regression testing stage, e.g.: Regression Cycle 1 (Preliminary feedback)
2. Coverage and results: Graph
3. Open Bugs
New (created today) and All Open Critical and Blocker bugs:
New 4 Critical&Blocker Bugs, (All 5 Critical&Blocker Bugs) (list or links to jira)
New (Created today) and All Open Major bugs opened during the release:
New 10 Major bugs, (All Major bugs created for current release) (links to jira)
4. Specify Time needed to finish regression testing and/or necessity in additional regression and other testing.
5. Point to the buggiest product component, platform, or functionality if needed.
6. Recommendations
www.qaexperts.pro
56. Release Readiness testing Report (by necessity)
1. Specify functional testing accomplished and its status:
Regression and cross-platform testing,
3dparty integration testing,
UAT
e.g.: Written explanations and Links to Test management system,
or Graphs of testing coverage and results.
2. Testing and testing activities NOT done – describe reasons, connected risks and recommendations.
3. Specify Non-functional testing status, like performance and security testing.
Whether the product is ready or not ready from non-functional point of view.
Describe connected risks and recommendations.
4. Open Bugs.
All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link)
Open Major bugs opened during the release and All Major bugs, e.g: 40 Major bugs open during release.,
70 Major bugs are Open in general
(list or links to jira)
5. Functionality Not ready for release, etc.: (list or link)+ written explanation
6. General Conclusions!
www.qaexperts.pro
59. Reminder Reports (Be necessity)
1. Critical and Blocker Bugs in open state.
2. A Lot of Open Major bugs
3. A lot of Bugs and stories Not Done in the middle or later
stage of sprints: In development, in Test, etc.
4. Availability of not tested stories during stabilization period.
5. Inability to finish testing in time because of certain issues.
Reminder emails or chat messages about:
www.qaexperts.pro
62. Milestones for trends
• AffectedVersion – AffectedVersion field in jira
• Time Span - e.g. period between start and end
date of sprint
Time based conditions in jira queries:
a) createdDate >= '2017-08-30' and
createdDate <='2017-09-12 ‘
b) status changed TO 'READY TO TEST' after
'2017-08-30' before '2017-09-12‘
c) assignee was in (‘user’)
• Sprint – Sprint Field in jira, etc.
www.qaexperts.pro
65. Found on Environment
All bugs
Shows leaks of
bugs to later
development
stages
Pivot Report
www.qaexperts.pro
66. Bugs By Priority – Any Status
All Bugs
by
Aff. Version
Shows absolute
release quality
history
– decrease in
number of all
and critical bugs
shows process
improvements
Pivot Report
Note: Should be considered carefully when the team size and product increment increase.
Note: Blank version is bad, and it shows that the Affected Version is not specified for
certain bugs and thus they can be lost www.qaexperts.pro
67. Bugs By Priority – OPEN (not-closed)
All Bugs
by
Aff. Version
Shows still open
bugs for
releases
– Ideally No
critical and
blocker bugs
should be here
Pivot Report Note: If critical bugs are shown here for previous releases, they might be lost, forgotten or
blocked. www.qaexperts.pro
68. Bugs by RootCause
Typical Root causes of Bugs:
FrontendOthers
Backend
Frontend
+ Backend
Environme
nt
issues
3d party
components
www.qaexperts.pro
69. Bugs by RootCause
Critical
and Blocker
bugs
Shows Root cause of all bugs
Pivot Report
All Bugs
Shows Root cause of the most
of important bugs
www.qaexperts.pro
70. Created Vs Resolved - Cumulative
Critical
and Blocker
Bugs
Created should be less than resolved
Jira “Created vs. Resolved Chart” Gadget
Bug found vs. fixed chart is a defect analysis line chart that lets us see defect
removal process patterns and understand defect management effectiveness
www.qaexperts.pro
71. Created Vs Resolved - Cumulative
All Bugs
Difference should be minimal
and If it is big and no positive
trend is shown then probably
Bagathon sprint should be
planned
Jira “Created vs. Resolved Chart” Gadget
www.qaexperts.pro
72. Defects Dept
Critical and Blocker Bugs
Note: Should Be always close to Zero for Critical bugs
All Bugs
Spreadsheets + advanced Jira time period based queries
Defects dept is a cumulative difference between Resolved and Open bugs
= (Open in sprint N - Resolved in sprint N)+ (Open in sprint N+1- Resolved in sprint N+1 )+…
Note: Shows accumulation of bugs in time
www.qaexperts.pro
73. Bugs Opened vs Stories Delivered
There should be correlation between the metrics - more stories you deliver more bugs may appear.
Ideal trend is to deliver more user stories with less bugs.
!!!Pay attention to anomalies and investigate them deeper.
Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
74. Bugs per Story Point
Critical
and Blocker Bugs
If number
decreases in
time it means
that we
improve QA
process Note: This metric is very good for cases when team size or
product increment is variable or change (increase)
All Bugs
Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
75. Bugs per Story Point
Other Variations:
1. Bugs per requirement
2. Bugs per developer
3. Bugs per 1000/Code
4. Time lost for bugs h - Time spent for Bugs vs time spent for stories
Note: Very useful to for making future project estimations.
www.qaexperts.pro
76. Fix Rate
Should be close to 100% or higher
Spreadsheets + Advanced Jira period based queries
Critical and Blocker Bugs
All Bugs
Fix Rate = Closed bugs / Open bugs
Ideally should be close to 100%
www.qaexperts.pro
77. “Average Age” and “Time in status”
For Critical
and Blocker
Bugs
Jira “Average Age” and “Time in Status” Gadgets
Average Age - Displays the average number of days issues have been unresolved.
For critical and Blocker bugs this time should be very short.
If process improves it should decrease in time.
Average Time in status, e.g. Open status
- Displays the average number of days issues have spent in status.
For critical and Blocker bugs this time should be very short.
If process improves it should decrease in time.
www.qaexperts.pro
78. Cumulative Flow Chart
Jira Agile Reports
To find process bottlenecks and predict project close time
www.qaexperts.pro
79. Bugs by Components/Platforms
Pivot or other spreadsheet Report
Bugs trend By Component – Critical and Blocker / All Bugs
Shows weakest component of the product, shows
where additional testing should be performed, shows
where technical Improvements should be made,
Shows whether component quality improves
Bugs trend By Platform – Critical and Blocker / All Bugs.
Shows amount of cross-platform bugs, shows the
weakest platform requiring more testing and
stabilization.
.
www.qaexperts.pro
80. Sprint and Business Metrics
Estimates Accuracy % = Estimated amount of
effort/Actual amount of effort-1
Story Delivery Rate % = Actual Stories Completed /
Committed Stories
% of Change Requests = Number of change requests /
Number of Initial Backlog stories
Change Requests Costs = Time spent of change
requests * Average Rate of developers on the project
% Costs of Defect = Average Time for defect fix *
Average developer rate on the project
Trends
For Sprint
General value
www.qaexperts.pro
82. User Satisfaction
Number of hot fixes had to be made in this or that release.
Shows leaks of Critical bugs on prod.
Bugs found by end-users
Critical/Blocker (Shows leaks of Critical bugs on prod.)
Major and Low priority
Customer Complaints and Feedbacks reports. (Net Promoter Score)
A/B Testing results
Shows which functionality is more preferred
Other Marketing investigations and surveys
All these answer the following questions:
a) What are the weakest points of the product?
b) What functionality is most used ?
c) Which focus in product development and testing should be made?
www.qaexperts.pro
83. Qualitative Metrics about the project state
If you want to gather more metrics about project processes, approaches,
technology, communication, mood in teams, etc.
Just ask people to provide evaluation and feedback
Experts evaluation and/or Audit
Surveys
Retrospectives
1-2-1 and other meeting
Rumors/small talks at Smoking room/kitchen
www.qaexperts.pro
85. Developers and Dev Team efficiency
REOPENED:
Reopened Bugs Ratio % = Reopened bugs vs Resolved bugs
Reopened Bugs Ratio % = Reopened Stories vs Stories
delivered to testing
STORY POINTS DELIVERED
STORY DEFECTS ISSUED
STORY DEFECTs/STORY POINTs rate %
QUALITATIVE METRICS: Gather qualitative metrics about
developers: approach to coding, complexity of code,
contribution to innovation, communication efficiency, quick
reaction to problems, etc. Per PROJECT in general
Per Team
Per specific Developer
www.qaexperts.pro
86. QAs and QA team Efficiency
REJECTED
Absolute Number of rejected bugs:
All resolutions,
Not a bug,
Duplicate
Cannot reproduce
Incomplete
Rejected Rate % = rejected vs created
BUGs LEAKS: to PROD other later stages of development.
Break down by
Per product part
Per functionality
So you can correlate it with a certain QA or group of QAs
Separately:
For critical and blocker
For Lower priority
defects
Per PROJECT in general
Per QA Team
Per specific QA
www.qaexperts.pro
87. QAs and QA team Efficiency
WORK DONE:
Stories Tested
Created bugs Critical and Blocker (not rejected)
Created bugs Major and lower priority (not Rejected)
Test cases Created (passed DoD). Test cases rejected.
Test cases Executed.
Automation scripts Created (passed DoD). Test cases
rejected.
Advanced test artifacts created and activities
accomplished: test suites, plans, release support, etc.
QUALITATIVE METRICS: Gather qualitative metrics about QAs:
Complexity of tasks, Communication efficiency, Proactivity,
Contribution to innovation, Test management skils, Conflict solving,
etc.
Per PROJECT in general
Per QA team
Per specific QA
For Sprint (Velocity)
Trends
General value
www.qaexperts.pro
90. Unit test coverage & number of test
API testing coverage & number of test
for product in general
by component
GUI Automation testing coverage & number of test
for product in general
by product part
by platform/device
by type of test
end-to-end test
Smoke test
Regression test
UAT test
by test priority:
Priority 1
Priority 2
Priority 3
Automation Coverage & Results
www.qaexperts.pro
91. Automation Efficiency
AUTOMATION SPEED:
Automation tests Frequency
Duration of tests Running
Velocity of script creation and maintenance
Time for script creation and maintenance
AUTOMATION EFFICIENCY:
Stability of automation
False Positives
Root Cause for failure
Bugs found by Automation
Critical and Blocker
Other bugs
Bugs missed with automation.
Critical and Blocker
Other bugs
ROI – Time automation economy, etc.
Cost per automated test
Per PROJECT in general
Per QA team
Per specific QA
For Sprint (Velocity)
Trends
General value
www.qaexperts.pro
94. Tools
Jira + Special time-
related JQL
Confluence + Jira and
other Macros in it
Spreadsheet processors:
Googlesheets, Excel
TestTrail
www.qaexperts.pro
95. JiraReportingInGsheets Tool
Link to Help:
http://jirareportingingsheets.qaexperts.pro/
JiraReportingInGSheets free add-on
for GoogleSheets
allows to automatically build custom jira reports
in google sheets
Created by (QAExperts.pro):
www.qaexperts.pro
Volodymyr PrymakovPilvinskyi Dmitry