Verification AuditHelping Test Teams to Improve
Purpose• This document outlines questions suitable for an audit on a test team• By asking themselves these questions the team can understand what areas are in need of improvement• The purpose of the audit is to support the test team, not to evaluate and compare to other test teams
Vision & MissionDoes the test team have a vision and a mission that are applicable tothem? What is the test team trying to achieve and how are they trying toachieve it?• Most likely the organization the test team is a part of has a vision and a mission of some kind – these should be known by the test team, and broken down to a suitable level that is useful to the team• What are the overall priorities? Time-to-market? Quality?• What are the teams goals? Finding all relevant defects? Providing accurate decision material? Reducing risk?
MindsetWhat is a tester’s role in the organization?• Quality Police? Development Support? Testing-as-a-service?• A test team must know what there role is• A test team must understand the difference between trying to find all defects in a piece of software, and helping the developer create better quality code• Depending on what the tester’s role is in the organization, the job of the test team differs and their focus must be different
Organization & ProcessHow is the test organization set up at the company? What does theoverall test process look like? Who tests what and when? How is the testprocess connected to the development process?• It is important for every test team to understand how they fit in to the bigger picture without this understanding the team can easily overlap other teams, and miss gaps in their test effort• It is also important that the test team knows who they should communicate with in other parts of the company• Knowing how the team’s activities fit in to the overall testing process is also critical to be able to work efficiently and effectively• How does the test process and the development process interact? This includes requirement work, stand-up meetings and similar activities
StakeholdersWho are the test team’s stakeholders and what is it that the test teamdelivers to these stakeholders?• Who receives the test reports of the test team?• What are the needs of these stakeholders?• Are other test teams dependant on the test report to set their test scopes?• What decisions are based on the test reports?• Is the test report good enough? Could it be improved?• How can the test team support stakeholders in other ways than test reports?• How can the test team support developers?
Tool & EnvironmentWhat specific test tools and test environment s are needed for the testteam to perform their testing? How are these tools optimized for the job?If the team could change or improve some tool, would they?• Test tools and environments should empower the tester to become more efficient and effective – is this the case in the test team?• How could the tools be improved?• Are there any tools missing today that would improve efficiency?• How are the tools version handled and maintained?• Tool documentation?
AutomationWhat is the maturity of test automation within the team? Why does theautomated scope look like it does?• Are there any limitations to automation? Tools? Difficulties in automating certain test cases?• How did the team come up with what test scope to automated? Did the team start by creating manual test cases and then try to automated these? Or did they start by looking at what is possible to automated and build the scope based on that?• How is competence in the team for test automation?• How is testability a part of the test teams work?
Risk & ScopeHow does the test team perform risk analysis? How does the test teamset their scope for different test activities?• What inputs does the test team user for their risk analysis?• How much is gut feeling when it comes to risk analysis?• Is there extensive historical data to use?• Are there any tools available to facilitate risk analysis?• How is scope setting influenced by what other teams test?• How do you take into account testing different versions, customizations and variants of the software?• Are there any guidelines available to support in the scope setting and risk analysis• How are test plans created?
MethodsHow does the test team perform their testing? Scripted tests orexploratory testing? Formal methods or gut feeling when creatingtests?• Does the team perform manual scripted testing? When and why?• Does the team perform exploratory testing? When and why?• Does the team use any quality model/heuristic model when creating tests?• Does the team use formal test techniques when creating tests?• Are there any guidelines available for creating tests?
KPI & MetricsWhich KPI & Metrics are relevant to the test team? How is the test teammeasured by management? Do these metrics drive good or badbehaviour?• How is the test team performance measured?• Are these measurements meaningful?• What behaviour do these measurements encourage?• How can the teams performance be measured in a meaningful way without promoting bad behaviour?
PrioritiesWhen the workload is high, how does the test team prioritize betweenactivities, and how do they prioritize during scope setting? When theworkload is low, how is the extra time spent?• Is it clear for the team how they should prioritize their work?• How is this prioritization done?• What does the test team do in periods of low activity? Improvement projects? Additional testing in the projects they are involved in?
Results• Rate the answers of each question using the following system Compliant Partly Compliant Not Compliant• And write a short comment for each question to summarize the answer
Summary• These questions should raise thoughts and ideas about how the test team can improve, both within the team, and from the auditor• The auditor must be an experienced tester to ask the right follow-up questions to the team in each question• These results should not be used to compare teams to each other, but instead to find potential improvement areas – both from the team itself and from the auditor• If all test teams in the organization can answer all these questions satisfactory then the state of verification in the company can be considered quite good