2. Purpose• This document outlines the two major trends in software testing on a very high level• Far from everything is covered, and this document is simplified to quickly give a broad understanding of what is being discussed in the software testing world
3. Software Testing Trends OverviewContext- Microsoft Driven &Testing Google
4. Context-Driven Testing Principles is Action • Testing groups exist to provide testing-related services. They do not run the development project; they serve the project.• Testing is done on behalf of stakeholders in the service of developing, qualifying, debugging, investigating, or selling a product. Entirely different testing strategies could be appropriate for these different objectives.• It is entirely proper for different test groups to have different missions. A core practice in the service of one mission might be irrelevant or counter-productive in the service of another.• Metrics that are not valid are dangerous.• The essential value of any test case lies in its ability to provide information (i.e. to reduce uncertainty).• All oracles are fallible. Even if the product appears to pass your test, it might well have failed it in ways that you (or the automated test program) were not monitoring.• Automated testing is not automatic manual testing: it’s nonsensical to talk about automated tests as if they were automated human testing.• Different types of defects will be revealed by different types of tests–tests should become more challenging or should focus on different risks as the program becomes more stable.• Test artifacts are worthwhile to the degree that they satisfy their stakeholders’ relevant requirements.
5. Context-Driven Trends [1,2,8,9,10]• Testing as a craft – The tester role is put in focus – Methods and practices that support the individual tester in performing better testing are advocated• Tests and Checks – Differentiation between checks and tests, where checks work as regression tests, and tests explore new grounds – Automated tests which are executed are checks and the general view is that they are mostly overvalued and cannot replace real testing• Unnecessary artefacts – Test cases, test plans and other test artefacts should be kept to a minimum to maximize the time a tester spends on actual testing• KPI and quality assessments – KPI are mostly considered to be of low value, since quality is very hard to quantify – The quality assessment of the individual tester is put in focus• Session Based Exploratory Testing – Session based exploratory testing combines the above bullets into a usable method which is advocated in combination
6. Testing at Google “This brings us to the current chapter in test which I call Testing 1.5. Thischapter is being written by computer scientists, applied scientists,engineers, developers, statisticians, and many other disciplines. Thesepeople come together in the Software Engineer in Test (SET) and TestEngineer (TE) roles at Google. SET/TEs focus on; developing softwarefaster, building it better the first time, testing it in depth, releasing itquicker, and making sure it works in all environments. We often put deeptest focus on Security, Reliability and Performance. I sometimes think ofthe SET/TE’s as risk assessors whose role is to figure out the probability offinding a bug, and then working to reduce that probability. Superinteresting computer science problems where we take a solid engineeringapproach, rather than a process oriented / manual / people intensivebased approach. We always look to scale with machines whereverpossible.”
7. Microsoft & Google Trends [3,4,5,6,7]• Focus on testing as an activity in the development process – Testing is done by many different roles; test engineer, software developer in test, software developer – Testing as an integrated part of the process, designing software with testability and automation in mind• Focus on good automation – Design good automated test cases, don’t try to automate manual test cases – Understand the costs of automation and use automation correctly• It is not a numbers game – Don’t hire a lot of testers – Maximize the power of the testers you have – Tool support for as many activities as possible to optimize tester performance to spend as much time as possible on actual testing – Integrated defect reporting, easy risk-management tools, low maintenance test plans, focus on exploratory testing for manual testing, etc.• Involving users as early as possible in an organized way – Live User testing as soon as possible• Continuous integration systems – Use of smart tools and cloud computing infrastructure in the continuous integration system makes it fast and reliable
8. Summary• Two trends with many things in common• The major differences are – Focus on the tester role vs. focus on testing activity independent of role – Focus on manual exploratory testing vs. focus on maximizing usage of test automation, starting early with testability built into code – More focus on the individual tester´s qualitative report vs. Continuous integration system with more quantitative reports – Customer s as requirement stakeholders vs. Customers as live user testers• The major similarities are – Testing requires qualified professionals – Reduce the time spent on useless artefacts, either by tool support, or by removing them completely – Mindless automation is not valuable – Customer involvement is important – Everything is risk-based!
9. Reference Michael Bolton [Context-Driven Testing]http://www.developsense.com/blog/ James Bach [Context-Driven Testing]http://www.satisfice.com/blog/ BJ Rollison [Microsoft]http://www.testingmentor.com/imtesty/ James Whittaker [Microsoft]http://blogs.msdn.com/b/jw_on_tech/ Google Testing Blog [Google]http://googletesting.blogspot.se/ Alan Page [Microsoft]http://angryweasel.com/blog/ Expert Testers [Microsoft]http://experttesters.com/ Jonathan Kohl [Mobile Testing]http://www.kohl.ca/blog/ Elizabeth Hendriksson [Agile Testing]http://testobsessed.com/ Cem Kaner [Context-Driven Testing]http://context-driven-testing.com/