You're under tight time pressure and have barely enough information to proceed with testing. How do you test quickly and inexpensively, yet still produce informative, credible, and accountable results? Rapid Software Testing, adopted by context-driven testers worldwide, offers a field-proven answer to this all-too-common dilemma. In this one-day sampler of the approach, Michael Bolton introduces you to the skills and practice of Rapid Software Testing through stories, discussions, and "minds-on" exercises that simulate important aspects of real testing problems. The rapid approach isn't just testing with speed or a sense of urgency; it's mission-focused testing that eliminates unnecessary work, assures that the most important things get done, and constantly asks how testers can help speed up the successful completion of the project. Join Michael to learn how Rapid Testing focuses on both the mind set and skill set of the individual tester, using tight loops of exploration and critical thinking skills to help continuously re-optimize testing to match clients' needs and expectations.
A Rapid Introduction to Rapid Software TestingTechWell
You're under tight time pressure and have barely enough information to proceed with testing. How do you test quickly and inexpensively, yet still produce informative, credible, and accountable results? Rapid Software Testing, adopted by context-driven testers worldwide, offers a field-proven answer to this all-too-common dilemma. In this one-day sampler of the approach, Paul Holland introduces you to the skills and practice of Rapid Software Testing through stories, discussions, and "minds-on" exercises that simulate important aspects of real testing problems. The rapid approach isn't just testing with speed or a sense of urgency; it's mission-focused testing that eliminates unnecessary work, assures that the most important things get done, and constantly asks how testers can help speed up the successful completion of the project. Join Paul to learn how rapid testing focuses on both the mind set and skill set of the individual tester who uses tight loops of exploration and critical thinking skills to help continuously re-optimize testing to match clients' needs and expectations.
A Rapid Introduction to Rapid Software TestingTechWell
This document provides a summary of a presentation on Rapid Software Testing. The presentation was given by Michael Bolton of DevelopSense and covered the methodology and mindset of rapid software testing. It emphasizes testing software expertly under uncertainty and time pressure. The presentation defines rapid testing as testing more quickly and less expensively while still achieving excellent results. It compares rapid testing to other approaches like exhaustive, ponderous, and slapdash testing. The presentation also discusses principles of rapid testing, how to recognize problems quickly using heuristics, and testing rapidly to fulfill the mission of testing.
A test strategy is the set of ideas that guides your test design. It's what explains why you test this instead of that, and why you test this way instead of that way. Strategic thinking matters because testers must make quick decisions about what needs testing right now and what can be left alone. You must be able to work through major threads without being overwhelmed by tiny details. James Bach describes how test strategy is organized around risk but is not defined before testing begins. Rather, it evolves alongside testing as we learn more about the product. We start with a vague idea of our strategy, organize it quickly, and document as needed in a concise way. In the end, the strategy can be as formal and detailed as you want it to be. In the beginning, though, we start small. If you want to focus on testing and not paperwork, this approach is for you.
This document provides an overview and introduction to the Rapid Software Testing course. It acknowledges those who contributed to developing the course material. The document outlines some assumptions about the audience for the course, including that attendees test software and want to improve their testing process. It presents the primary goal of the course as teaching how to test under uncertainty and with scrutiny. Key themes of Rapid Testing are also summarized, including putting the tester's mind at the center and considering cost versus value in testing activities.
Herman- Pieter Nijhof - Where Do Old Testers Go?TEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Where Do Old Testers Go? by Herman- Pieter Nijhof. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
Michael Bolton - Heuristics: Solving Problems RapidlyTEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Heuristics: Solving Problems Rapidly by Michael Bolton. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
Rikard Edgren - Testing is an Island - A Software Testing DystopiaTEST Huddle
This document summarizes trends in software testing that could diminish its effectiveness and enjoyment. It notes an increasing focus on verification over validation, precise measurement over subjective judgement, and short-term metrics over long-term quality. This narrowing scope risks making testers isolated and limiting their creativity, motivation and ability to consider the full context of a project. The document advocates a holistic and subjective approach that considers people and intangible factors, not just short-term quantifiable results. Subjectivity and considering the whole system, not just parts, are presented as useful for testing.
Ho Chi Minh City Software Testing Conference January 2015
Software Testing in the Agile World
Website: www.hcmc-stc.org
Author: Lee Copeland
The IEEE 829 Test Documentation standard is thirty years old this year. Boris Beizer’s first book on software testing also turned thirty. Testing Computer Software, the best selling book on software testing, is twenty-five. During the last three decades, hardware platforms have evolved from mainframes to minis to desktops to laptops to tablets to smartphones. Development paradigms have shifted from waterfall to agile. Consumers expect more functionality, demand higher quality, and are less loyal to brands. The world has changed dramatically and testing must change to match it. Testing processes that helped us succeed in the past may prevent our success in the future. Lee Copeland shares his insights into the future of testing, sharing his Do’s and Don’ts in the areas of technology, organization, test processes, test plans, and automation. Join Lee for a thought provoking look at creating a better testing future.
A Rapid Introduction to Rapid Software TestingTechWell
You're under tight time pressure and have barely enough information to proceed with testing. How do you test quickly and inexpensively, yet still produce informative, credible, and accountable results? Rapid Software Testing, adopted by context-driven testers worldwide, offers a field-proven answer to this all-too-common dilemma. In this one-day sampler of the approach, Paul Holland introduces you to the skills and practice of Rapid Software Testing through stories, discussions, and "minds-on" exercises that simulate important aspects of real testing problems. The rapid approach isn't just testing with speed or a sense of urgency; it's mission-focused testing that eliminates unnecessary work, assures that the most important things get done, and constantly asks how testers can help speed up the successful completion of the project. Join Paul to learn how rapid testing focuses on both the mind set and skill set of the individual tester who uses tight loops of exploration and critical thinking skills to help continuously re-optimize testing to match clients' needs and expectations.
A Rapid Introduction to Rapid Software TestingTechWell
This document provides a summary of a presentation on Rapid Software Testing. The presentation was given by Michael Bolton of DevelopSense and covered the methodology and mindset of rapid software testing. It emphasizes testing software expertly under uncertainty and time pressure. The presentation defines rapid testing as testing more quickly and less expensively while still achieving excellent results. It compares rapid testing to other approaches like exhaustive, ponderous, and slapdash testing. The presentation also discusses principles of rapid testing, how to recognize problems quickly using heuristics, and testing rapidly to fulfill the mission of testing.
A test strategy is the set of ideas that guides your test design. It's what explains why you test this instead of that, and why you test this way instead of that way. Strategic thinking matters because testers must make quick decisions about what needs testing right now and what can be left alone. You must be able to work through major threads without being overwhelmed by tiny details. James Bach describes how test strategy is organized around risk but is not defined before testing begins. Rather, it evolves alongside testing as we learn more about the product. We start with a vague idea of our strategy, organize it quickly, and document as needed in a concise way. In the end, the strategy can be as formal and detailed as you want it to be. In the beginning, though, we start small. If you want to focus on testing and not paperwork, this approach is for you.
This document provides an overview and introduction to the Rapid Software Testing course. It acknowledges those who contributed to developing the course material. The document outlines some assumptions about the audience for the course, including that attendees test software and want to improve their testing process. It presents the primary goal of the course as teaching how to test under uncertainty and with scrutiny. Key themes of Rapid Testing are also summarized, including putting the tester's mind at the center and considering cost versus value in testing activities.
Herman- Pieter Nijhof - Where Do Old Testers Go?TEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Where Do Old Testers Go? by Herman- Pieter Nijhof. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
Michael Bolton - Heuristics: Solving Problems RapidlyTEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Heuristics: Solving Problems Rapidly by Michael Bolton. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
Rikard Edgren - Testing is an Island - A Software Testing DystopiaTEST Huddle
This document summarizes trends in software testing that could diminish its effectiveness and enjoyment. It notes an increasing focus on verification over validation, precise measurement over subjective judgement, and short-term metrics over long-term quality. This narrowing scope risks making testers isolated and limiting their creativity, motivation and ability to consider the full context of a project. The document advocates a holistic and subjective approach that considers people and intangible factors, not just short-term quantifiable results. Subjectivity and considering the whole system, not just parts, are presented as useful for testing.
Ho Chi Minh City Software Testing Conference January 2015
Software Testing in the Agile World
Website: www.hcmc-stc.org
Author: Lee Copeland
The IEEE 829 Test Documentation standard is thirty years old this year. Boris Beizer’s first book on software testing also turned thirty. Testing Computer Software, the best selling book on software testing, is twenty-five. During the last three decades, hardware platforms have evolved from mainframes to minis to desktops to laptops to tablets to smartphones. Development paradigms have shifted from waterfall to agile. Consumers expect more functionality, demand higher quality, and are less loyal to brands. The world has changed dramatically and testing must change to match it. Testing processes that helped us succeed in the past may prevent our success in the future. Lee Copeland shares his insights into the future of testing, sharing his Do’s and Don’ts in the areas of technology, organization, test processes, test plans, and automation. Join Lee for a thought provoking look at creating a better testing future.
Fabian Scarano - Preparing Your Team for the FutureTEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Preparing Your Team for the Future by Fabian Scarano. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
This document provides an overview of exploratory testing techniques. It discusses that exploratory testing involves simultaneous learning, test design, and test execution. Exploratory testing is tester-centric and focuses on problem solving strategies like heuristics rather than scripts. The document dispels some myths about exploratory testing, including that it is unstructured and cannot involve documentation. It provides examples of how documents can be used for reflection, information sharing, and reporting in exploratory testing.
The document provides guidance for managing a team of junior testers. It discusses challenges such as lack of skills and experience in junior testers. It recommends setting clear expectations, providing frequent communication and feedback, ensuring knowledge sharing, and protecting the team to help them succeed. Patience and structure are important, as is repeating key messages, to help junior testers learn and improve. The goal is for the team to work cooperatively toward a common objective.
- The speaker proposes 16 "test axioms" that are intended to provide a framework for testing approaches and represent principles that are context-insensitive and self-evidently true.
- The axioms are grouped into three categories: stakeholders, design, and delivery. The speaker argues the axioms can help testers think critically about testing and identify flaws in arguments.
- It is argued that process improvement models are not effective for improving testing because there is no consensus on best practices and processes must be tailored to context. True improvement requires understanding why current approaches are used given the context.
The Test Coverage Outline: Your Testing Road MapTechWell
To assist in risk analysis, prioritization of testing, and test reporting (telling your testing story), you need a thorough Test Coverage Outline (TCO)—a road map of your proposed testing activities. By creating a TCO, you can prepare for testing without having to create a giant pile of detailed test cases. Paul Holland says that a comprehensive TCO helps the test team to get buy-in for the overall test strategy very early in the project and is valuable for identifying risk areas, testability issues, and resource constraints. Paul describes how to create a TCO including the use of heuristic-based checklists to help ensure you don’t overlook important elements in your testing. Learn multiple approaches for critical information gathering, the artifacts used as input for creating a TCO, and how you can use a TCO to maintain testing focus. Take back a new, lightweight tool to help you tell the testing story throughout your project.
This talk suggests how we might make sense of the tools landscape of the near future, where the pressure to modernise processes and automate is greatest, and what a new test process supported by tools might look like.
Takeaways:
- We need to take machine learning in testing seriously, but it won’t be taking our jobs just yet
- We don’t need more test automation tools; today we need tools that capture tester knowledge
- Tools that that learn and think can’t work for testers until we solve the knowledge capture challenge.
View On-Demand Webinar: https://youtu.be/EzyUdJFuzlE
Santa Barbara Agile: Exploratory Testing Explained and ExperiencedMaaret Pyhäjärvi
Exploratory Testing Explained and Experienced
- Exploratory testing is an approach to software testing that involves dynamically testingsoftware without a fixed plan, using the results of previous tests to determine subsequent tests.
- It is a disciplined approach that finds unknown unknowns and helps testers examine software from different perspectives to uncover more bugs. Tests are performances rather than fixed artifacts.
- Exploratory testing requires testers to be able to strategically choose and defend their test approaches, explain what they have tested, and determine when they are done testing rather than just finding bugs randomly. It is a more systematic approach than unplanned testing.
Ho Chi Minh City Software Testing Conference January 2015
Software Testing in the Agile World
Website: www.hcmc-stc.org
Author: Nhat Do, Vu Duong
Context-Driven Testing (CDT) rejects the notion of generalized “best practices” that apply to all projects, and instead accepts that different practices work best under different circumstances. The third principle of the seven defined in CDT states that people are the most important part of any project’s context. Less of a focus on processes and tools, with more emphasis on people and their collaboration empowers testers with the freedom to make choices about how best to do their job without following a restrictive plan.
In joining the game of workshop and some theory sharing in slides, you will a better understanding of Context-Driven Testing practices, principles and its benefits as well as know how is a nice Marriage of Agile and Context-Driven Testing.
Erkki Poyhonen - Software Testing - A Users GuideTEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Software Testing - A Users Guide by Erkki Poyhonen. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
This document discusses why checklists are better than test cases for documentation in quality assurance. It argues that test cases become overcrowded and focus too much on documentation rather than core functions. Checklists are more time-saving and easy to update. An example compares a test case to a checklist for login/registration flows. The author's company Hipo uses a test pad and robot framework integrated with checklists to share with clients and team members.
Ho Chi Minh City Software Testing Conference January 2015
Software Testing in the Agile World
Website: www.hcmc-stc.org
Author: Lee Copeland
Over the years writers have defined testing as a process of finding, a process of evaluating, a process of measuring, a process of improving. For a quarter of a century we as testers have been focused on the internal process of testing, while generally disregarding its real purpose. The real purpose of testing is to create information. James Bach nailed it when he wrote, “The ultimate reason testers exist is to provide information that others on the project use to create things of value.” That is why testing exists — to provide information of value. So, when managers complain that testing “costs too much” perhaps they are really trying to say, “I’m not getting enough valuable information to justify the cost of testing.” When testers say “my management doesn’t see the value in our work” perhaps they are really trying to say, “My management doesn’t value the information I’m providing to them.” To prove our worth, to increase the value of testing, we must first focus on testing’s purpose — providing valuable information — not its process. Join Lee as he discusses why quantifying the value of testing is difficult work — perhaps that’s why we concentrate so much on testing process—that’s much easier. But until we do this difficult work, until we prove our worth through quantifying our contribution, we should expect the bombardments to continue.
Graham Thomas - The Testers Toolbox - EuroSTAR 2010TEST Huddle
EuroSTAR Software Testing Conference 2010 presentation on The Testers Toolbox by Graham Thomas. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
Things Could Get Worse: Ideas About Regression TestingTechWell
Michael Bolton, DevelopSense
Tester, consultant, and trainer Michael Bolton is the coauthor (with James Bach) of Rapid Software Testing, a course that presents a methodology and mindset for testing software expertly in uncertain conditions and under extreme time pressure. Michael is a leader in the context-driven software testing movement with twenty years of experience testing, developing, managing, and writing about software. Currently, he leads DevelopSense, a Toronto-based consultancy.
Using Stories to Test Requirements and SystemsPaul Gerrard
The document discusses using business stories to test requirements and systems. It explains that stories can help identify omissions, inconsistencies, and ambiguity in requirements. Stories are applicable at any stage of a project for different purposes. Structured stories follow a common format with a header, scenarios with given/when/then structures, and can have multiple scenarios to test different conditions. Stories can validate requirements by example and generate both manual and automated test cases. The document argues that a structured, disciplined approach to stories can benefit both agile and structured development approaches.
Exploratory testing is an approach to testing that emphasizes the freedom and responsibility of testers to continually optimize the value of their work. It is the process of three mutually supportive activities done in parallel: learning, test design, and test execution. With skill and practice, exploratory testers typically uncover an order of magnitude more problems than when the same amount of effort is spent on procedurally scripted testing. All testers conduct exploratory testing in one way or another, but few know how to do it systematically to obtain the greatest benefits. Even fewer can articulate the process. Jon Bach looks at specific heuristics and techniques of exploratory testing that will help you get the most from this highly productive approach. Jon focuses on the skills and dynamics of exploratory testing, and how it can be combined with scripted approaches.
Exploratory testing is an approach that emphasizes freedom and responsibility of individual testers in a process where continuous learning, test design, and execution occur simultaneously. It is a disciplined, planned, and controlled form of testing that focuses on continuous learning. Research has shown there is no significant difference in results between exploratory testing and preplanned test cases, but exploratory testing requires significantly less effort overall. Effective exploratory testing requires skills like making models, keeping an open mind, and risk-based testing approaches. Both the strengths and potential blind spots of exploratory testing are discussed.
This document discusses the need to rethink the role of testers in agile and structured projects. It argues that changes in business demands and development practices are squeezing testers and that many current testing roles and skills may disappear. Specifically, it predicts that half of onshore testing roles will be eliminated in 5 years. It recommends testers focus on more strategic roles like business analysis, requirements management, and assurance rather than traditional testing tasks.
Stefaan Lukermans & Dominic Maes - Testers And Garbage Men - EuroSTAR 2011TEST Huddle
EuroSTAR Software Testing Conference 2013 presentation on Testers And Garbage Men by Stefaan Lukermans & Dominic Maes. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
The Snail Entrepreneur: The 7-year-old kid every startup should learn fromClaudio Perrone
Matteo faced a seemly impossible problem, but didn't give up. He used daddy's #PopcornFlow and pivoted. 17 options and 5 experiments later, he converged to success.
PopcornFlow is impacting businesses (large and small) but also families and kids.
If you like this story, please contribute to Matteo's cause.
The performance of your application affects your business more than you might think. Top engineering organizations think of performance as not just a nice-to-have feature but as a crucial feature of their product. Those organizations understand that performance has a direct impact on user experience and, ultimately, their bottom line. Unfortunately, most engineering teams do not regularly test the performance and scalability of their infrastructure. Jim Hirschauer shares the latest performance testing tools and insights into why your team should add performance testing to an agile development process. Learn how to evaluate performance and scalability with MultiMechanize, Bees with Machine Guns, and Google PageSpeed. Take back an understanding of how to automate performance testing and evaluate the impact it has on performance and on your business.
Application Performance Testing: A Simplified Universal ApproachTechWell
Scott Barber presented on approaches for performance testing throughout the software development lifecycle. He advocated for taking a preventative approach through continuous collaboration between developers and testers from the start of a project. This includes activities like code profiling, load testing components, and establishing performance goals at each stage with a focus on catching issues early.
Fabian Scarano - Preparing Your Team for the FutureTEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Preparing Your Team for the Future by Fabian Scarano. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
This document provides an overview of exploratory testing techniques. It discusses that exploratory testing involves simultaneous learning, test design, and test execution. Exploratory testing is tester-centric and focuses on problem solving strategies like heuristics rather than scripts. The document dispels some myths about exploratory testing, including that it is unstructured and cannot involve documentation. It provides examples of how documents can be used for reflection, information sharing, and reporting in exploratory testing.
The document provides guidance for managing a team of junior testers. It discusses challenges such as lack of skills and experience in junior testers. It recommends setting clear expectations, providing frequent communication and feedback, ensuring knowledge sharing, and protecting the team to help them succeed. Patience and structure are important, as is repeating key messages, to help junior testers learn and improve. The goal is for the team to work cooperatively toward a common objective.
- The speaker proposes 16 "test axioms" that are intended to provide a framework for testing approaches and represent principles that are context-insensitive and self-evidently true.
- The axioms are grouped into three categories: stakeholders, design, and delivery. The speaker argues the axioms can help testers think critically about testing and identify flaws in arguments.
- It is argued that process improvement models are not effective for improving testing because there is no consensus on best practices and processes must be tailored to context. True improvement requires understanding why current approaches are used given the context.
The Test Coverage Outline: Your Testing Road MapTechWell
To assist in risk analysis, prioritization of testing, and test reporting (telling your testing story), you need a thorough Test Coverage Outline (TCO)—a road map of your proposed testing activities. By creating a TCO, you can prepare for testing without having to create a giant pile of detailed test cases. Paul Holland says that a comprehensive TCO helps the test team to get buy-in for the overall test strategy very early in the project and is valuable for identifying risk areas, testability issues, and resource constraints. Paul describes how to create a TCO including the use of heuristic-based checklists to help ensure you don’t overlook important elements in your testing. Learn multiple approaches for critical information gathering, the artifacts used as input for creating a TCO, and how you can use a TCO to maintain testing focus. Take back a new, lightweight tool to help you tell the testing story throughout your project.
This talk suggests how we might make sense of the tools landscape of the near future, where the pressure to modernise processes and automate is greatest, and what a new test process supported by tools might look like.
Takeaways:
- We need to take machine learning in testing seriously, but it won’t be taking our jobs just yet
- We don’t need more test automation tools; today we need tools that capture tester knowledge
- Tools that that learn and think can’t work for testers until we solve the knowledge capture challenge.
View On-Demand Webinar: https://youtu.be/EzyUdJFuzlE
Santa Barbara Agile: Exploratory Testing Explained and ExperiencedMaaret Pyhäjärvi
Exploratory Testing Explained and Experienced
- Exploratory testing is an approach to software testing that involves dynamically testingsoftware without a fixed plan, using the results of previous tests to determine subsequent tests.
- It is a disciplined approach that finds unknown unknowns and helps testers examine software from different perspectives to uncover more bugs. Tests are performances rather than fixed artifacts.
- Exploratory testing requires testers to be able to strategically choose and defend their test approaches, explain what they have tested, and determine when they are done testing rather than just finding bugs randomly. It is a more systematic approach than unplanned testing.
Ho Chi Minh City Software Testing Conference January 2015
Software Testing in the Agile World
Website: www.hcmc-stc.org
Author: Nhat Do, Vu Duong
Context-Driven Testing (CDT) rejects the notion of generalized “best practices” that apply to all projects, and instead accepts that different practices work best under different circumstances. The third principle of the seven defined in CDT states that people are the most important part of any project’s context. Less of a focus on processes and tools, with more emphasis on people and their collaboration empowers testers with the freedom to make choices about how best to do their job without following a restrictive plan.
In joining the game of workshop and some theory sharing in slides, you will a better understanding of Context-Driven Testing practices, principles and its benefits as well as know how is a nice Marriage of Agile and Context-Driven Testing.
Erkki Poyhonen - Software Testing - A Users GuideTEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Software Testing - A Users Guide by Erkki Poyhonen. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
This document discusses why checklists are better than test cases for documentation in quality assurance. It argues that test cases become overcrowded and focus too much on documentation rather than core functions. Checklists are more time-saving and easy to update. An example compares a test case to a checklist for login/registration flows. The author's company Hipo uses a test pad and robot framework integrated with checklists to share with clients and team members.
Ho Chi Minh City Software Testing Conference January 2015
Software Testing in the Agile World
Website: www.hcmc-stc.org
Author: Lee Copeland
Over the years writers have defined testing as a process of finding, a process of evaluating, a process of measuring, a process of improving. For a quarter of a century we as testers have been focused on the internal process of testing, while generally disregarding its real purpose. The real purpose of testing is to create information. James Bach nailed it when he wrote, “The ultimate reason testers exist is to provide information that others on the project use to create things of value.” That is why testing exists — to provide information of value. So, when managers complain that testing “costs too much” perhaps they are really trying to say, “I’m not getting enough valuable information to justify the cost of testing.” When testers say “my management doesn’t see the value in our work” perhaps they are really trying to say, “My management doesn’t value the information I’m providing to them.” To prove our worth, to increase the value of testing, we must first focus on testing’s purpose — providing valuable information — not its process. Join Lee as he discusses why quantifying the value of testing is difficult work — perhaps that’s why we concentrate so much on testing process—that’s much easier. But until we do this difficult work, until we prove our worth through quantifying our contribution, we should expect the bombardments to continue.
Graham Thomas - The Testers Toolbox - EuroSTAR 2010TEST Huddle
EuroSTAR Software Testing Conference 2010 presentation on The Testers Toolbox by Graham Thomas. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
Things Could Get Worse: Ideas About Regression TestingTechWell
Michael Bolton, DevelopSense
Tester, consultant, and trainer Michael Bolton is the coauthor (with James Bach) of Rapid Software Testing, a course that presents a methodology and mindset for testing software expertly in uncertain conditions and under extreme time pressure. Michael is a leader in the context-driven software testing movement with twenty years of experience testing, developing, managing, and writing about software. Currently, he leads DevelopSense, a Toronto-based consultancy.
Using Stories to Test Requirements and SystemsPaul Gerrard
The document discusses using business stories to test requirements and systems. It explains that stories can help identify omissions, inconsistencies, and ambiguity in requirements. Stories are applicable at any stage of a project for different purposes. Structured stories follow a common format with a header, scenarios with given/when/then structures, and can have multiple scenarios to test different conditions. Stories can validate requirements by example and generate both manual and automated test cases. The document argues that a structured, disciplined approach to stories can benefit both agile and structured development approaches.
Exploratory testing is an approach to testing that emphasizes the freedom and responsibility of testers to continually optimize the value of their work. It is the process of three mutually supportive activities done in parallel: learning, test design, and test execution. With skill and practice, exploratory testers typically uncover an order of magnitude more problems than when the same amount of effort is spent on procedurally scripted testing. All testers conduct exploratory testing in one way or another, but few know how to do it systematically to obtain the greatest benefits. Even fewer can articulate the process. Jon Bach looks at specific heuristics and techniques of exploratory testing that will help you get the most from this highly productive approach. Jon focuses on the skills and dynamics of exploratory testing, and how it can be combined with scripted approaches.
Exploratory testing is an approach that emphasizes freedom and responsibility of individual testers in a process where continuous learning, test design, and execution occur simultaneously. It is a disciplined, planned, and controlled form of testing that focuses on continuous learning. Research has shown there is no significant difference in results between exploratory testing and preplanned test cases, but exploratory testing requires significantly less effort overall. Effective exploratory testing requires skills like making models, keeping an open mind, and risk-based testing approaches. Both the strengths and potential blind spots of exploratory testing are discussed.
This document discusses the need to rethink the role of testers in agile and structured projects. It argues that changes in business demands and development practices are squeezing testers and that many current testing roles and skills may disappear. Specifically, it predicts that half of onshore testing roles will be eliminated in 5 years. It recommends testers focus on more strategic roles like business analysis, requirements management, and assurance rather than traditional testing tasks.
Stefaan Lukermans & Dominic Maes - Testers And Garbage Men - EuroSTAR 2011TEST Huddle
EuroSTAR Software Testing Conference 2013 presentation on Testers And Garbage Men by Stefaan Lukermans & Dominic Maes. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
The Snail Entrepreneur: The 7-year-old kid every startup should learn fromClaudio Perrone
Matteo faced a seemly impossible problem, but didn't give up. He used daddy's #PopcornFlow and pivoted. 17 options and 5 experiments later, he converged to success.
PopcornFlow is impacting businesses (large and small) but also families and kids.
If you like this story, please contribute to Matteo's cause.
The performance of your application affects your business more than you might think. Top engineering organizations think of performance as not just a nice-to-have feature but as a crucial feature of their product. Those organizations understand that performance has a direct impact on user experience and, ultimately, their bottom line. Unfortunately, most engineering teams do not regularly test the performance and scalability of their infrastructure. Jim Hirschauer shares the latest performance testing tools and insights into why your team should add performance testing to an agile development process. Learn how to evaluate performance and scalability with MultiMechanize, Bees with Machine Guns, and Google PageSpeed. Take back an understanding of how to automate performance testing and evaluate the impact it has on performance and on your business.
Application Performance Testing: A Simplified Universal ApproachTechWell
Scott Barber presented on approaches for performance testing throughout the software development lifecycle. He advocated for taking a preventative approach through continuous collaboration between developers and testers from the start of a project. This includes activities like code profiling, load testing components, and establishing performance goals at each stage with a focus on catching issues early.
A Funny Thing Happened on the Way to User Acceptance TestingTechWell
Randy Rice presented on lessons learned from user acceptance testing (UAT) on four different projects. The first project involved a new laboratory testing system that had severe performance issues and required three redeployments. The second project with the same company was more successful due to improved testing practices. The third project involved designing many tests based on business scenarios before knowing the user interface. The last project involved a complex legal system where system testing found most defects and UAT was limited due to user capabilities. Key lessons included not relying solely on UAT, having contingency plans, and flexibility in testing plans.
When implementing test automation, many people encounter problems: where to start with automation, high maintenance costs for the automated tests, or unrealistic management expectations. The good news is that solutions to these problems exist and have been effectively used by many. A “pattern” is a general reusable solution to a commonly occurring problem. Patterns have been popular in software development for many years, but they are not commonly recognized in system-level test automation. Dorothy Graham shares a collection of common problems (issues) and their solutions (patterns) which she and others are now developing as a wiki. To help resolve typical issues, Dot gives you a brief guided tour of some patterns—from Maintainable Testware and Domain-Driven Testing to Fail Gracefully and Kill the Zombies. Dot helps you recognize test automation issues and shows you how to identify appropriate patterns to help solve them.
Security Testing for Testing ProfessionalsTechWell
Today’s software applications are often security critical, making security testing an essential part of a software quality program. Unfortunately, most testers have not been taught how to effectively test the security of the software applications they validate. Join Jeff Payne as he shares what you need to know to integrate effective security testing into your everyday software testing activities. Learn how software vulnerabilities are introduced into code and exploited by hackers. Discover how to define and validate security requirements. Explore effective test techniques for assuring that common security features are tested. Learn about the most common security vulnerabilities and how to identify key security risks within applications, and use testing to mitigate them. Understand how to security test applications—both web- and GUI-based—during the software development process. Review examples of how common security testing tools work and assist the security testing process. Take home valuable tools and techniques for effectively testing the security of your applications going forward.
Anyone who has ever attempted to estimate software testing effort realizes just how difficult the task can be. The number of factors that can affect the estimate is virtually unlimited. The key to good estimates is to understand the primary variables, compare them to known standards, and normalize the estimates based on their differences. This is easy to say but difficult to accomplish because estimates are frequently required even when very little is known about the project and what is known is constantly changing. Throw in a healthy dose of politics and a bit of wishful thinking and estimation can become a nightmare. Rob Sabourin provides a foundation for anyone who must estimate software testing work effort. Learn about the test team’s and tester’s roles in estimation and measurement, and how to estimate in the face of uncertainty. Analysts, developers, leads, test managers, testers, and QA personnel can all benefit from this tutorial.
Introducing Keyword-driven Test AutomationTechWell
In both agile and traditional projects, keyword-driven testing—when done correctly—has proven to be a powerful way to attain a high level of automation. Many testing organizations use keyword-driven testing but aren't realizing the full benefits of scalability and maintainability that are essential to keep up with the demands of testing today's software. Hans Buwalda describes the keyword approach, and how you use it to can meet the very aggressive goal that he calls the "5 percent challenge"―automate 95 percent of your tests with no more than 5 percent of your total testing effort. Hans also discusses how the keyword approach relates to other automation techniques like scripting and data-driven testing, and in what way keywords can be used for specific situations like graphics, multimedia, and mobile. Use the information and the real-world examples that Hans presents to attain a very high level of maintainable automation with the lowest possible effort.
Leveraging Open Source Automation: A Selenium WebDriver ExampleTechWell
The document summarizes a case study of a company that leveraged the open source Selenium WebDriver framework for test automation. It describes the company's needs, decision to use Selenium over a commercial tool, planning and design of the Selenium framework, implementation details, outcomes of increased test coverage and reduced execution time, and conclusions about using Selenium with proper planning.
Jeroen Mengerink presented on test process improvement in Agile environments. He discussed how current TPI models focus on testing and structure but may not apply well to Agile. He proposed maturity levels for Agile testing - forming, norming, and performing. The presentation provided an assessment model to evaluate key areas like teamwork, test management, and regression testing. It offered examples and recommendations for improving processes in an Agile way, focusing on people, the development process, and testing flexibly.
Testing with Limited, Vague, and Missing RequirementsTechWell
Requirements are essential for the success of projects―or are they? As testers, we often demand concrete requirements, specified and documented in minute detail. However, does the business really know what they want early in the project? Can they actually produce such a document? Is it acceptable to test with limited or vague requirements? Lloyd Roden challenges your most basic beliefs, explaining how detailed requirements can damage and hinder the progress of testing. Lloyd provides example applications that have no requirements, vague requirements, evolving requirements, complex requirements, and detailed requirements. You will assess and test each of these examples in turn and will establish what is the best approach in these situations and for what reasons. Learn how to question applications and provide feedback on their quality using your experience and appropriate techniques regardless of the level of detail provided in the requirements.
Test Automation for Mobile Applications: A Practical GuideTechWell
The world of information technology is undergoing revolutionary changes. Advancements in mobile computing, fueled by mobile applications, are playing an important role in driving these changes. While developers build their technical skills to accommodate these evolving trends, it is equally important for testers to understand what it takes to test mobile applications. Testers must understand the scope of mobile device applications testing, whether automation is feasible, and what challenges will face the test team. Kunal Chauhan presents an optimized approach to testing smart devices, specifically focusing on mobile applications test automation, the various forms of applications (web, native, hybrid), and the tools available to assist in the automation process. Kunal demonstrates an automation framework using open source tools, providing a practical implementable solution to add to your mobile test automation toolkit.
Creating a Better Testing Future: The World Is Changing and We Must Change Wi...TechWell
The IEEE 829 Test Documentation standard is thirty years old this year. Boris Beizer’s first book on software testing also turned thirty. Testing Computer Software, the best selling book on software testing, is twenty-five. During the last three decades, hardware platforms have evolved from mainframes to minis to desktops to laptops to tablets to smartphones. Development paradigms have shifted from waterfall to agile. Consumers expect more functionality, demand higher quality, and are less loyal to brands. The world has changed dramatically and testing must change to match it. Testing processes that helped us succeed in the past may prevent our success in the future. Lee Copeland shares his insights into the future of testing, sharing his Do’s and Don’ts in the areas of technology, organization, test processes, test plans, and automation. Join Lee for a thought provoking look at creating a better testing future.
Whether you are new to testing or looking for a better way to organize your test practices and processes, the Systematic Test and Evaluation Process (STEP™) offers a flexible approach to help you and your team succeed. Dale Perry describes this risk-based framework—applicable to any development lifecycle model—to help you make critical testing decisions earlier and with more confidence. The STEP™ approach helps you decide how to focus your testing effort, what elements and areas to test, and how to organize test designs and documentation. Learn the fundamentals of test analysis and how to develop an inventory of test objectives to help prioritize your testing efforts. Discover how to translate these objectives into a concrete strategy for designing and developing tests. With a prioritized inventory and focused test architecture, you will be able to create test cases, execute the resulting tests, and accurately report on the quality of your application and the effectiveness of your testing. Take back a proven approach to organize your testing efforts and new ways to add more value to your project and organization.
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and MoreTechWell
Large-scale and complex testing projects can stress the testing and automation practices we have learned through the years, resulting in less than optimal outcomes. However, a number of innovative ideas and concepts are emerging to better support industrial-strength testing for big projects. Hans Buwalda shares his experiences and strategies he's developed for organizing and managing testing on large projects. Learn how to design tests specifically for automation, including how to incorporate keyword testing and other techniques. Learn what roles virtualization and the cloud can play, and the potential pitfalls of such options. Take away tips and tricks to make automation more stable, and to deal with the numerous versions and configurations common in large projects. Hans also describes the main challenges with global teams including time zones and cultural differences, and offers seven common problem "patterns," and what you can do to address them.
Principles Before Practices: Transform Your Testing by Understanding Key Conc...TechWell
It’s one thing to be exposed to new techniques from conferences and training courses, but it’s quite another thing to apply them in real life. A major reason is that people tend to focus on learning the technique without first grasping the underlying principles. Basic testing principles, such as the pesticide paradox of software defects and defect clustering, have been known for many years. Other principles, such as “Test automation is not automatic” and “Not every software failure is a defect,” are learned by experience. Once you grasp the principle, particular techniques become more applicable and extensible. However, principles take time to learn and much practice to apply well. Randy Rice explains why true learning and application are not instant and what it takes to really absorb what we learn. Randy shows how two specific techniques—pairwise testing and risk-based testing—can be misapplied unless the key concepts are first understood. Leave knowing how to build your own set of software testing principles that can be applied in many contexts
Testing Cloud Services: SaaS, PaaS, and IaaSTechWell
Cloud computing has changed the environment of testing. Its use is increasing for hosting business applications (SaaS) and testing (TaaS). Martin Pol and Jeroen Mengerink focus on SaaS, describing the relevant infrastructure and platform services (IaaS and PaaS). How do we test performance of the cloud itself? How do we make sure that the continuity of services is guaranteed? How do we cope with elasticity and the philosophy of bring-your-own-device (BYOD)? Martin and Jeroen discuss the risks that arise when implementing cloud computing―some traditional, but others completely new. Learn how to mitigate these risks with current, modified, and new test techniques. As testers, we must be involved earlier in the cloud selection process. Testers should help to create and evaluate selection criteria to minimize risk. In addition, testers should be involved in the project longer as testing in production is needed to determine if the Service Level Agreements are being met.
A Rapid Introduction to Rapid Software TestingTechWell
You're under tight time pressure and have barely enough information to proceed with testing. How do you test quickly and inexpensively, yet still produce informative, credible, and accountable results? Rapid Software Testing, adopted by context-driven testers worldwide, offers a field-proven answer to this all-too-common dilemma. In this one-day sampler of the approach, Michael Bolton introduces you to the skills and practice of Rapid Software Testing through stories, discussions, and "minds-on" exercises that simulate important aspects of real testing problems. The rapid approach isn't just testing with speed or a sense of urgency; it's mission-focused testing that eliminates unnecessary work, assures that the most important things get done, and constantly asks how testers can help speed up the successful completion of the project. Join Michael to see how rapid testing focuses on both the mind set and skill set of the individual tester who uses tight loops of exploration and critical thinking skills to help continuously re-optimize testing to match clients' needs and expectations.
Testing the unknown: the art and science of working with hypothesisArdita Karaj
Testing what we know, or have a clear understanding of, is relatively straight forward, as is making decisions based on the expected result. But today’s world is presenting us with the Unknown and the Ambiguous, which can only be approached by hypothesizing and experimenting - a lot! This requires intentional thinking, and a different strategy to observe in context.
This session will uncover how testers are helping their teams and product owners, by basing their testing on the science behind creating hypotheses and running experiments. A testing mindset and probing the context around use cases are some of the most valuable competencies testers bring to the team in order to enable decisions based on data.
This document discusses an introduction to a class on rapid software testing. It states that the class aims to make students stronger, smarter and more confident testers by challenging them to think for themselves rather than simply listening to what the instructors say. The class can be beneficial for testers of all experience levels who want to improve at their work. Heuristics are discussed as techniques that can help substitute for complete analysis and involve guidewords, triggers, reframing ideas, and procedures to help solve problems.
Presented at Ford's 2017 Global IT Learning Summit (GLITS)Ron Lazaro
Presentation Details: The best way to think about product discovery is to think about it in relation to product delivery. It's not possible to build a product without doing both discovery and delivery. Discovery encompasses all the activities that we do to decide what to build. It includes all the decisions we make to decide what to build next, whereas delivery is all the activities we do to write code, package releases, ship products. It's how we deliver value to our customers.
Key takeaway for the participants will be to help them understand the difference between Product Discovery and Product Delivery and how to apply techniques in doing both.
The document discusses how to achieve a happy marriage between context-driven and agile approaches to software development and testing. It advocates for involving testers from the start of projects and having them work closely with developers as part of integrated teams. The document also provides advice on skills needed for testers, such as domain knowledge and a willingness to learn, and emphasizes pairing with other roles like developers to facilitate collaboration and knowledge sharing.
Huib Schoots Testing in modern times - a story about Quality and Value - Test...FiSTB
Huib introduces himself as an experienced IT professional with over 25 years of experience in roles such as developer, tester, consultant, manager, trainer, and coach. He currently works as a managing consultant and senior consultant focused on quality and testing.
The document discusses testing and quality, noting that quality is defined by the value provided to stakeholders rather than conformance to requirements. Testing is described as evaluating a product through experience and exploration to build understanding.
It emphasizes the importance of learning for both individuals and organizations. High-performing teams and organizations are able to learn continuously and adapt their processes accordingly. Continuous learning is key for developing complex software products and keeping up with changing needs.
Michael Bolton - Two Futures of Software TestingTEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Two Futures of Software Testing by Michael Bolton. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
The document discusses the Context Driven School of Testing. It outlines the seven basic principles of the Context Driven School, which focus on the importance of context, the unpredictability of projects, solving problems rather than following practices, testing as an intellectual process, and cooperation throughout projects. It also briefly describes five schools of testing - Analytical, Factory, Quality Assurance, Context Driven, and Agile - and notes that the Context Driven community values challenges and debate. The document emphasizes finding community and understanding different perspectives to improve testing practices.
- The document discusses the growth of the QA team at North from 1 person to 18 full-time employees and 2 co-ops over 3 years. It describes challenges faced such as hiring candidates and establishing processes as the team grew rapidly. Lessons learned include starting with basic tools, focusing on lightweight processes, and tailoring interviews and challenges to the role. The future includes expanding test automation and driving quality practices earlier in development.
Вадим Давидов та Людмила Гребенюк “LEAN: Dream Maker Developments” Kharkiv Pr...Lviv Startup Club
1. Agile project delivery focuses on four key values: people over processes, working prototypes over excessive documentation, customer collaboration over rigid contracts, and responding to change over following a plan.
2. Lean principles for software development include eliminating waste, amplifying learning, deciding as late as possible, and delivering as fast as possible. This emphasizes continuous improvement through building, measuring, and learning feedback loops.
3. The seven principles of lean software development are: eliminate waste, amplify learning, decide as late as possible, deliver as fast as possible, empower the team, build quality in, and see the whole system and how the parts fit together.
1. Agile project delivery focuses on four key values: people over processes, working prototypes over excessive documentation, customer collaboration over rigid contracts, and responding to change over following a plan.
2. Lean principles for software development include eliminating waste, amplifying learning, deciding as late as possible, and delivering as fast as possible. This emphasizes continuous improvement through building, measuring, and learning quickly.
3. The seven principles of lean software development are: eliminate waste, amplify learning, decide as late as possible, deliver as fast as possible, empower the team, build quality in, and see the whole. This advocates for continuous flow, learning, quality, and empowerment.
This document provides an overview of Agile software development. It begins by defining Agile as a project management process that encourages frequent inspection and adaptation. It then discusses some common Agile practices like Scrum and eXtreme Programming. The Agile Manifesto values individuals and interactions, working software, customer collaboration, and responding to change. Finally, it provides advice for different roles on how Agile can benefit them and their work.
Perhaps in no other professional field is the dichotomy between theory and practice more starkly different than in the realm of software testing. Researchers and thought leaders claim that testing requires a high level of cognitive and interpersonal skills, in order to make judgments about the ability of software to fulfill its operational goals. In their minds, testing is about assessing and communicating the risks involved in deploying software in a specific state.
However, in many organizations, testing remains a necessary evil, and a cost to drive down as much as possible. Testing is merely a measure of conformance to requirements, without regard to the quality of requirements or how conformance is measured. This is certainly an important measure, but tells an incomplete story about the value of software in support of our business goals.
We as testers often help to perpetuate the status quo. Although in many cases we realize we can add far more value than we do, we continue to perform testing in a manner that reduces our value in the software development process.
This presentation looks at the state of the art as well of the state of common practice, and attempts to provide a rationale and roadmap whereby the practice of testing can be made more exciting and stimulating to the testing professional, as well as more valuable to the product and the organization.
Exploratory testing is an approach to testing that emphasizes the freedom and responsibility of testers to continually optimize the value of their work. It is the process of three mutually supportive activities done in parallel: learning, test design, and test execution. With skill and practice, exploratory testers typically uncover an order of magnitude more problems than when the same amount of effort is spent on procedurally scripted testing. All testers conduct exploratory testing in one way or another, but few know how to do it systematically to obtain the greatest benefits. Even fewer can articulate the process. James Bach looks at specific heuristics and techniques of exploratory testing that will help you get the most from this highly productive approach. James focuses on the skills and dynamics of exploratory testing, and how it can be combined with scripted approaches.
The document discusses agile testing and bug prevention. It advocates for embedding testers within development teams to focus on prevention rather than detection of bugs. The ideal approach involves continuous testing parallel to development with the entire team involved in testing.
A test strategy is the set of ideas that guides your test design. It's what explains why you test this instead of that, and why you test this way instead of that way. Strategic thinking matters because testers must make quick decisions about what needs testing right now and what can be left alone. You must be able to work through major threads without being overwhelmed by tiny details. James Bach describes how test strategy is organized around risk but is not defined before testing begins. Rather, it evolves alongside testing as we learn more about the product. We start with a vague idea of our strategy, organize it quickly, and document as needed in a concise way. In the end, the strategy can be as formal and detailed as you want it to be. In the beginning, though, we start small. If you want to focus on testing and not paperwork, this approach is for you.
Dallas Education ISO 9001:2008,20000-2005,27001:2013 Certified, based at Bangalore India, Providing services in software consulting, application development, outsourcing services, Recruitment and Training. Started operation in the year 2001. We design, build, and support customized applications for businesses large and small. We are the market leader in training and outsourcing in various technologies. We ,Dallas Education, serve and support IT companies in the areas of Mainframes, ERP, .net, Java/J2EE,Data Warehousing and Business Intelligence, etc. we also train and outsource fresh talents to our clients.
Discovery is an iterative process of reducing uncertainty. It's an essential part of how we do Product Development; routinely involving our customers in the act of deciding what we build, before we build it. Without discovery we increase the risk of building solutions that our customer won’t want, use or value. With discovery we maximise our chances of investing in ideas that are likely to succeed.
Similar to A Rapid Introduction to Rapid Software Testing (20)
Isabel Evans stopped drawing and painting after being told she was not very good at it, which led to a loss of confidence in her creative and professional abilities. However, she realized that attempting creative activities is important for cognitive and emotional development, and that making mistakes and learning from failures allows for growth. By reengaging with failure through art and with support from others, Isabel was able to regain confidence in her abilities and reboot her career. The document discusses different perspectives on failure and the importance of learning from mistakes.
Instill a DevOps Testing Culture in Your Team and Organization TechWell
The DevOps movement is here. Companies across many industries are breaking down siloed IT departments and federating them into product development teams. Testing and its practices are at the heart of these changes. Traditionally, IT organizations have been staffed with mostly manual testers and a limited number of automation and performance engineers. To keep pace with development in the new “you build it, you own it” environment, testing teams and individuals must develop new technical skills and even embrace coding to stay relevant and add greater value to the business. DevOps really starts with testing. Join Adam Auerbach as he explains what DevOps is and how it relates to testing. He describes how testing must change from top to bottom and how to access your own environment to identify improvement opportunities. Adam dives into practices like service virtualization, test data management, and continuous testing so you can understand where you are now and identify steps needed to instill a DevOps testing culture in your team and organization.
Test Design for Fully Automated Build ArchitectureTechWell
This document summarizes a half-day tutorial on test design for fully automated build architectures presented by Melissa Benua of mParticle at STAREAST 2018. The tutorial covered guiding principles for test design including prioritizing important and reliable tests, structuring automated pipelines around components, packages, and releases, and monitoring test results through code coverage, flaky test handling, and logging versus counters. It also included exercises mapping test cases to functional boundaries and categories of tests to pipeline stages.
System-Level Test Automation: Ensuring a Good StartTechWell
Many organizations invest a lot of effort in test automation at the system level but then have serious problems later on. As a leader, how can you ensure that your new automation efforts will get off to a good start? What can you do to ensure that your automation work provides continuing value? This tutorial covers both “theory” and “practice”. Dot Graham explains the critical issues for getting a good start, and Chris Loder describes his experiences in getting good automation started at a number of companies. The tutorial covers the most important management issues you must address for test automation success, particularly when you are new to automation, and how to choose the best approaches for your organization—no matter which automation tools you use. Focusing on system level testing, Dot and Chris explain how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts to promote success, what you can realistically expect in benefits and how to report them. They explain—for non-techies—the key technical issues that can make or break your automation effort. Come away with your own clarified automation objectives, and a draft test automation strategy to use to plan your own system-level test automation.
Build Your Mobile App Quality and Test StrategyTechWell
Let’s build a mobile app quality and testing strategy together. Whether you have a web, hybrid, or native app, building a quality and testing strategy means (1) knowing what data and tools you have available to make agile decisions, (2) understanding your customers and your competitors, and (3) testing your app under real-world conditions. Jason Arbon guides you through the latest techniques, data, and tools to ensure the awesomeness of your mobile app quality and testing strategy. Leave this interactive session with a strategy for your very own app—or one you pretend to own. The information Jason shares is based on data from Appdiff’s next-gen mobile app testing platform, lessons from Applause/uTest’s crowd, text mining hundreds of millions of app store reviews, and in-depth discussions with top mobile app development teams.
Testing Transformation: The Art and Science for SuccessTechWell
Technologies, testing processes, and the role of the tester have evolved significantly in the past few years with the advent of agile, DevOps, and other new technologies. It is critical that we testing professionals evaluate ourselves and continue to add tangible value to our organizations. In your work, are you focused on the trivial or on real game changers? Jennifer Bonine describes critical elements that help you artfully blend people, process, and technology to create a synergistic relationship that adds value. Jennifer shares ideas on mastering politics, maneuvering core vs. context, and innovating your technology strategies and processes. She explores how new processes can be introduced in an organization, what the role of organizational culture is in determining the success of a project, and how you can know what tools will add value vs. simply adding overhead and complexity. Jennifer reviews critically needed tester skills and discusses a continual learning model to evolve your skills and stay relevant. This discussion can lead you to technologies, processes, and skills you can stake your career on.
We’ve all been there. We work incredibly hard to develop a feature and design tests based on written requirements. We build a detailed test plan that aligns the tests with the software and the documented business needs. And when we put the tests to the software, it all falls apart because the requirements were changed without informing everyone. Mary Thorn says help is at hand. Enter behavior-driven development (BDD), and Cucumber and SpecFlow, tools for running automated acceptance tests and facilitating BDD. Mary explores the nuances of Cucumber and SpecFlow, and shows you how to implement BDD and agile acceptance testing. By fostering collaboration for implementing active requirements via a common language and format, Cucumber and SpecFlow bridge the communication gap between business stakeholders and implementation teams. In this workshop, practice writing feature files with the best practices Mary has discovered over numerous implementations. If you experience developers not coding to requirements, testers not getting requirements updates, or customers who feel out of the loop and don’t get what they ask for, Mary has answers for you.
Develop WebDriver Automated Tests—and Keep Your SanityTechWell
Many teams go crazy because of brittle, high-maintenance automated test suites. Jim Holmes helps you understand how to create a flexible, maintainable, high-value suite of functional tests using Selenium WebDriver. Learn the basics of what to test, what not to test, and how to avoid overlapping with other types of testing. Jim includes both philosophical concepts and hands-on coding. Testers who haven't written code should not be intimidated! We'll pair you up to make sure you're successful. Learn to create practical tests dealing with advanced situations such as input validation, AJAX delays, and working with file downloads. Additionally, discover when you need to work together with developers to create a system that's more easily testable. This tutorial focuses primarily on automating web tests, but many of the same concepts can be applied to other UI environments. Demos and labs will be in C# and Java using WebDriver. Leave this tutorial having learned how to write high-value WebDriver tests—and stay sane while doing so.
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Eliminate Cloud Waste with a Holistic DevOps StrategyTechWell
Chris Parlette maintains that renting infrastructure on demand is the most disruptive trend in IT in decades. In 2016, enterprises spent $23B on public cloud IaaS services. By 2020, that figure is expected to reach $65B. The public cloud is now used like a utility, and like any utility, there is waste. Who's responsible for optimizing the infrastructure and reducing wasted expenses? It’s DevOps. The excess expense, known as cloud waste, comprises several interrelated problems: services running when they don't need to be, improperly sized infrastructure, orphaned resources, and shadow IT. There are a few core tenets of DevOps—holistic thinking, no silos, rapid useful feedback, and automation—that can be applied to reducing your cloud waste. Join Chris to learn why you should include continuous cost optimization in your DevOps processes. Automate cost control, reduce your cloud expenses, and make your life easier.
Transform Test Organizations for the New World of DevOpsTechWell
With the recent emergence of DevOps across the industry, testing organizations are being challenged to transform themselves significantly within a short period of time to stay meaningful within their organizations. It’s not easy to plan and approach these changes considering the way testing organizations have remained structured for ages. These challenges start from foundational organizational structures and can cut across leadership influence, competencies, tools strategy, infrastructure, and other dimensions. Sumit Kumar shares his experience assisting various organizations to overcome these challenges using an organized DevOps enablement framework. The framework includes radical restructuring, turning the tools strategy upside down, a multidimensional workforce enablement supported by infrastructure changes, redeveloped collaborations models, and more. From his real world experiences Sumit shares tips for approaching this journey and explains the roadmap for testing organizations to transform themselves to lead the quality in DevOps.
The Fourth Constraint in Project Delivery—LeadershipTechWell
All too often, the triple constraints—time, cost, and quality—are bandied about as if they are the be-all, end-all. While they are important, leadership—the fourth and larger underpinning constraint—influences the first three. Statistics on project success and failure abound, and these measurements are usually taken against the triple constraints. According to the Project Management Institute, only 53 percent of projects are completed within budget, and only 49 percent are completed on time. If so many projects overrun budget and are late, we can’t really say, “Good, fast, or cheap—pick two.” Rob Burkett talks about leadership at every level of a team. He shares his insights and stories gleaned from his years of IT and project management experience. Rob speaks to some of the glaring difficulties in the workplace in general and some specifically related to IT delivery and project management. Leave with a clearer understanding of how to communicate with teams and team members, and gain a better understanding of how you can be a leader—up and down your organization.
Resolve the Contradiction of Specialists within Agile TeamsTechWell
As teams grow, organizations often draw a distinction between feature teams, which deliver the visible business value to the user, and component teams, which manage shared work. Steve Berczuk says that this distinction can help organizations be more productive and scale effectively, but he recognizes that not all shared work fits into this model. Some work is best handled by “specialists,” that is people with unique skills. Although teams composed entirely of T-shaped people is ideal, certain skills are hard to come by and are used irregularly across an organization. Since these specialists often need to work closely with teams, rather than working from their own backlog, they don’t fit into the component team model. The use of shared resources presents challenges to the agile planning model. Steve Berczuk shares how teams such as those providing infrastructure services and specialists can fit into a feature+component team model, and how variations such as embedding specialists in a scrum team can both present process challenges and add significant value to both the team and the larger organization.
Pin the Tail on the Metric: A Field-Tested Agile GameTechWell
Metrics don’t have to be a necessary evil. If done right, metrics can help guide us to make better forward-looking decisions, rather than being used for simply managing or monitoring. They can help us identify trade-offs between options for what to do next versus punitive or worse, purely managerial measures. Steve Martin won’t be giving the Top Ten List of field-tested metrics you should use. Instead, in this interactive mini-workshop, he leads you through the critical thinking necessary for you to determine what is right for you to measure. First, Steve explores why you want to measure something—whether it’s for a team, a portfolio, or even an agile transformation. Next, he provides multiple real-life metrics examples to help drive home concepts behind characteristics of good and bad metrics. Finally, Steve shows how to run his field-tested agile game—Pin the Tail on the Metric. Take back this activity to help you guide metrics conversations at your organization.
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsTechWell
A hierarchy is an organizational network that has a top and a bottom, and where position is determined by rank, importance, and value. A holarchy is a network that has no top or bottom and where each person’s value derives from his ability, rather than position. As more companies seek the benefits of agile, leaders need to build and sustain delivery capability while scaling agile without introducing unnecessary process and overhead. The Agile Performance Holarchy (APH) is an empirical model for scaling and sustaining agility while continuing to deliver great products. Jeff Dalton designed the APH by drawing from lessons learned observing and assessing hundreds of agile companies and teams. The APH helps implement a holarchy—a system composed of interacting organizational units called holons—centered on a series of performance circles that embody the behaviors of high performing agile organizations. Jeff describes how APH provides guidelines in the areas of leadership, values, teaming, visioning, governing, building, supporting, and engaging within an all-agile organization. Join Jeff to see what the APH is all about and how you can use it in your team and organization.
A Business-First Approach to DevOps ImplementationTechWell
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Databases in a Continuous Integration/Delivery ProcessTechWell
The document summarizes a presentation about including databases in a continuous integration/delivery process. It discusses treating database code like application code by placing it under version control and integrating databases into the DevOps software development pipeline. This allows databases to be built, tested, and released like other software through continuous integration, delivery, and deployment.
Mobile Testing: What—and What Not—to AutomateTechWell
Organizations are moving rapidly into mobile technology, which has significantly increased the demand for testing of mobile applications. David Dangs says testers naturally are turning to automation to help ease the workload, increase potential test coverage, and improve testing efficiency. But should you try to automate all things mobile? Unfortunately, the answer is not always clear. Mobile has its own set of complications, compounded by a wide variety of devices and OS platforms. Join David to learn what mobile testing activities are ripe for automation—and those items best left to manual efforts. He describes the various considerations for automating each type of mobile application: mobile web, native app, and hybrid applications. David also covers device-level testing, types of testing, available automation tools, and recommendations for automation effectiveness. Finally, based on his years of mobile testing experience, David provides some tips and tricks to approach mobile automation. Leave with a clear plan for automating your mobile applications.
Cultural Intelligence: A Key Skill for SuccessTechWell
Diversity is becoming the norm in everyday life. However, introducing global delivery models without a proper understanding of intercultural differences can lead to difficulty, frustration, and reduced productivity. Priyanka Sharma and Thena Barry say that in our diverse world, we need teams with people who can cross these boundaries, communicate effectively, and build the diverse networks necessary to avoid problems. We need to learn about cultural intelligence (CI) and cultural quotient (CQ). CI is the ability to relate and work effectively across cultures. CQ is the cognitive, motivational, and behavioral capacity to understand and respond to beliefs, values, attitudes, and behaviors of individuals and groups. Together, CI and CQ can help us build behavioral capacities that aid motivation, behavior, and productivity in teams as well as individuals. Priyanka and Thena show how to build a more culturally intelligent place with tools and techniques from Leading with Cultural Intelligence, as well as content from the Hofstede cultural model. In addition, they illustrate the model with real-life experiences and demonstrate how they adapted in similar circumstances.
Turn the Lights On: A Power Utility Company's Agile TransformationTechWell
Why would a century-old utility with no direct competitors take on the challenge of transforming its entire IT application organization to an agile methodology? In an increasingly interconnected world, the expectations of customers continue to evolve. From smart meters to smart phones, IoT is creating a crisis point for industries not accustomed to rapid change. Glen Morris explains that pizzas can be tracked by the minute and packages at every stop, and customers now expect this same customer service model should exist for all industries—including power. Glen examines how to create momentum and transform non-IT-focused industries to an agile model. If you are struggling with gaining traction in your pursuit of agile within your business, Glen gives you concrete, practical experiences to leverage in your pursuit. Finally, he communicates how to gain buy-in from business partners who have no idea or concern about agile or its methodologies. If your business partners look at you with amusement when you mention the need for a dedicated Product Owner, join Glen as he walks you through the approaches to overcoming agile skepticism.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor IvaniukFwdays
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.
"Choosing proper type of scaling", Olena SyrotaFwdays
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...Alex Pruden
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Generating privacy-protected synthetic data using Secludy and MilvusZilliz
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providersakankshawande
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Taking AI to the Next Level in Manufacturing.pdfssuserfac0301
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
What is an RPA CoE? Session 1 – CoE VisionDianaGray10
In the first session, we will review the organization's vision and how this has an impact on the COE Structure.
Topics covered:
• The role of a steering committee
• How do the organization’s priorities determine CoE Structure?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
5th LF Energy Power Grid Model Meet-up SlidesDanBrown980551
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
1. MA
Full-day Tutorials
5/5/2014 8:30:00 AM
A Rapid Introduction to
Rapid Software Testing
Presented by:
Michael Bolton
DevelopSense
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
2. Michael Bolton
DevelopSense
Tester, consultant, and trainer Michael Bolton is the co-author (with James Bach) of Rapid
Software Testing, a course that presents a methodology and mindset for testing software
expertly in uncertain conditions and under extreme time pressure. Michael is a leader in the
context-driven software testing movement, with twenty years of experience testing,
developing, managing, and writing about software. Currently, he leads DevelopSense, a
Toronto-based consultancy. Prior to DevelopSense, he was with Quarterdeck Corporation,
where he managed the company’s flagship products and directed project and testing
teams—both in-house and worldwide. Contact Michael at michael@developsense.com.
3. 1
A Rapid Introduction
to Rapid Software Testing
James Bach, Satisfice, Inc.
james@satisfice.com
www.satisfice.com
+1 (360) 440-1435
Michael Bolton, DevelopSense
mb@developsense.com
www.developsense.com
+1 (416) 656-5160
2
Acknowledgements
• Some of this material was developed in collaboration with
Dr. Cem Kaner, of the Florida Institute of Technology. See
www.kaner.com and www.testingeducation.org.
• Doug Hoffman (www.softwarequalitymethods.com) has also
contributed to and occasionally teaches from this material.
• Many of the ideas in this presentation were also inspired by or
augmented by other colleagues including Jonathan Bach, Bret
Pettichord, Brian Marick, Dave Gelperin, Elisabeth Hendrickson, Jerry
Weinberg, Noel Nyman, and Mary Alton.
• Some of our exercises were introduced to us by Payson Hall, Jerry
Weinberg, Ross Collard, James Lyndsay, Dave Smith, Earl Everett,
Brian Marick, Cem Kaner and Joe McMahon.
• Many ideas were improved by students who took earlier versions of
the class going back to 1996.
4. 2
3
Assumptions About You
• You test software, or any other complex human
creation.
• You have at least some control over the design of your
tests and some time to create new tests.
• You are worried that your test process is spending too
much time and resources on things that aren’t
important.
• You test under uncertainty and time pressure.
• Your major goal is to find important problems quickly.
• You want to get very good at (software) testing.
A Question
What makes
testing harder or
slower?
5. 3
Premises of Rapid Testing
1. Software projects and products are relationships
between people.
2. Each project occurs under conditions of uncertainty
and time pressure.
3. Despite our best hopes and intentions, some
degree of inexperience, carelessness, and
incompetence is normal.
4. A test is an activity; it is performance, not artifacts.
Premises of Rapid Testing
5. Testing’s purpose is to discover the status of the
product and any threats to its value, so that our
clients can make informed decisions about it.
6. We commit to performing credible, cost-effective
testing, and we will inform our clients of anything that
threatens that commitment.
7. We will not knowingly or negligently mislead our
clients and colleagues or ourselves.
8. Testers accept responsibility for the quality of their
work, although they cannot control the quality of the
product.
6. 4
7
Rapid Testing
Rapid testing is a mind-set
and a skill-set of testing
focused on how to do testing
more quickly,
less expensively,
with excellent results.
This is a general testing
methodology. It adapts to
any kind of project or product.
8
Rapid testing may not
be exhaustive, but it is
thorough enough and
quick enough. It’s less
work than ponderous
testing. It might be less
work than slapdash
testing.
It fulfills the mission
of testing.
How does Rapid Testing compare
with other kinds of testing?
Management likes to talk
about exhaustive testing, but
they don’t want to fund it and
they don’t know how to do it.
You can always test
quickly...
But it might be poor
testing.
When testing is turned into an
elaborate set of rote tasks,
it becomes ponderous without
really being thorough.
MoreWork&Time
(Cost)
Better Thinking & Better Testing
(Value)
Slapdash
Much faster, cheaper,
and easier
Ponderous
Slow, expensive,
and easier
Rapid
Faster, less expensive,
still challenging
Exhaustive
Slow, very expensive,
and difficult
7. 5
9
Excellent Rapid Technical Work
Begins with You
When the ball comes to you…
Do you know you have the ball?
Can you receive the pass?
Do you know what your
role and mission is?
Do you know where
your teammates are?
Are you ready to act, right now?
Can you let your teammates help you?
Do you know your options?
Is your equipment ready?
Can you read the
situation on the field?
Are you aware of the
criticality of the situation?
10
• Rapid test teams are about diverse talents cooperating
• We call this the elliptical team, as opposed to the team of
perfect circles.
• Some important dimensions to vary:
• Technical skill
• Domain expertise
• Temperament (e.g. introvert vs. extrovert)
• Testing experience
• Project experience
• Industry experience
• Product knowledge
• Educational background
• Writing skill
• Diversity makes exploration far more powerful
• Your team is powerful because of your unique contribution
…but you don’t have to be great at everything.
8. 6
What It Means To Test Rapidly
• Since testing is about finding a potentially infinite
number of problems in an infinite space in a finite
amount of time, we must…
• understand our mission and obstacles to fulfilling it
• know how to recognize problems quickly
• model the product and the test space to know where to look
for problems
• prefer inexpensive, lightweight, effective tools
• reduce dependence on expensive, time-consuming artifacts,
while getting value from the ones we’ve got
• do nothing that wastes time or effort
• tell a credible story about all that
11
One Big Problem in Testing
Formality Bloat
• Much of the time, your testing doesn’t need to be very formal*
• Even when your testing does need to be formal, you’ll need to
do substantial amounts of informal testing in order figure out
how to do excellent formal testing.
• Who says? The FDA. See http://www.satisfice.com/blog/archives/602
• Even in a highly regulated environment, you do formal testing primarily
for the auditors. You do informal testing to make sure you don’t
lose money, blow things up, or kill people.
* Formal testing means testing that must be done to verify a specific fact,
or that must be done in a specific way.
9. 7
EXERCISE
Test the Famous Triangle
14
What is testing?
Serving Your Client
If you don’t have an understanding and an agreement
on what is the mission of your testing, then doing it
“rapidly” would be pointless.
10. 8
Not Enough
Product and Project Information?
Where do we get
test information?
What Is A Problem?
A problem is…
11. 9
How Do We Recognize Problems?
An oracle is…
a way to recognize
a problem.
Learn About Heuristics
Heuristics are fallible, “fast and frugal” methods of solving
problems, making decisions, or accomplishing tasks.
“The engineering method is
the use of heuristics
to cause the best change
in a poorly understood situation
within the available resources.”
Billy Vaughan Koen
Discussion of the Method
12. 10
Heuristics: Generating Solutions
Quickly and Inexpensively
• Heuristic (adjective):
serving to discover or learn
• Heuristic (noun):
a fallible method for solving a problem
or making a decision
“Heuristic reasoning is not regarded as final and strict
but as provisional and plausible only, whose purpose
is to discover the solution to the present problem.”
- George Polya, How to Solve It
Oracles
An oracle is a heuristic principle or mechanism
by which we recognize a problem.
“...it appeared at least once to meet some
requirement to some degree”
“...uh, when I ran it”
“...that one time”
“...on my machine.”
“It works!”
really means…
13. 11
Familiar Problems
If a product is consistent with problems we’ve seen before,
we suspect that there might be a problem.
Explainability
If a product is inconsistent with our ability to explain it
(or someone else’s), we suspect that there might be a problem.
14. 12
World
If a product is inconsistent with the way the world works,
we suspect that there might be a problem.
History
If a product is inconsistent with previous versions of itself,
we suspect that there might be a problem.
Okay,
so how the
#&@ do I print
now?
15. 13
Image
If a product is inconsistent with an image that
the company wants to project, we suspect a problem.
Comparable Products
WordPad Word
When a product seems inconsistent with a product that is
in some way comparable, we suspect that there might be a problem.
16. 14
Claims
When a product is inconsistent with claims that important
people make about it, we suspect a problem.
User Expectations
When a product is inconsistent with expectations that a
reasonable user might have, we suspect a problem.
17. 15
Purpose
When a product is inconsistent with its designers’ explicit
or implicit purposes, we suspect a problem.
Product
When a product is inconsistent internally—as when it
contradicts itself—we suspect a problem.
18. 16
Statutes and Standards
When a product is inconsistent with laws or widely
accepted standards, we suspect a problem.
32
Consistency (“this agrees with that”)
an important theme in oracle principles
• Familiarity: The system is not consistent with the pattern of any familiar problem.
• Explainability: The system is consistent with our ability to describe it clearly.
• World: The system is consistent with things that we recognize in the world.
• History: The present version of the system is consistent with past versions of it.
• Image: The system is consistent with an image that the organization wants to project.
• Comparable Products: The system is consistent with comparable systems.
• Claims: The system is consistent with what important people say it’s supposed to be.
• Users’ Expectations: The system is consistent with what users want.
• Product: Each element of the system is consistent with comparable elements in the
same system.
• Purpose: The system is consistent with its purposes, both explicit and implicit.
• Standards and Statutes: The system is consistent with applicable laws, or relevant
implicit or explicit standards.
Consistency heuristics rely on the quality of your
models of the product and its context.
19. 17
Consistency heuristics rely on the quality of
your models of the product and its context.
An oracle doesn’t tell you that there IS a problem.
An oracle tells you that you might be seeing a problem.
Rely solely on documented, anticipated sources of oracles,
and your testing will likely be slower and weaker.
Train your mind to recognize patterns of oracles
and your testing will likely be faster
and your ability to spot problems will be sharper.
All Oracles Are Heuristic
An oracle can alert you to a possible problem,
but an oracle cannot tell you that there is no problem.
• A person whose opinion matters.
• An opinion held by a person who matters.
• A disagreement among people who matter.
• A reference document with useful information.
• A known good example output.
• A known bad example output.
• A process or tool by which the output is checked.
• A process or tool that helps a tester identify patterns.
• A feeling like confusion or annoyance.
• A desirable consistency between related things.
General Examples of Oracles
things that suggest “problem” or “no problem”
34
People
Mechanisms
Feelings
Principles
20. 18
Tacit Explicit
OtherPeopleTester
Your
Feelings &
Mental Models
Shared Artifacts
(specs, tools, etc.)
Stakeholders’
Feelings &
Mental Models
Inference
Observable
Consistencies
ReferenceConference
Experience
Oracles from the Inside Out
Oracle Cost and Value
• Some oracles are more authoritative
• but more responsive to change
• Some oracles are more consistent
• but maybe not up to date
• Some oracles are more immediate
• but less reliable
• Some oracles are more precise
• but the precision may be misleading
• Some oracles are more accurate
• but less precise
• Some oracles are more available
• but less authoritative
• Some oracles are easier to interpret
• but more narrowly focused
21. 19
Feelings As Heuristic Triggers For Oracles
• An emotional reaction is a trigger to attention
and learning
• Without emotion, we don’t reason well
• See Damasio, The Feeling of What Happens
• When you find yourself mildly concerned about
something, someone else could be very
concerned about it
• Observe emotions to help overcome your
biases and to evaluate significance
An emotion is a signal; consider looking into it
38
All Oracles Are Heuristic
• We often do not have oracles that establish a definite correct or incorrect
result, in advance. Oracles may reveal themselves to us on the fly, or later.
That’s why we use abductive inference.
• No single oracle can tell us whether a program (or a feature) is working
correctly at all times and in all circumstances.
That’s why we use a variety of oracles.
• Any program that looks like it’s working, to you, may in fact be failing in
some way that happens to fool all of your oracles. That’s why we proceed
with humility and critical thinking.
• We never know when a test is finished.
That’s why we try to maintain uncertainty when everyone else on the
project is sure.
• You (the tester) can’t know the deep truth about any result.
That’s why we report whatever seems likely to be a bug.
22. 20
39
Oracles are Not Perfect
And Testers are Not Judges
• You don’t need to know for sure if something is a bug;
it’s not your job to decide if something is a bug; it’s your
job to decide if it’s worth reporting.
• You do need to form a justified belief that it MIGHT be
a threat to product value in the opinion of someone
who matters.
• And you must be able to say why you think so; you must
be able to cite good oracles… or you will lose credibility.
MIP’ing VS. Black Flagging
40
Coping With Difficult Oracle Problems
• Ignore the Problem
• Ask “so what?” Maybe the value of the information doesn’t justify the cost.
• Simplify the Problem
• Ask for testability. It usually doesn’t happen by accident.
• Built-in oracle. Internal error detection and handling.
• Lower the standards. You may be using an unreasonable standard of correctness.
• Shift the Problem
• Parallel testing. Compare with another instance of a comparable algorithm.
• Live oracle. Find an expert who can tell if the output is correct.
• Reverse the function. (e.g. 2 x 2 = 4, then 4/2 = 2)
• Divide and Conquer the Problem
• Spot check. Perform a detailed inspection on one instance out of a set of outputs.
• Blink test. Compare or review overwhelming batches of data for patterns that stand
out.
• Easy input. Use input for which the output is easy to analyze.
• Easy output. Some output may be obviously wrong, regardless of input.
• Unit test first. Learn about the pieces that make the whole.
• Test incrementally. Learn about the product by testing over a period of time.
23. 21
41
“Easy Input”
• Fixed Markers. Use distinctive fixed input patterns that are easy to
spot in the output.
• Statistical Markers. Use populations of data that have
distinguishable statistical properties.
• Self-Referential Data. Use data that embeds metadata about itself.
(e.g. counterstrings)
• Easy Input Regions. For specific inputs, the correct output may be
easy to calculate.
• Outrageous Values. For some inputs, we expect error handling.
• Idempotent Input. Try a case where the output will be the same as
the input.
• Match. Do the “same thing” twice and look for a match.
• Progressive Mismatch. Do progressively differing things over time
and account for each difference. (code-breaking technique)
42
Oracles Are Linked To Threats
To Quality Criteria
Any inconsistency may represent diminished value.
Many test approaches focus on capability (functionality)
and underemphasize the other criteria.
Capability Scalability
Reliability Compatibility
Usability Performance
Charisma Installability
Security Development
24. 22
43
Oracles Are Linked To Threats
To Quality Criteria
Any inconsistency may represent diminished value.
Many test approaches focus on capability (functionality)
and underemphasize the other criteria.
Supportability
Testability
Maintainability
Portability
Localization
Focusing on Preparation and Skill
Can Reduce Documentation Bloat
3.0 Test Procedures
3.1 General testing protocol.
• In the test descriptions that follow, the word “verify" is used to highlight specific items
that must be checked. In addition to those items a tester shall, at all times, be alert for
any unexplained or erroneous behavior of the product. The tester shall bear in mind
that, regardless of any specific requirements for any specific test, there is the
overarching general requirement that the product shall not pose an unacceptable risk
of harm to the patient, including an unacceptable risk using reasonably foreseeable
misuse.
• Test personnel requirements: The tester shall be thoroughly familiar with the
generator and workstation FRS, as well as with the working principles of the devices
themselves. The tester shall also know the working principles of the power test jig and
associated software, including how to configure and calibrate it and how to recognize if
it is not working correctly. The tester shall have sufficient skill in data analysis and
measurement theory to make sense of statistical test results. The tester shall be
sufficiently familiar with test design to complement this protocol with exploratory
testing, in the event that anomalies appear that require investigation. The tester shall
know how to keep test records to credible, professional standard.
25. 23
Remember…
For skilled testers,
good testing isn’t just about
pass vs. fail.
For skilled testers,
testing is about
problem vs. no problem.
Where Do We Look For Problems?
Coverage is…
how much of the
product has been tested.
26. 24
Coverage is “how much of the product we have tested.”
What IS Coverage?
It’s the extent to which we have
traveled over some map of the product.
MODELS
Models
• A model is an idea, activity, or object…
such as an idea in your mind, a diagram, a list of words, a spreadsheet,
a person, a toy, an equation, a demonstration, or a program
such as something complex that you need to work with or study
- A map is a model that helps to navigate across a terrain.
- 2+2=4 is a model for adding two apples to a basket that already has two apples.
- Atmospheric models help predict where hurricanes will go.
- A fashion model helps understand how clothing would look on actual humans.
- Your beliefs about what you test are a model of what you test.
• …that heuristically represents (literally,
re-presents) another idea, activity, or object…
• …whereby understanding something about the
model may help you to understand or manipulate
the thing that it represents.
27. 25
There are as many kinds of test coverage as there are
ways to model the system.
Intentionally OR Incidentally
One Way to Model Coverage:
Product Elements (with Quality Criteria)
Capability
Reliability
Usability
Charisma
Security
Scalability
Compatibility
Performance
Installability
Supportability
Testability
Maintainability
• Structure
• Function
• Data
• Interfaces
• Platform
• Operations
• Time
28. 26
51
To test a very simple product meticulously,
part of a complex product meticulously,
or to maximize test integrity…
1. Start the test from a known (clean) state.
2. Prefer simple, deterministic actions.
3. Trace test steps to a specified model.
4. Follow established and consistent lab procedures.
5. Make specific predictions, observations and records.
6. Make it easy to reproduce (automation may help).
General Focusing Heuristics
• use test-first approach or unit testing for better code
coverage
• work from prepared test coverage outlines and risk lists
• use diagrams, state models, and the like, and cover them
• apply specific test techniques to address particular coverage
areas
• make careful observations and match to expectations
To do this more rapidly, make preparation and artifacts fast and frugal:
leverage existing materials and avoid repeating yourself.
Emphasize doing; relax planning. You’ll make discoveries along the way!
29. 27
53
To find unexpected problems,
elusive problems that occur in sustained field use,
or more problems quickly in a complex product…
1. Start from different states (not necessarily clean).
2. Prefer complex, challenging actions.
3. Generate tests from a variety of models.
4. Question your lab procedures and tools.
5. Try to see everything with open expectations.
6. Make the test hard to pass, instead of easy to reproduce.
That’s a
PowerPoint
bug!
General Defocusing Heuristics
• diversify your models; intentional coverage in one area can lead
to unintentional coverage in other areas—this is a Good Thing
• diversify your test techniques
• be alert to problems other than the ones that you’re actively
looking for
• welcome and embrace productive distraction
• do some testing that is not oriented towards a specific risk
• use high-volume, randomized automated tests
30. 28
DISCUSSION
How Many Test Cases?
What About Quantifying Coverage Overall?
• A nice idea, but we don’t know how to do it in a way
that is consistent with basic measurement theory
• If we describe coverage by counting test cases, we’re
committing reification error.
• If we use percentages to quantify coverage, we need to
establish what 100% looks like.
• But we might do that with respect to some specific models.
• Complex systems may display emergent behaviour.
31. 29
Extent of Coverage
• Smoke and sanity
• Can this thing even be tested at all?
• Common, core, and critical
• Can this thing do the things it must do?
• Does it handle happy paths and regular input?
• Can it work?
• Complex, harsh, extreme and exceptional
• Will this thing handle challenging tests, complex data flows,
and malformed input, etc.?
• Will it work?
How Might We Organize,
Record, and Report Coverage?
• automated tools (e.g. profilers, coverage tools)
• annotated diagrams and mind maps
• coverage matrices
• bug taxonomies
• Michael Hunter’s You Are Not Done Yet list
• James Bach’s Heuristic Test Strategy Model
• described at www.satisfice.com
• articles about it at www.developsense.com
• Mike Kelly’s MCOASTER model
• product coverage outlines and risk lists
• session-based test management
• http://www.satisfice.com/sbtm
See three articles here:
http://www.developsense.com/publications.html#coverage
32. 30
59
What Does Rapid Testing Look Like?
Concise Documentation Minimizes Waste
Risk ModelCoverage Model Test Strategy
Reference
Risk CatalogTesting Heuristics
General
Project-
Specific
Testing
Playbook
Status
Dashboard
Schedule BugsIssues
Rapid Testing Documentation
• Recognize
• a requirements document is not the requirements
• a test plan document is not a test plan
• a test script is not a test
• doing, rather than planning, produces results
• Determine where your documentation is on the continuum:
product or tool?
• Keep your tools sharp and lightweight
• Obtain consensus from others as to what’s necessary and what’s
excess in products
• Ask whether reporting test results takes priority over
obtaining test results
• note that in some contexts, it might
• Eliminate unnecessary clerical work
34. 32
Visualizing Test Progress
See “A Sticky Situation”, Better Software, February 2012
What IS Exploratory Testing?
• Simultaneous test design, test
execution, and learning.
• James Bach, 1995
But maybe it would be a good idea to underscore
why that’s important…
35. 33
What IS Exploratory Testing?
• I follow (and to some degree contributed to) Kaner’s definition, which
was refined over several peer conferences through 2007:
Exploratory software testing is…
• a style of software testing
• that emphasizes the personal freedom and responsibility
• of the individual tester
• to continually optimize the value of his or her work
• by treating test design, test execution, test result interpretation,
and test-related learning
• as mutually supportive activities
• that run in parallel
• throughout the project.
See Kaner, “Exploratory Testing After 23 Years”, www.kaner.com/pdfs/ETat23.pdf
So maybe it would be
a good idea to keep it
brief most of the
time…
Why Exploratory Approaches?
• Systems are far
more than
collections of
functions
• Systems typically
depend upon and
interact with
many external
systems
36. 34
Why Exploratory Approaches?
• Systems are too
complex for
individuals to
comprehend and
describe
• Products evolve
rapidly in ways
that cannot be
anticipated
In the future, developers will likely do more verification and validation at the
unit level than they have done before.
Testers must explore, discover, investigate, and learn about the system.
Why Exploratory Approaches?
• Developers are using tools and frameworks that
make programming more productive, but that
may manifest more emergent behaviour.
• Developers are increasingly adopting unit testing
and test-driven development.
• The traditional focus is on verification, validation,
and confirmation.
The new focus must be on exploration, discovery,
investigation, and learning.
37. 35
Why Exploratory Approaches?
• We don’t have time to waste
• preparing wastefully elaborate written plans
• for complex products
• built from many parts
• and interacting with many systems
• (many of which we don’t control…
• …or even understand)
• where everything is changing over time
• and there’s so much learning to be done
• and the result, not the plan, is paramount.
Questions About Scripts…
arrows and cycles
Where do scripts
come from?
What happens when the
unexpected happens
during a script?
What do we do
with what we
learn?
Will everyone follow the same
script the same way?
(task performing)
38. 36
Questions About Exploration…
arrows and cycles
(value seeking)
Where does
exploration
come from?
What happens when
the unexpected
happens during
exploration?
What do we do
with what we
learn?
Will everyone
explore the same way?
Exploration is Not Just Action
arrows and cycles
39. 37
You can put them together!
arrows and cycles
You can put them together!
arrows and cycles
40. 38
What Exploratory Testing Is Not
• Touring
• http://www.developsense.com/blog/2011/12/what-exploratory-testing-is-not-part-1-
touring/
• After-Everything-Else Testing
• http://www.developsense.com/blog/2011/12/what-exploratory-testing-is-not-part-2-after-
everything-else-testing/
• Tool-Free Testing
• http://www.developsense.com/blog/2011/12/what-exploratory-testing-is-not-part-3-tool-
free-testing/
• Quick Tests
• http://www.developsense.com/blog/2011/12/what-exploratory-testing-is-not-part-4-quick-
tests/
• Undocumented Testing
• http://www.developsense.com/blog/2011/12/what-exploratory-testing-is-not-part-5-
undocumented-testing/
• “Experienced-Based” Testing
• http://www.satisfice.com/blog/archives/664
• defined by any specific example of exploratory testing
• http://www.satisfice.com/blog/archives/678
Exploratory Testing
• IS NOT “random testing” (or sloppy,
or slapdash testing)
• IS NOT “unstructured testing”
• IS NOT procedurally structured
• IS NOT unteachable
• IS NOT unmanageable
• IS NOT scripted
• IS NOT a technique
• IS “ad hoc”, in the dictionary sense,
“to the purpose”
• IS structured and rigorous
• IS cognitively structured
• IS highly teachable
• IS highly manageable
• IS chartered
• IS an approach
The way we practice and teach it, exploratory testing…
41. 39
Contrasting Approaches
Scripted Testing
• Is directed from elsewhere
• Is determined in advance
• Is about confirmation
• Is about controlling tests
• Emphasizes predictability
• Emphasizes decidability
• Like making a speech
• Like playing from a score
Exploratory Testing
• Is directed from within
• Is determined in the moment
• Is about investigation
• Is about improving test design
• Emphasizes adaptability
• Emphasizes learning
• Like having a conversation
• Like playing in a jam session
Exploratory Testing IS Structured
• Exploratory testing, as we teach it, is a structured process conducted by a
skilled tester, or by lesser skilled testers or users working under supervision.
• The structure of ET comes from many sources:
• Test design heuristics
• Chartering
• Time boxing
• Perceived product risks
• The nature of specific tests
• The structure of the product being tested
• The process of learning the product
• Development activities
• Constraints and resources afforded by the project
• The skills, talents, and interests of the tester
• The overall mission of testing
In other words,
it’s not “random”,
but systematic.
Not procedurally
structured, but
cognitively structured.
42. 40
79
In excellent exploratory testing, one structure tends to
dominate all the others:
Exploratory testers construct a compelling story of
their testing. It is this story that
gives ET a backbone.
Exploratory Testing IS Structured
80
To test is to compose, edit, narrate, and justify
THREE stories.
A story about the status of the PRODUCT…
…about what it does, how it failed, and how it might fail...
…in ways that matter to your various clients.
A story about HOW YOU TESTED it…
…how you configured, operated and observed it…
…how you recognized problems…
…about what you have and haven’t tested yet…
…and what you won’t test at all (unless the client objects)…
A story about how GOOD that testing was…
…the risks and costs of (not) testing…
…what made testing harder or slower…
…how testable (or not) the product is…
…what you need and what you recommend.
Bugs
Issues
Product any good?
How do you know?
Why should I be pleased
with your work?
43. 41
81
What does “taking advantage of resources” mean?
• Mission
• The problem we are here to solve for our customer.
• Information
• Information about the product or project that is needed for testing.
• Developer relations
• How you get along with the programmers.
• Team
• Anyone who will perform or support testing.
• Equipment & tools
• Hardware, software, or documents required to administer testing.
• Schedule
• The sequence, duration, and synchronization of project events.
• Test Items
• The product to be tested.
• Deliverables
• The observable products of the test project.
82
“Ways to test…”?
General Test Techniques
• Function testing
• Domain testing
• Stress testing
• Flow testing
• Scenario testing
• Claims testing
• User testing
• Risk testing
• Automatic checking
44. 42
83
• Happy Path
• Tour the Product
• Sample Data
• Variables
• Files
• Complexity
• Menus & Windows
• Keyboard & Mouse
A quick test is a cheap test that has some value
but requires little preparation, knowledge,
or time to perform.
• Interruptions
• Undermining
• Adjustments
• Dog Piling
• Continuous Use
• Feature Interactions
• Click on Help
Cost as a Simplifying Factor
Try quick tests as well as careful tests
84
• Input Constraint Attack
• Click Frenzy
• Shoe Test
• Blink Test
• Error Message Hangover
A quick test is a cheap test that has some value
but requires little preparation, knowledge,
or time to perform.
• Resource Starvation
• Multiple Instances
• Crazy Configs
• Cheap Tools
Cost as a Simplifying Factor
Try quick tests as well as careful tests
45. 43
Touring the Product:
Mike Kelly’s FCC CUTS VIDS
• Feature tour
• Complexity tour
• Claims tour
• Configuration tour
• User tour
• Testability tour
• Scenario tour
• Variability tour
• Interoperability tour
• Data tour
• Structure tour
46. 44
88
Summing Up:
Themes of Rapid Testing
• Put the tester's mind at the center of testing.
• Learn to deal with complexity and ambiguity.
• Learn to tell a compelling testing story.
• Develop testing skills through practice, not just talk.
• Use heuristics to guide and structure your process.
• Replace “check for…” with “look for problems in…”
• Be a service to the project community, not an obstacle.
• Consider cost vs. value in all your testing activity.
• Diversify your team and your tactics.
• Dynamically manage the focus of your work.
• Your context should drive your choices, both of which
evolve over time.
47. - 1 -
Designed by James Bach Version 5.2.1
james@satisfice.com 9/17/2013
www.satisfice.com
Copyright 1996-2013, Satisfice, Inc.
The Heuristic Test Strategy Model is a set of patterns for designing a test strategy. The immediate purpose of
this model is to remind testers of what to think about when they are creating tests. Ultimately, it is intended to
be customized and used to facilitate dialog and direct self-learning among professional testers.
Project Environment includes resources, constraints, and other elements in the project that may enable or
hobble our testing. Sometimes a tester must challenge constraints, and sometimes accept them.
Product Elements are things that you intend to test. Software is complex and invisible. Take care to cover all of
it that matters, not just the parts that are easy to see.
Quality Criteria are the rules, values, and sources that allow you as a tester to determine if the product has
problems. Quality criteria are multidimensional and often hidden or self-contradictory.
Test Techniques are heuristics for creating tests. All techniques involve some sort of analysis of project
environment, product elements, and quality criteria.
Perceived Quality is the result of testing. You can never know the "actual" quality of a software product, but
through the application of a variety of tests, you can make an informed assessment of it.
48. - 2 -
General Test Techniques
A test technique is a heuristic for creating tests. There are many interesting techniques. The list includes nine general
techniques. By “general technique” we mean that the technique is simple and universal enough to apply to a wide variety
of contexts. Many specific techniques are based on one or more of these nine. And an endless variety of specific test
techniques may be constructed by combining one or more general techniques with coverage ideas from the other lists in
this model.
Function Testing
Test what it can do
1. Identify things that the product can do (functions and
sub-functions).
2. Determine how you’d know if a function was capable of
working.
3. Test each function, one at a time.
4. See that each function does what it’s supposed to do and
not what it isn’t supposed to do.
Claims Testing
Verify every claim
1. Identify reference materials that include claims about
the product (implicit or explicit). Consider SLAs, EULAs,
advertisements, specifications, help text, manuals, etc.
2. Analyze individual claims, and clarify vague claims.
3. Verify that each claim about the product is true.
4. If you’re testing from an explicit specification, expect it
and the product to be brought into alignment.
Domain Testing
Divide and conquer the data
1. Look for any data processed by the product. Look at
outputs as well as inputs.
2. Decide which particular data to test with. Consider
things like boundary values, typical values, convenient
values, invalid values, or best representatives.
3. Consider combinations of data worth testing together.
User Testing
Involve the users
1. Identify categories and roles of users.
2. Determine what each category of user will do (use
cases), how they will do it, and what they value.
3. Get real user data, or bring real users in to test.
4. Otherwise, systematically simulate a user (be careful—
it’s easy to think you’re like a user even when you’re
not).
5. Powerful user testing is that which involves a variety of
users and user roles, not just one.
Stress Testing
Overwhelm the product
1. Look for sub-systems and functions that are vulnerable
to being overloaded or “broken” in the presence of
challenging data or constrained resources.
2. Identify data and resources related to those sub-
systems and functions.
3. Select or generate challenging data, or resource
constraint conditions to test with: e.g., large or complex
data structures, high loads, long test runs, many test
cases, low memory conditions.
Risk Testing
Imagine a problem, then look for it.
1. What kinds of problems could the product have?
2. Which kinds matter most? Focus on those.
3. How would you detect them if they were there?
4. Make a list of interesting problems and design tests
specifically to reveal them.
5. It may help to consult experts, design documentation,
past bug reports, or apply risk heuristics.
Flow Testing
Do one thing after another
1. Perform multiple activities connected end-to-end; for
instance, conduct tours through a state model.
2. Don’t reset the system between actions.
3. Vary timing and sequencing, and try parallel threads.
Automatic Checking
Check a million different facts
1. Look for or develop tools that can perform a lot of
actions and check a lot things.
2. Consider tools that partially automate test coverage.
3. Consider tools that partially automate oracles.
4. Consider automatic change detectors.
5. Consider automatic test data generators.
6. Consider tools that make human testing more powerful.
Scenario Testing
Test to a compelling story
1. Begin by thinking about everything going on around the
product.
2. Design tests that involve meaningful and complex
interactions with the product.
3. A good scenario test is a compelling story of how
someone who matters might do something that matters
with the product.
49. - 3 -
Project Environment
Creating and executing tests is the heart of the test project. However, there are many factors in the project environment
that are critical to your decision about what particular tests to create. In each category, below, consider how that factor
may help or hinder your test design process. Try to exploit every resource.
Mission. Your purpose on this project, as understood by you and your customers.
Do you know who your customers are? Whose opinions matter? Who benefits or suffers from the work you do?
Do you know what your customers expect of you on this project? Do you agree?
Maybe your customers have strong ideas about what tests you should create and run.
Maybe they have conflicting expectations. You may have to help identify and resolve those.
Information. Information about the product or project that is needed for testing.
Whom can we consult with to learn about this project?
Are there any engineering documents available? User manuals? Web-based materials? Specs? User stories?
Does this product have a history? Old problems that were fixed or deferred? Pattern of customer complaints?
Is your information current? How are you apprised of new or changing information?
Are there any comparable products or projects from which we can glean important information?
Developer Relations. How you get along with the programmers.
Hubris: Does the development team seem overconfident about any aspect of the product?
Defensiveness: Is there any part of the product the developers seem strangely opposed to having tested?
Rapport: Have you developed a friendly working relationship with the programmers?
Feedback loop: Can you communicate quickly, on demand, with the programmers?
Feedback: What do the developers think of your test strategy?
Test Team. Anyone who will perform or support testing.
Do you know who will be testing? Do you have enough people?
Are there people not on the “test team” that might be able to help? People who’ve tested similar products before and might
have advice? Writers? Users? Programmers?
Are there particular test techniques that the team has special skill or motivation to perform?
Is any training needed? Is any available?
Who is co-located and who is elsewhere? Will time zones be a problem?
Equipment & Tools. Hardware, software, or documents required to administer testing.
Hardware: Do we have all the equipment you need to execute the tests? Is it set up and ready to go?
Automation: Are any test tools needed? Are they available?
Probes: Are any tools needed to aid in the observation of the product under test?
Matrices & Checklists: Are any documents needed to track or record the progress of testing?
Schedule. The sequence, duration, and synchronization of project events
Test Design: How much time do you have? Are there tests better to create later than sooner?
Test Execution: When will tests be executed? Are some tests executed repeatedly, say, for regression purposes?
Development: When will builds be available for testing, features added, code frozen, etc.?
Documentation: When will the user documentation be available for review?
Test Items. The product to be tested.
Scope: What parts of the product are and are not within the scope of your testing responsibility?
Availability: Do you have the product to test? Do you have test platforms available? When do you get new builds?
Volatility: Is the product constantly changing? What will be the need for retesting?
New Stuff: What has recently been changed or added in the product?
Testability: Is the product functional and reliable enough that you can effectively test it?
Future Releases: What part of your tests, if any, must be designed to apply to future releases of the product?
Deliverables. The observable products of the test project.
Content: What sort of reports will you have to make? Will you share your working notes, or just the end results?
Purpose: Are your deliverables provided as part of the product? Does anyone else have to run your tests?
Standards: Is there a particular test documentation standard you’re supposed to follow?
Media: How will you record and communicate your reports?
50. - 4 -
Product Elements
Ultimately a product is an experience or solution provided to a customer. Products have many dimensions. So, to test well,
we must examine those dimensions. Each category, listed below, represents an important and unique aspect of a product.
Testers who focus on only a few of these are likely to miss important bugs.
Structure. Everything that comprises the physical product.
Code: the code structures that comprise the product, from executables to individual routines.
Hardware: any hardware component that is integral to the product.
Non-executable files: any files other than multimedia or programs, like text files, sample data, or help files.
Collateral: anything beyond software and hardware that is also part of the product, such as paper documents, web links and content,
packaging, license agreements, etc.
Function. Everything that the product does.
Application: any function that defines or distinguishes the product or fulfills core requirements.
Calculation: any arithmetic function or arithmetic operations embedded in other functions.
Time-related: time-out settings; daily or month-end reports; nightly batch jobs; time zones; business holidays; interest calculations; terms and
warranty periods; chronograph functions.
Transformations: functions that modify or transform something (e.g. setting fonts, inserting clip art, withdrawing money from account).
Startup/Shutdown: each method and interface for invocation and initialization as well as exiting the product.
Multimedia: sounds, bitmaps, videos, or any graphical display embedded in the product.
Error Handling: any functions that detect and recover from errors, including all error messages.
Interactions: any interactions between functions within the product.
Testability: any functions provided to help test the product, such as diagnostics, log files, asserts, test menus, etc.
Data. Everything that the product processes.
Input: any data that is processed by the product.
Output: any data that results from processing by the product.
Preset: any data that is supplied as part of the product, or otherwise built into it, such as prefabricated databases, default values, etc.
Persistent: any data that is stored internally and expected to persist over multiple operations. This includes modes or states of the product,
such as options settings, view modes, contents of documents, etc.
Sequences/Combinations: any ordering or permutation of data, e.g. word order, sorted vs. unsorted data, order of tests.
Cardinality: Numbers of objects or fields may vary (e.g. zero, one, many, max, open limit). Some may have to be unique (e.g. database keys).
Big/Little: variations in the size and aggregation of data.
Noise: any data or state that is invalid, corrupted, or produced in an uncontrolled or incorrect fashion.
Lifecycle: transformations over the lifetime of a data entity as it is created, accessed, modified, and deleted.
Interfaces. Every conduit by which the product is accessed or expressed.
User Interfaces: any element that mediates the exchange of data with the user (e.g. displays, buttons, fields, whether physical or virtual).
System Interfaces: any interface with something other than a user, such as other programs, hard disk, network, etc.
API/SDK: Any programmatic interfaces or tools intended to allow the development of new applications using this product.
Import/export: any functions that package data for use by a different product, or interpret data from a different product.
Platform. Everything on which the product depends (and that is outside your project).
External Hardware: hardware components and configurations that are not part of the shipping product, but are required (or
optional) in order for the product to work: systems, servers, memory, keyboards, the Cloud.
External Software: software components and configurations that are not a part of the shipping product, but are required (or
optional) in order for the product to work: operating systems, concurrently executing applications, drivers, fonts, etc.
Internal Components: libraries and other components that are embedded in your product but are produced outside your project.
Operations. How the product will be used.
Users: the attributes of the various kinds of users.
Environment: the physical environment in which the product operates, including such elements as noise, light, and distractions.
Common Use: patterns and sequences of input that the product will typically encounter. This varies by user.
Disfavored Use: patterns of input produced by ignorant, mistaken, careless or malicious use.
Extreme Use: challenging patterns and sequences of input that are consistent with the intended use of the product.
Time. Any relationship between the product and time.
Input/Output: when input is provided, when output created, and any timing relationships (delays, intervals, etc.) among them.
Fast/Slow: testing with “fast” or “slow” input; fastest and slowest; combinations of fast and slow.
Changing Rates: speeding up and slowing down (spikes, bursts, hangs, bottlenecks, interruptions).
Concurrency: more than one thing happening at once (multi-user, time-sharing, threads, and semaphores, shared data).
51. - 5 -
Quality Criteria Categories
A quality criterion is some requirement that defines what the product should be. By thinking about different kinds of
criteria, you will be better able to plan tests that discover important problems fast. Each of the items on this list can be
thought of as a potential risk area. For each item below, determine if it is important to your project, then think how you
would recognize if the product worked well or poorly in that regard.
Capability. Can it perform the required functions?
Reliability. Will it work well and resist failure in all required situations?
Robustness: the product continues to function over time without degradation, under reasonable conditions.
Error handling: the product resists failure in the case of errors, is graceful when it fails, and recovers readily.
Data Integrity: the data in the system is protected from loss or corruption.
Safety: the product will not fail in such a way as to harm life or property.
Usability. How easy is it for a real user to use the product?
Learnability: the operation of the product can be rapidly mastered by the intended user.
Operability: the product can be operated with minimum effort and fuss.
Accessibility: the product meets relevant accessibility standards and works with O/S accessibility features.
Charisma. How appealing is the product?
Aesthetics: the product appeals to the senses.
Uniqueness: the product is new or special in some way.
Necessity: the product possesses the capabilities that users expect from it.
Usefulness: the product solves a problem that matters, and solves it well.
Entrancement: users get hooked, have fun, are fully engaged when using the product.
Image: the product projects the desired impression of quality.
Security. How well is the product protected against unauthorized use or intrusion?
Authentication: the ways in which the system verifies that a user is who he says he is.
Authorization: the rights that are granted to authenticated users at varying privilege levels.
Privacy: the ways in which customer or employee data is protected from unauthorized people.
Security holes: the ways in which the system cannot enforce security (e.g. social engineering vulnerabilities)
Scalability. How well does the deployment of the product scale up or down?
Compatibility. How well does it work with external components & configurations?
Application Compatibility: the product works in conjunction with other software products.
Operating System Compatibility: the product works with a particular operating system.
Hardware Compatibility: the product works with particular hardware components and configurations.
Backward Compatibility: the products works with earlier versions of itself.
Resource Usage: the product doesn’t unnecessarily hog memory, storage, or other system resources.
Performance. How speedy and responsive is it?
Installability. How easily can it be installed onto its target platform(s)?
System requirements: Does the product recognize if some necessary component is missing or insufficient?
Configuration: What parts of the system are affected by installation? Where are files and resources stored?
Uninstallation: When the product is uninstalled, is it removed cleanly?
Upgrades/patches: Can new modules or versions be added easily? Do they respect the existing configuration?
Administration: Is installation a process that is handled by special personnel, or on a special schedule?
Development. How well can we create, test, and modify it?
Supportability: How economical will it be to provide support to users of the product?
Testability: How effectively can the product be tested?
Maintainability: How economical is it to build, fix or enhance the product?
Portability: How economical will it be to port or reuse the technology elsewhere?
Localizability: How economical will it be to adapt the product for other places?