Beyond "Quality Assurance"
•
1 like•1,402 views
Beyond "Quality Assurance" The secrets of testing on an agile team
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Beyond "Quality Assurance"
Similar to Beyond "Quality Assurance" (20)
Recently uploaded
Recently uploaded (20)
Beyond "Quality Assurance"
- 1. Jason Benton Agile Birmingham June 2014 Beyond “Quality Assurance” The secrets of testing on an agile team
- 2. Software Testing Agile Coaching Jason Benton Twitter: @jasonfbenton Email: jasonfbenton@gmail.com
- 3. AGILE You keep using that word. I do not think it means what you think it means The Princess Bride (1987) ©
- 4. Teams are small and focused
- 5. Smaller, more frequent releases
- 8. Self organizing teams produce the best software Christoffer A Rasmussen
- 10. Challenges…
- 11. As a Association Director, I would like to create a child care site so that I can schedule programs As a Association Director, I would like to create a child care site so that I can schedule programs As a Association Director, I would like to manage fee charts so that I can charge members different rates for different programs As a Site Director, I would like to register a child for a program so that I can schedule programs As a Site Director, I would like to look up a child so that I can view their registered programs
- 12. Smaller, more frequent releases
- 14. TestingPercentage 0 25 50 75 100 Iteration Beginning of sprint End of sprint Testing happens late
- 15. As a <type of user> , I would like <some functionality>, so that <I can accomplish some goal>. Acceptance Criteria Acceptance Criteria Acceptance Criteria Acceptance Criteria Acceptance Criteria
- 16. Agile Testing at Daxko
- 17. http://context-driven-testing.com/ • The value of any practice depends on its context. • There are good practices in context, but there are no best practices. • People, working together, are the most important part of any project’s context. • Projects unfold over time in ways that are often not predictable. • The product is a solution. If the problem isn’t solved, the product doesn’t work. • Good software testing is a challenging intellectual process. • Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products. The Seven Basic Principles of the Context-Driven School
- 19. Testing Principles • The essential value of any test is its ability to provide information and reduce uncertainty. • The value of any practice depends on its context. • There are good practices in context, but there are no best practices. Effective testing requires contextual analysis and consideration. Factors such as risk, complexity, fragility, priority, value, and performance need to be considered. • Testing activities should begin as early as possible. Be practical! Testers should collaborate with development to break stories into small, testable pieces during planning. • Manual test sessions need to vary over time. The same tests repeated manually over time provide diminishing value. This is one of the biggest advantages of structured exploratory testing. • Test documentation is a valuable tool to the extent that it helps you manage your testing project and find defects. Beyond that, it is a diversion of resources. • Know the user. Know the problem we are trying to solve. Focus on delivering value to customers. Finding and fixing defects does not help if the system built is unusable and does not fulfill the user's needs and expectations. • To be effective and efficient, testing activities must include: plan or charter for testing, test design, preparing test environments, executing tests, and evaluating results. These activities may overlap or take place concurrently. • Defects should be communicated in a constructive way. Bad feelings between testers, designers, and developers can be avoided. Communicate findings in a neutral, fact focused way without criticism. • Inspect and adapt. Reflect on what went well and what can be improved with each iteration. Strive to continuously learn and improve. • Automated testing is better than manual testing. • Manual testing is better than automated testing.
- 20. As a registered user, I would like to be able to login so that I can access my files from anywhere 3 unsuccessful login attempts results in a 10 minute lockout Account is protected by username (email) and password Authenticated users can see their account info (name, email, and password)
- 21. 3 unsuccessful login attempts results in a 10 minute lockout Account is protected by username (email) and password Authenticated users can see their account info (name, email, and password) As a registered user, I would like to be able to login so that I can access my files from anywhere
- 22. 3 unsuccessful login attempts results in a 10 minute lockout Account is protected by username (email) and password Authenticated users can see their account info (name, email, and password) As a registered user, I would like to be able to login so that I can access my files from anywhere
- 23. Iterative Plan Design Develop Test Plan Design Develop Test Plan Design Develop Test Plan Design Develop Test 2 Weeks 2 Weeks 2 Weeks 2 Weeks
- 24. Eliminate Waste
- 27. Exploratory Testing: Simultaneous learning, test design and test execution.
- 28. 1) Charter - the goal or mission for the test session ! 2) Session - an uninterrupted period of time spent testing ! 3) Session Report - detailed record of the testing session and results ! 4) Debrief - a short discussion about the session report Session Based Exploratory Testing
- 29. Charter Examples • Explore all workflows for adding members (in-house and online) • Explore and analyze the Reciprocity app. Produce a Feature Map that outlines the capabilities and features. • Explore the budget workflow from start to finish. • Compare data in various custom reports from before and after an upgrade to Logi. • Verify all maintenance item fixes in the Fundraising component for Operations 11.6 • Test all fields that allow data entry for SQL injection. • Check UI against Daxko design standards. • Test all 3rd party integration with Operations • Explore the ability to change member info using the online account feature • Validate the accuracy of the financial batch jobs for credit card processing
- 31. • Require testers to identify test objectives and focus their testing efforts on fulfilling them. • Encourage testers to modify existing tests and add new tests to meet those objectives as they test. • Provide a useful way to manage and review exploratory testing. • Focus attention on the testing that’s actually performed, not just on test case results. • Give testers the flexibility to respond to changes and re-plan their testing quickly. Benefits
- 33. Feature Maps
- 34. Feature Maps
- 36. Risk Heuristics: Generic risks are risks that are universal to any system. These are my favorite generic risks: ! • Complex: anything disproportionately large, intricate, or convoluted. ! • New: anything that has no history in the product. ! • Changed: anything that has been tampered with or "improved". ! • Upstream Dependency: anything whose failure will cause cascading failure in the rest of the system. ! • Downstream Dependency: anything that is especially sensitive to failures in the rest of the system. ! • Critical: anything whose failure could cause substantial damage. ! • Precise: anything that must meet its requirements exactly. ! • Popular: anything that will be used a lot. ! • Strategic: anything that has special importance to your business, such as a feature that sets you apart from the competition. ! • Third-party: anything used in the product, but developed outside the project. ! • Distributed: anything spread out in time or space, yet whose elements must work together. ! • Buggy: anything known to have a lot of problems. ! • Recent failure: anything with a recent history of failure.
- 37. SFDPO - San Francisco Depot" ! • Structure (what the product is): What files does it have? Do I know anything about how it was built? Is it one program or many? What physical material comes with it? Can I test it module by module? ! • Function (what the product does): What are its functions? What kind of error handling does it do? What kind of user interface does it have? Does it do anything that is not visible to the user? How does it interface with the operating system? ! • Data (what it processes): What kinds of input does it process? What does its output look like? What kinds of modes or states can it be in? Does it come packaged with preset data? Is any of its input sensitive to timing or sequencing? ! • Platform (what it depends upon): What systems does it run on? Does the environment have to be configured in any special way? Does it depend on third- party components? ! • Operations (how it will be used): Who will use it? Where and how will they use it? What will they use it for? Are there certain things that users are more likely to do? Is there user data we could get to help make the tests more realistic?
- 42. Functional Test Automation • Whole team responsibility • Page Object Model design pattern • Watir & RSpec • Breadth > Depth • Run tests often • Make broken tests visible to the team
- 43. Compliance
- 44. Lessons Learned
- 45. TestingPercentage 0 25 50 75 100 Iteration Beginning of sprint End of sprint Test early, test often
- 46. Build quality in Unit Testing Code Reviews Pairing Automated Tests Continuous Integration Story Mapping
- 47. Collaborate on test ideas
- 49. Get testers involved in planning and estimation Jakuza
- 51. Get out of the building! Office Space (1999) ©
- 54. Question value vs. cost for all of your testing efforts.
- 55. Use prod-like environments for testing
- 56. Learn where defects cluster
- 60. Questions? Software Testing Agile Coaching Jason Benton Twitter: @jasonfbenton Email: jasonfbenton@gmail.com
Editor's Notes
- Intro (QA+Agile Coach background) Are there any testers here?! Ask what people are struggling with related to agile and testing?
- Set the context for what I mean when I say “Agile teams”
- Teams are small, cross-functional and dedicated to a single product or product area.
- Focus on frequent delivery of value No single, big releases every 6-12 months We measure cycle time to help us keep this short
- We embrace the notion that requirements change, unexpected requirements appear, and priorities shift. Our development practices must enable quick, accurate adaptation to these changes. How many times have you heard someone say, “That is cool - but... what if it also did X?”
- Everyone is involved in story writing, estimation, planning, etc. Builds a shared understanding! Teams all sit together in a single open space
- Teams should be empowered to focus on doing whatever is necessary to deliver the features. Encourage teams to swarm on tasks that need to be completed even if it puts them out of their comfort zone.
- Always strive to improve. Constantly look at how we are doing and ask “What could we do to be better?” At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior. Agile is not a destination - We work differently today, than we did 6 months ago. (Does this all sound familiar?)
- This creates a lot of challenges from a software testing perspective. A lot of teams haven’t changed their approach to testing. They are doing the same things that were being taught 20 years ago. Most of those things didn’t work then, and certainly don’t now.
- Requirements aren’t as detailed - There’s no “the system shall…” level of detail User stories with acceptance criteria
- More releases mean more regression testing Some teams push code to production daily!
- Pace is much faster. We no longer have periods of weeks to plan our testing
- Sprints become mini waterfalls Testing only happens at the end of the sprint Defects carry over into the next sprint
- Late changes to the requirements that necessitate additional testing Some teams create additional stories for those
- We are context driven. Ultimately, context-driven testing is about doing the best we can with what we get. Rather than trying to apply “best practices,” we accept that very different practices will work best under different circumstances. Understand that as you learn more about how we work. Our way of doing things may or may not work well in your specific context.
- Quality is everyone’s responsibility. There’s no throw it over the wall mentality. Happened organically. We don’t play teams against one another. There is a culture of learning, not blame. If we ship a critical defect, the whole team reflects on what they have learned from it and what they should do differently.
- We came up with a set of principles that can be used to guide and direct the team. We review them with the team on a regular basis and modify as needed
- Requirements are expressed as User Stories with Acceptance Criteria Story mapping helps build a shared understanding Product Owner team creates stories, and the rest of the team helps to refine them.
- We focus on who the users are and what are their goals. We can build software that does not have defects and still not meet the users needs.
- Tests start around acceptance criteria and user personas
- 2 weeks sprints. ScrumBan Definition of “Done” for each team. Working software is demonstrated every 2 weeks. Sprint retrospective at the end of each sprint
- Foster a culture where it is OK to question the value of a particular process or activity. Worked to optimize the teams workflow and eliminate unnecessary activities and deliverables.
- Biggest example of us eliminating waste is eliminating detailed test cases from our testing process By test cases, I mean “Open this page, enter this value, click this button, expect this result, etc.” type of test cases
- ALL of our manual testing is done using exploratory testing
- A lot of the old ways of testing assumed that we knew up front all of the ways that the system would be used and how it should behave. That’s simply not possible.
- James & Jon Bach created this as a way to put structure and organization around ET.
- Charters are created prior to the start of testing, but may be added or changed at any time. Others can suggest charters!
- Using Capture plugin for Jira for managing session-based ET.
- We found that switching to this process made us more productive and effective. Caveats: You have to have really good testers that can think for themselves!
- We don’t have scripted test cases. The Feature Map is simply a map of application and features Broken down into functional areas of the system. Recently updated the menu in our flagship product. It only took a few hours to update the Feature Map.
- Checklist Doesn’t tell you how to test. Can help you with managing risk when making changes and doing manual regression. Can help tell you where to focus.
- Mental shortcuts experience-based techniques for problem solving, learning, and discovery
- Example of a heuristic around determining risk
- Example of a test technique heuristic. This one is used to help generate different test ideas
- We use a LOT of flip chart paper and sticky notes Cycle time, sprint burnup, user story maps, release plan, PO team backlog, etc. Big on communication and transparency
- One team’s task board. There’s a LOT of info on here. Stories are in priority order. Color coded by task. Magnetic avatars to indicate “who” is working on something. Columns indicate progress.
- Automation Triangle - Mike Cohn Our CI runs unit tests with each build
- QA is focused on functional test automation at the UI. These are expensive to maintain QA can give input to Unit tests (field validation as an example) We are still working on this! Some teams are doing better than others.
- QA Team learned Ruby together using online training. dev/tester mentoring We created a weekly workshop to write tests together. Picked Ruby because of popularity, support, and ease.
- These processes and tools can work in a highly regulated environment. We have passed our SSAE16 and PCI audits with flying colors for the last 3 years.
- Do things that promote early learning Don’t wait until the car is built to test it. Test the parts! Break stories down into smaller stories that can be completed in 1-2 days. Consider pushing acceptance criteria to test instead of complete stories
- Promote things that help build quality into your development process. Coaching other team members on testing Trying to improve quality by increasing your testing efforts is like trying to lose weight by weighing yourself more.
- Collaborate on test ideas - you may learn things that you don’t need to test (Covered by Unit tests) Paired testing! Dev + Tester, Designer + Tester, Tester + Tester, etc.
- Make testing problems visible to the rest of the team! They may be able to help you solve them!
- Get testers involved in planning & estimation! They will find problems early. Testers tend to know the whole system and not just parts. They can often point out potential issues well before actual development begins.
- When testing is needed, does everyone test? -Alistair Cockburn Think of “roles” as positions on a sports team. The punter will make a tackle if he’s the only defender left between the ball and the endzone - Jeff Patton
- Get out of the building. Get your team in front of customers if at all possible. They will build empathy and better understand your customers problems, goals, and needs. You have to really understand the problem before you can develop a solution.
- Focus on measuring outcomes instead of output. It’s what really matters. Not velocity, closed cases, support calls, defects, LoC, etc! Does the software help people solve problems and achieve their goals? Don’t count defects or use metrics that count defects. That doesn’t promote behavior that you want. Trust me.
- Get everyone testing! Test drive site for everyone (sales, marketing, support, etc.) Rewards and recognition for people that find issues.
- This can help you prioritize and know when to stop testing “Is this the most valuable thing I can do to move the project forward?”
- Use production-like data and test environments for development and testing.
- Create taxonomies for defects. See where they cluster. Work to eliminate entire classes of defects. Ex. Business logic in stored procedures, field validation errors, etc.
- Inspect and adapt! Reflect on how the team is working and what could be improved. Do this frequently! I actually do this with my family as well!
- Most importantly - Have fun!
- Most importantly - Have fun!
- Let me know if you have comments or questions!