Download to read offline
Uploaded byTEST Huddle
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
The document discusses how agile testing can create customer value by reducing waste and improving quality in software development. It highlights the shift towards agile methods, predicting they will be used in 80% of projects by 2012, and emphasizes the need for collaboration between developers and testers to enhance communication and efficiency. Additionally, it outlines common challenges faced by QA managers and presents the role of QA in agile teams to better address evolving customer requirements.
More Related Content
Similar to Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
- 1. Creating Customer Value through Agile Testing Ben Walters (benwal@microsoft.com) Director of Program Management Visual Studio Test and Lab Management, Microsoft Corporation
- 2. Value Differentiation =Competitive Advantage • New Ways of Conducting Business • Faster, more Streamlined Organizations • Reduced Cost, Increased Return • Improved Customer Service, Higher Satisfaction
- 3. Risk Differentiation =Custom Software • Complex or Unique! • Hasn’t Been Done Before • Or is too complex to Buy • Do you know Exactly What you Need?
- 4. Change is theOnly Constant • Technological Evolution and Revolution • The Art of Software Creation • Customer Requirements
- 5. Changing Customer Requirements The Rise Of Agile
- 6. By 2012, agiledevelopment methods will be utilized in 80% of all software development projects. 80 % (Gartner Predicts 2010: Agile and Cloud Impact Application Development Directions, Gartner, Dec 2009)
- 7. Changing the Faceof Software Development 3 Fundamental Principles: Agile Consensus Flow of Value Reduction of Waste Increased Transparency
- 8. Result in MoreSuccessful Outcomes By: Decreasing Risk Improving Quality
- 9. The Art ofSoftware Creation Collaboration: The ‘90s: Basic Tools Early tools focused on isolated tasks, such as source control Dev Dev Dev Dev Dev
- 10. The Art ofSoftware Creation The ‘00s: LEAN Software Engineering Trustworthy Transparencyb brirdidggeess the team PM Devs DB Test Arch
- 11.
- 12. Common Concerns forTest/QA Managers “developers and testers work in silos and don’t communicate/speak the same language” “By the time we found a major issue with the architecture it was too late to economically fix it” “my testers are spending too long tes“dtienvge tlhoep esrasm aen dth tiensgt”ers work in silos “we have a changed requirement, what do I need to test?” “the developers say the defects are useless” “when is and my don’t software communicate/ready to speak ship?” “the developers say the “we hatvhee as acmhaen lgaendg ureaqguei”rement, what do I need to test?” defects are useless” “my testers are spending too long testing the same thing” “By the time we found a major issue with the architecture it was too late to economically fix it” “when is my software ready to ship?”
- 13. Common Concerns forTest/QA Managers “how do we test software without a specification?”
- 14. A process comparisonspectrum UAdnapplatnivneed PrPeldaincntievde
- 15. What is QA’srole?
- 19. Participating in predictiveprojects Develope r Tester Business Analyst
- 20. Participating in adaptiveprojects One Team Developer Tester Product Owner
- 21. BREAK it? Howcan I
- 22. Cost to fixdefects throughout the lifecycle 1x 5x 10x 15x 30x Requirements, Analysis, Design Coding/Unit Test Integration and System Test Beta Test Programs Post-product Release Source: The Economic Impacts of Inadequate Infrastructure for Software Testing, NIST
- 23. The mini-waterfall trap Requirements Code Code and Fix Test Iteration 1 Requirements Code Code and Fix Test Iteration 2 Requirements Code Code Iteration 3 Test
- 24. Requirements Active Participation Design Standups Story Development Retrospective s Feedback
- 25.
- 26. Empathy and Feedback That’s what we asked for, but it’s not what we need.” “
- 27. The role ofQA in Agile teams • Critical thinking throughout the cycle • Discovering unstated requirements • Scenario focused viewpoint • End user proxy
- 28.
- 29. Testing Agile Requirements Stories Storyboards Exploratory Testing Automation and Regression Testing Stakeholder feedback
- 30.
- 31. 80% Today 100% Tomorrow
- 32.
- 33.
- 34. Feedback • Requirementsand Tests • Priorities • Working Software
- 35. In Conclusion ALMis all about delivering maximum value to the customer Agile testing is about applying QA skills to create value throughout the development cycle Agile techniques and tools that implement them help you accelerate value creation in your projects
- 36.
Editor's Notes
- #2 I’m excited to be here to talk to you today about the opportunity we have to increase the impact of the testing profession by leveraging new approaches to maximize customer value delivered in our projects.
- #3 Differentiation is core to creating value, because it can equal competitive advantage. The Key to success in business today is the ability to be different and specifically to differentiate yourself from your competitors. More often than not technology is at the forefront of that differentiation enabling organizations to unlock new ways of conducting business, faster, with reduced cost and improved customer service.
- #4 However, differentiation implies that you are going to do something different; Something that hasn’t been done before. If the differentiation you are looking for is going to be delivered through software it probably means you are going to have to build it yourself. You are going to build something custom….! Something Unique and probably something complex And often you are not exactly sure what it is you need to build. After all if you can buy some piece of software that meets your needs, which you should if at all possible, then so can your competitors and that reduces’ your ability to differentiate yourself from them. Our Record as an industry for delivering software that truly differentiates an organization from the pack is not where we would like it to be. Why is this?
- #5 There are many reasons, but most fundamentally, as an industry, we struggle to cope with change. With change comes uncertainty and uncertainty makes us uncomfortable. But one thing is certain – Change is the only constant – As technology evolves you constantly find yourself relearning your skills and having to CHANGE your approach to the way you leverage technology The ART of creating software is changing over time to meet the increases in complexity and use of software and to continually strive for higher success rates Even as individual projects progress CHANGES in requirements, organizational needs, or even the business landscape need to be handled seamlessly -- and even planned for !! Don’t fear change, change is good, but do ensure you are prepared for it and have a way to manage it ------------------------------------------------------------- Long Talk Track Change is constant in the IT industry and with every evolution it comes faster than before. Across all the generations and changes that have happened within the software industry two things have remained consistent if you want to be at the forefront of the revolution. Each wave of innovation in the software industry brings the ability to do more, to differentiate more, while evolving the ‘art of the possible’. Today we have the ability to create more immersive experiences and experiences that integrate multiple sources of data to deliver breakthrough insight. Just the simple fact that in order to differentiate you have to build something that hasn’t been built before and leverage the latest and greatest of the new technological advances is at the core of what makes custom software development hard and an inherently risky proposition which has significant implication on the processes you follow in order to be successful. The art of creating software has also changed. Historically a loan developer could pull crazy hours and produce something amazing because the number of moving pieces and skills that they needed were manageable for an individual. As computer systems have become more complex involving more moving parts this is less and less the case. Most software development projects today involve a team or teams of people all tasked with delivering the best solution they can. Every member on the team has some core skills and then specific domain expertise that adds to the overall capabilities of the team, whether that is core software development skills targeting specific platforms or capabilities, project management capabilities that make sure the team operates efficiently or testing knowledge that helps ensure that the quality of the final delivery is far superior to anything previously produced. It has become more important than ever to help these people work together regardless of whether they are technical or not, or not even on the core team at all but valuable for the feedback they give on what is being built. Another change in team structure is the advent and rise of outsourcing where some or all of the project team is not a part of the IT organization but provided by a third party either onshore or offshore. This split leads to an ever more important need to sync across the team with actionable data for all involved parties. There is also a propensity for software projects to change over time. The simple fact that the what you are building hasn’t been built before leads to changes as the software evolves over time to meet the needs of the customer, as the customer better understands their needs for the software or as the market around them changes as well. Additionally with new capabilities comes the ability to build things that had only been imagined before while at the same time the ever changing business landscape means that it has become more difficult than ever to completely understand what a final solution needs to be. The inherent uncertainty in software projects makes it difficult to establish the entirety of your vision right at the outset and break this into tasks, which creates a high variance in the accuracy of the estimates. A common misconception is that the variance is acceptable because the positive and negative variations average out. However, because software projects are long chains of dependent events, the variation itself accumulates in the form of downstream delays7. For all these reasons it is more important than ever to be able to manage change throughout the application development lifecycle.
- #6 This trend has caused Agile development to gain mainstream acceptance as businesses look to enjoy the benefits gained from embracing change by taking an agile view on software development. Let me pause for a moment to get to know you all better. How many people in the audience are currently working on agile projects? How many people have a project coming up that will use an agile methodology?
- #7 The Analysts all agree that Agile Development methods have gone mainstream. Gartner predicts that by next year 80% of all software projects will utilize some form of Agile development methodology
- #8 This new paradigm for software development is based on many of the techniques that were birthed from the creation of the Agile Manifesto This new paradigm focuses on three fundamental principles for software development Flow of Value - Where value is defined by the customer Continual reduction of the waste that impedes the flow of value delivery Increased Transparency - Enabling team members to continually improve the above two. <CLICK> Lets call this set of fundamentals the Agile Consensus
- #9 Focusing on these 3 fundamental principles when approaching a software project will ultimately help your teams achieve more successful outcomes by: Decreasing Risk By managing the flow of value across the lifecycle and providing greater visibility into your project status you can manage the risk associated with the software development tasks that you undertake Raising Quality By improving the flow of information across the lifecycle you can collect timely and actionable feedback on the status of the software project allowing you to proactively address the quality of the software early and often
- #10 Early ALM tooling that began surfacing in the early to mid 90’s focused almost exclusively on isolated tasks, such as source control, requirements, test automation etc. The thinking was that the value was gained at the individual category level where you wanted to give people the “Best” requirements product or the “Best” source control system. <CLICK> From a developer perspective some changes that occurred did bring them closer together, for example integrated source control made it easier for developers to work together but excluded other members of the team and created siloes between different parts of the team
- #11 As we headed into the ‘00s new techniques and thinking emerged about how to better enable teams to work together. Concepts like Agile and Lean were beginning to emerge that looked at opportunities to Optimize the flow of information across a team And Reduce waste in handoffs between team members <CLICK> Microsoft’s Visual Studio Team System 2005 and its successor Visual Studio Team System 2008 in conjunction with Team foundation Server were among the first commercial products to support a more connected view of the lifecycle where the focus was not exclusively on the point products but instead targeted firmly on how the information flowed between team members and the various products that they worked with, reducing the waste inherent in the handoff between different areas of the team. Visual Studio 2010, Test Professional 2010 and Team Foundation Server 2010 has made another great leap forward to create transparency, improve flow and reduce waste in software development in particular by bringing developers and testers closer communications. VS 2010 is also one of the first products to tackle end-to-end agile engineering and project management practices.
- #12 Lets take a closer look at the role of QA in lean, agile software development
- #13 Here are some top concerns that test and QA managers shared with us. The developers say the defects are useless My testers are spending too long testing the same thing Tooling is expensive (time, licenses, people) When is my software ready to ship? <Question & Show of Hands> RAISE YOUR HAND IF YOU’VE NEVER HAD THESE OR SIMILAR PROBLEMS.
- #14 Another concern frequently raised by QA managers transitioning to agile practices comes from a need to adapt to a situation that was considered an anti-pattern in the v-model but is now the norm on agile teams “How do we test software without a specification?”
- #15 This really comes from a common misperception about agile. At first it is easy to mistake the lack of a specification in detailed documentation as a failure to plan. <CLICK> But in fact this is not a fair characterization. <CLICK> Agile teams focus on adaptive planning where lightweight practices are used to discover just enough detail about what customers need and iterate with customers to a shared understanding of the product to be delivered. This change means that QA teams need to adapt and consume requirements in new forms and flows.
- #16 This simple question, “how do we test without a spec?” raises a meta-question that we need to examine to continue our evolving understanding of QA’s contribution. What is QA’s role on an agile software team? I’m going to show you a few pictures. I’d like you to consider whether you have ever been on a project where your experience felt similar and whether you think that any of these examples represents the way you ideally believe QA should contribute to the project.
- #17 One approach is to act as the Quality Police, enforcing the law of the spec.
- #18 Another approach is to set yourself up as a center of excellence, partitioned from the main development team with communications happening in spurts. Product gets thrown over the wall, bugs get thrown back.
- #19 What agile teams strive for is to have testers and developers collaborate with product owners right from the beginning of the development cycle as equal partners.
- #20 In projects that use predictive planning, the Business Analyst is responsible for doing all of the thinking regarding customer needs, developers are expected to mechanically implement those requirements, and testers are expected to be the quality police that enforce that the software matches the specification.
- #21 Agile teams blur these roles and require every team member to collaborate more deeply, breaking down the barriers that impede communications and slow the flow of value. <CLICK> By acting as one team, with all team members equal participants agile lets us focus on delivering maximum value to the customer rather than narrowly restricting our view to a silo-ed role. At the same time it requires us as QA engineers to expand beyond traditional core skills – to think critically about requirements, to understand more deeply the detailed design of the software, to become better at communicating with customers. While the ‘One Team’ philosophy asks us to step outside our comfort zone, each team member still retains a core competency.
- #22 For testers, the core competency remains devising ways to break the software. Another way to describe this skill is critical thinking. Considering not just the positive, but the negative flows; understanding how to put just the right stress on the system; asking what if the unexpected happens and how can I construct an experiment to force it to happen. These skills are just as valuable on agile teams, and are not areas where your non-test team members are likely to be able to contribute as deeply as they can in their own core competency.
- #23 One of the biggest changes agile does ask of testers, is to apply your skills at earlier in the cycle. The data in this chart is from just one of a long series of studies stretching back to the 1970s showing that fixing defects earlier in the development cycle is incredibly efficient. The best possible time for QA engineers to apply critical thinking skills to the project is during the requirements and design phase of the project. Agile methodologies bring you to the table right from the beginning of the cycle, giving you a chance to notice and discuss problems during the earliest stages of the project, hopefully before coding has even begun. By eliminating the waste of your entire team’s time represented by coding and testing a broken or incomplete requirement you enable the rapid completion of each story with high quality and can then quickly move on to the next story.
- #24 Mini-waterfalls represents an anti-pattern that is easy for teams new to agile to fall into. Whether our iteration length is a week or a month, trying to divide it into milestones for definition, coding and testing puts us right back into the problems we had in waterfall. QA in particular suffers from in approach, as the back loaded test effort invariably gets squeezed by the end of the iteration. Particularly unhealthy teams even take an approach of having QA test the previous iteration’s stories, ignoring the agile mantra that “Done means DONE!”
- #25 So how do you get involved at all stages? It comes down to active participation in every stage of the iteration. While creating tasks for a story, also create acceptance tests and discuss them with the developer. Some teams go so far as to say that functionality is only written once an acceptance test has been defined that can verify it - this called Acceptance Test Driven Development, a cousin of Unit Test Driven Development. If you aren’t being invited to the design discussions, fix it! While you’re in a design discussion, apply your critical thinking skills and bring up potential problems right there so that the team incorporate your insight into the design. When you’re in the standup meeting, remember it’s not about reporting your status, it’s about understanding what is going on for each teammate. When you hear a statement that sounds like a developer didn’t understand a requirement or acceptance test, speak to them after the standup and clarify. You may have misheard, but they may have misunderstood the requirement and having the discussion immediately is much more efficient than waiting for a build and filing a bug. As developers are getting close to checking in code, go to their desk and test on their system first, a practice known as Pair Testing. Again, catching a problem earlier in the cycle saves everyone time which can be spent on delivering additional customer value.
- #26 As you participate throughout an iteration, the perspective you bring to the team is as valuable as your core critical thinking skills. Developers frequently get very focused on the detailed task at hand. While doing unit testing they tend to focus on testing a single class or a few classes in isolation. While testing stories they tend to focus just on that one story description. In some cases this can make them so focused on the tree that they lose sight of the forest. By thinking about the end to end scenario that is being built, by considering the impact of a change throughout the rest of the system you will catch problems that developer testing misses. Bringing this perspective to the team with each story being developed as opposed to waiting for an integration and system testing phase of the project is critical to making agile products succeed.
- #27 Has anyone here every completed a project successfully and had a customer respond like this? Was the project really a success? There are a couple of reasons for this: Customers frequently can’t envision what they’re asking for Customer frequently don’t know what they need There are tools that can help with both of these problems by helping customers visualize what the team will deliver and enabling them to frequently give feedback on working software. But before taking the software back to the customers for feedback, the QA team has the opportunity to walk a mile in your user’s shoes. While maintaining your test lab you will be the first operators of the software. While doing end to end scenario testing you will be the first people to feel what it’s like to actually use the software. Are there things that drive you crazy? Things that seem cumbersome or fundamentally broken? Bring it up with the rest of the team and talk about possible solutions. Go beyond the story definitions to create the best software you can. This end user empathy overlaps with the product owner’s viewpoint, but has a slightly different focus. Product owners main focus is on delighting the customer, who is frequently a different person than the user. Good product owners will try to keep both sets of needs in mind, but while doing product definition work the product owner is working on theory. The testers are the first people to experience the software from the end user perspective. As you feel their pain be a strong advocate on the team for their needs.
- #29 Lets take a look at a few approaches you can use to fulfill these role
- #30 Don’t go into detail
- #31 A picture is worth a thousand words. Building visual requirements with lightweight storyboarding tools allows your team to have detailed conversations about how the product should behave inside the team and with business sponsors, end users and other stakeholders. With tools that are easy enough, you can use this technique to rapidly iterate on functionality, acceptance criteria and even test cases before software creation begins.
- #32 As you go through the process of analyzing stories and storyboards you can create a nearly infinite list of test cases. Exhaustively testing all possible test cases is rarely possible, and yet you want to ensure that you deliver a high quality product that delights customers. Prioritization is the key. Identifying the areas to test which will provide the most value to customer should be your guiding principle. Areas of high risk Areas where defects have high impact Areas that are likely to break Areas with many dependencies
- #33 Use the risks and high value targets you identify to map out an approach plan with set of charters that need to be explored. Each charter should map out an area that you need to spend detailed time using critically once the story has been implemented, but does not need to describe a detailed step by step set of test cases. Mapping out these high priority areas to explore provides you the core value of test planning without taking extended amounts of time. It also empowers you while doing exploratory testing to keep your critical thinking skills active and not just mechanically verify a preset test plan. When doing exploratory testing it is helpful to use a tool which can capture you usage and transform it into a detailed bug description so that you do not need to waste time on paperwork. Your time in exploratory testing should be focused on creativity and critical thinking - you want to stay in the zone of productive analysis. If you’d like to see a demo of the Microsoft Test Manager Exploratory Testing feature please stop by our booth.
- #34 As you find bugs it is important to capture them not just as bug reports, but also as test cases. When it comes time to do a full regression test before a major release you will want a detailed list of test cases for each way the product has broken in the past. For test cases which are on core scenarios or test cases which break repeatedly, automate them and execute your automated tests as part of your build process so that the latency between introducing a regression and catching it is minimal. Good exploratory testing tools allow you to create test cases and automation as easily as you file bugs.
- #35 Communications is key to project success, and feedback throughout the cycle is critical to enabling agile value delivery. We discussed creating visual requirements and acceptance tests, prioritizing your work and exploring working software as part of your iteration flow. It is imperative that you take the next step by reaching out to your customer at each stage and requesting feedback on your plan. Work with the product owner to review test definitions and priorities with your stakeholders. As each sprint completes, publish a working build and request your stakeholders to play with it and give you their feedback. To see a demo of the Stakeholder Feedback tool in Visual Studio vNext please stop by our booth.