Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

p_1 col template


Published on

  • Be the first to comment

  • Be the first to like this

p_1 col template

  1. 1. Borland ® Agile Testing Survey Results 2009 - Update W h i t e Pa p e r J u ly 2 0 0 9
  2. 2. Borland Agile Testing Survey Results 2009 - Update Contents Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Delegate Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Agile Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 The Challenges of Tooling for Agile Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 The Role of QA in Agile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2
  3. 3. Borland Agile Testing Survey Results 2009 - Update Executive Summary This survey contains primary research data collected at the EuroStar Software Testing event held in Den Hague, Holland, on November 11-13, 2008, and at the STAREAST event held in Orlando, Florida, on May 5-8, 2009. The survey was given to attendees regardless of whether they used, or were aware of, Agile software delivery methods. This is an update to the 2008 Survey from November 2008 to include data surveyed at the STAREAST event. In total, 394 respondents completed the survey from which this combined and updated report has been produced. Respondents represented many countries, including those outside Europe and the United States, and worked in a wide range of industry sectors and company sizes, from retail, such as eBay; telecommunications, such as T-Mobile; and large scale IT providers, like Cap Gemini and Microsoft. As you will see, the additional data captured further enhances our findings from the previous report. The driver for this study was to understand in greater detail the needs of Quality Assurance (QA) organizations in Agile software delivery projects. Borland is managing its own Agile transformation and has discovered challenges that appear to be consistent with other enterprises who have embarked on a company–wide transformation to Agile. As part of our own transformation, the Borland development team in Austria, who develop our software testing suite known as Borland® Silk™, found it necessary to adapt the automated testing applications so that they could scale to meet the needs of testers integrated into the Agile process. As a result, Borland has now produced a unique set of products to fulfil the needs of Agile and traditional testing, Borland SilkCentral® Test Manager™, Borland SilkTest® with Silk4J® and Borland SilkPerformer®. The following sections discuss the findings of the surveys and reflect the needs of genuine QA practitioners from all aspects of the industry, whether current Borland customers or not. We have summarized what we believe to be the key ‘takeaways’ from the findings of our survey and provided comparisons to the earlier report where differences are highlighted. If you wish to learn more about Borland Lifecycle Quality Management (LQM) solutions, visit 3
  4. 4. Borland Agile Testing Survey Results 2009 - Update Delegate Responses Delegates were asked to give the title that closest described their function within the organization. The majority of responses came from test management positions and quality practitioners. Figure 1: Roles of respondents Agile Experience Delegates were then asked about the level of Agile adoption within their organization. Recent trends have indicated an increase in Agile in the IT industry from 18% to 30% in the last 3 years. 4
  5. 5. Borland Agile Testing Survey Results 2009 - Update Figure 2: Agile experience The survey indicates a much larger growth trend as perceived by this audience of quality practitioners, indeed 61% of respondents declared some experience of Agile adoption. Furthermore, the largest portion, 42%, was currently in the pilot phase of Agile Transformation, and now considers it a viable alternative to current development techniques. The survey also shows that only 23% of respondents had not considered Agile: no knowledge of Agile adoption (7%) or were in an organization where it was not being considered (16%). This indicates that the level of Agile adoption within established IT organizations is gaining pace more quickly than previous predictions. However, it should also be noted that the numbers that have full adoption is low, only 5%, even though this is an increase in the figure from the November report (2%). This reflects two key findings from Borland; I Agile adoption is not a destination, it is a journey. It is in the journey where the benefits are realized and amongst these is finding a toolset to support the Agile organization. I A pragmatic approach to Agile transformation is required, since Agile and traditional projects are going to have to operate in an integrated, collaborative way Agile transformation is a continuous and iterative process, including the adoption of true Agile testing into Agile development to create Agile delivery teams. 5
  6. 6. Borland Agile Testing Survey Results 2009 - Update The Challenges of Tooling for Agile Testing We asked users what they saw as the biggest challenge for testing in Agile delivery today. The options we gave were based upon challenges that we saw during our own Agile adoption, and also those of our customers. Figure 3: The Challenges to testing in Agile The chart shown reflects only those users that have had actual experience of Agile adoption, i.e. those that responded ‘c’, ‘d’ and ‘e’ as seen in Figure 2. Almost one third (32%) responded that they did not see how the testing cycles could possibly cope with the short timescales of Agile iterations. This is indicative of teams that are reliant upon manual testing, inherently slower than automated testing, or a dependence upon tools that are incapable of supporting short iterative delivery. In order for QA to deliver, they require test automation, and they need it to be fast. This is somewhat validated by another 25% of respondents who expressed dissatisfaction with the capability of their tools. It follows that over half the respondents with experience in Agile (57%) identified significant hurdles to Agile adoption because of 6
  7. 7. Borland Agile Testing Survey Results 2009 - Update tooling limitations for automated Agile testing, specifically with the short timescales involved. This indicates a general lack of flexibility, or capability in automating testing processes in Agile today, and the speed with which tests could be generated and executed. When looking at responses from Agile users as to which tools they actually were using, 42% were not using any form of actual automation for their testing (18% nothing and 24% basic office products). However, the numbers using the traditional commercial toolset (19%) and Open Source tools (13%) was comparable. Figure 4: Tools used This indicates that the Agile testing community has considerable exposure and knowledge of current automated testing tool offerings, but still leaves the question as to why still so little adoption of genuine test automation, and a reliance instead on manual, office products. So when asked further about test automation in Agile (Figure 5), it was interesting to see the results and why so many felt they were unable to use automated testing tools. 7
  8. 8. Borland Agile Testing Survey Results 2009 - Update Figure 5: I think Test Automation in Agile…? 37% of the Agile testing community felt that the tools they had did not provide a satisfactory capability for their needs. This is from a community using current testing tools from both the traditional and Open Source community. A further 37% believed that process and test automation should be done, but could not see a way to do it, potentially indicative of why so many users have not made a choice of tool; are relying on simple office tools like spread sheets and word processors or have developed an in-house solution. Looking at these specific results in isolation, we see a total of 74% of Agile community respondents felt either ‘short-changed’ or were given a lack of viable options when selecting a tool to automate their testing. In fact, only 12% were actually happy with the tools they had. Again, indicating a widespread dissatisfaction amongst the Agile testing community with the offerings available. So in order to understand greater what these issues or short-comings are, Borland asked respondents what it was that they really needed from a tool as their top priority. In compiling the questionnaire, Borland considered some of the issues they had themselves been forced to confront, and highlighted the top five as options, with the ability for other answers to be entered. 8
  9. 9. Borland Agile Testing Survey Results 2009 - Update Figure 6: What is needed from a tool The interesting aspect of this chart is that 91% of respondents agreed with Borland on what they consider to be the biggest issues in tool provision. The biggest issues surrounded regression testing and metrics respectively. Many of the issues around regression testing are the lack of use of tool automation which takes away the grind of repetitive testing and allows testing automation to be run overnight without the need for user oversight or operation, in particular, the need for speed. The issue of metrics is a long standing one, giving test management the visibility and oversight of all testing and application quality so that qualitative and quantitative judgements can be made on product quality. 15% of respondents were concerned about the level of effort required to ensure delivery of complete testing within required timescales and the burdens that this then places on QA staff, the need for long working hours, typical of manual testing operations or slower tool automation. The next two issues were about having a tool with the flexibility to actually operate in an Agile way and not wasting valuable time correcting environments and scripts instead of testing, and finally allowing quality to be seen as a team responsibility, ensuring that quality is built into the applications from the start with the involvement of developers and testers working together. 9
  10. 10. Borland Agile Testing Survey Results 2009 - Update The first three issues are typical of any testing environment, whether traditional or Agile in the need to ensure test coverage. Of particular interest are the final two issues, cumulatively accounting for 30% of the respondents. These issues map significantly to the unique demands and needs of an Agile development environment, that is, providing an environment to sustain and support users, rather than hinder their operation in a fast, collaborative team manner. We also asked users with Agile experience about programming languages used. It is widely considered appropriate, and even important, that the test scripts for Agile testing be written in the same language as the application under test, so we asked which languages was your organization using. There has been a long held assumption that Agile and Open Source go hand-in-hand, principally because both were founded in small development shops wishing to keep costs to a minimum. There is therefore, a high expectation that the use of Java® with Eclipse™ will be the highest used language and resource amongst these groups. Figure 7: Programming Language used From the above chart, filtered to show Agile users only, it can be seen that Java is, as expected, the most widely used language (49%), however, with the introduction of the latest .NET release, there has been a much greater uptake in this area (40%). The economics of .NET programming resources have helped contribute to an obvious, but slow transition to .NET. In fact, a large number of respondents (17%) indicated that they were using both languages. 10
  11. 11. Borland Agile Testing Survey Results 2009 - Update The Role of QA in Agile Our survey not only considered the tools, but also importantly, the people and their roles and responsibilities for quality in an Agile organization. We asked the respondents if they believed that the concepts of a modern QA organization in IT development would change and the result was emphatic. 56% considered that the traditional role of QA will change, and significantly so. These respondents were then asked further, what will be the most important way that the role of QA will change. 11
  12. 12. Borland Agile Testing Survey Results 2009 - Update By far the greatest number (39%) believed that the concept of an isolated QA team will disappear and that instead, QA will become an integrated part of development. While not surprising, as this is considered a fundamental part of the Agile manifesto, referring back to figure 3, the challenges to testing in Agile, 16% believe that developers and testers do not mix. The cultural aspect of this change is a huge challenge to overcome. Borland has experience in this, with testers becoming part of the Agile development teams, creating Agile engineering teams. There was a concern that this would weaken the quality position, but the opposite was found. Instead, the perceived value of quality was raised and became something that was proactively built into the software, rather than treated as an afterthought. It was also interesting to note that the second largest response was that there would be a change in perspective on test automation, from ‘nice to have’ to ‘absolutely necessary’ (23%). So despite the earlier reservations from respondents about current tool offerings, the need for tool automation is still seen as a priority in order to ensure quality software. An interesting variation, however, from the last report, is the increase in belief that testers are going to have to do more to champion the cause of integration testing, user experience and sophisticated test scenarios. This is a reaction to the continued isolation of QA teams from the Agile process and the need to introduce and maintain quality levels in the systems being delivered. 12
  13. 13. Borland Agile Testing Survey Results 2009 - Update Conclusions In a review of these study results we can come to three key conclusions; Agile adoption is increasingly common in the Enterprise, but is still at a relatively immature stage – With 42% of respondents involved in Agile projects that are in pilot, it’s clear that more and more Enterprises are recognizing the business advantages of shifting to Agile, but have yet to still fully realize all of the potential benefit. Of particular concern is the issue of integrating true functional and performance testing into an Agile organization. Automated Agile Testing is recognized as an area of need, but one that is unfulfilled – 57% wished to introduce automated testing into their Agile projects but felt that either the tools were not satisfactory or could not see how testing could be run in such short timescales. 23% believed that test automation was in fact the most important change that QA will have to face as part of Agile transformation. The need for speed is paramount. The role of QA will change as a result of Agile adoption – 39% see a fundamental change in the way that QA sits in the IT organization and therefore the way that testing will operate as part of software development. However, possibly the most important conclusion is that the transformation to Agile is happening everywhere and quality assurance must adapt. QA must develop Agile testing processes that harmonize with current Agile development methods – this will change the way we view QA forever. As a result of Agile testing, QA will become truly about defect prevention, rather than defect cure. Borland is the leading vendor of Open Application Lifecycle Management (ALM) solutions - open to customers' processes, tools and platforms – providing the flexibility to manage, measure and improve the software delivery process. Copyright © 2009 Borland Software Corporation. Borland and all other brand and product names are service marks, trademarks, or registered trademarks of Borland Software Corporation or its subsidiaries in the United States and other countries. All other marks are properties of their respective owners. 26736