Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. B E S T P R AC T I C E S July 17, 2006 SOA Raises The Stakes For Software Quality How Software Testing Methods Must Change To Suit Service Orientation by Carey Schwaber with Randy Heffner and Megan Daniels EXECUT I V E S U M MA RY SOA makes software quality both more important and more difficult to achieve. But traditional approaches to software testing are insufficient in an SOA environment. IT organizations pursuing SOA find that they must rethink their testing methods and revise testing roles and responsibilities. TARGET AUDIENCE Application development professional SOFTWARE QUALITY IS EVEN MORE IMPORTANT FOR SERVICE-ORIENTED SHOPS Software quality poses a significant challenge for corporate IT organizations, especially when it comes to the quality of custom applications.1 Defects in any type of application can disrupt the business and waste precious IT resources; in an SOA environment, the consequences of defects can be even more severe. Defects in services can lead to: · Defects in applications that tap those services. An SOA initiative is in many ways an asset reuse initiative, and the success of any asset reuse initiative is directly tied to the quality of the assets being reused. Defects in services lead to defects in applications that consume those services. · Redundant services and lower benefits from the overall SOA initiative. If a particular service is known to have quality problems, development teams will avoid using it whenever they can, since using a buggy service often takes more time than using no service at all. Or, instead of eschewing services altogether, they’ll build ones that they can trust to function properly. The result? Redundancy of services, which reduces the efficiency of the SOA initiative. · Strained relations between the departments, business units, and even companies. Services that live at organizational boundaries are an enterprise’s public face in the digital world, and these services tend to have high usage levels, high visibility, and high criticality. Quality problems in these services can be embarrassing and can even jeopardize business relationships. Corporate IT organizations recognize that SOA requires more rigorous quality practices and are investing accordingly: Our data shows a dramatic correlation between SOA adoption and intentions to purchase software testing tools among North American and European enterprises (see Figure 1).2 Headquarters Forrester Research, Inc., 400 Technology Square, Cambridge, MA 02139 USA Tel: +1 617/613-6000 • Fax: +1 617/613-5000 •
  2. 2. Best Practices | SOA Raises The Stakes For Software Quality 2 Figure 1 SOA Adoption Correlates With Software Testing Tool Purchase Intentions “In 2006, will your company purchase software testing tools?”* 5% 5% Not using SOA 72% 17% 2% 3% Using SOA 44% 41% 8% 4% Not purchasing First-time purchase Minor upgrade Major upgrade Don’t know Base: 540 technology decision-makers involved with infrastructure software or application architecture (percentages may not total 100 because of rounding) * Respondents in the “using SOA” category either “have an enterprise-level commitment for SOA” or “use SOA selectively with no clear strategy.” Respondents in the “not using SOA” category are “not pursuing SOA and have no immediate plans to do so” or “will pursue SOA within 12 months.” Source: Forrester’s Business Technographics® November 2005 North American And European Enterprise Software And Services Survey 39947 Source: Forrester Research, Inc. SOA REQUIRES CHANGES TO TESTING METHODS Enterprise IT shops can’t overcome the software quality challenges posed by SOA just by doing more of what they’re already doing today. SOA requires changes to the way that IT shops test the services they expose and the applications that consume them. IT shops need to test services: · In isolation. Traditional software testing methodologies proceed from unit testing to integration testing to system testing.3 But with SOA, there’s another level to test: the service level (see Figure 2). Just because an application functions correctly as a whole doesn’t mean that there aren’t defects in the services it exposes; someone might have fixed a business logic problem in the UI rather than in the service that contains the business logic error.4 The defect in this service could then reemerge when another application consumed it. To avoid this, shops must confirm that services function properly in isolation. · Earlier in the life cycle. Early testing is a best practice for all types of software, because the cost of defect repair increases with time. But for shops pursuing SOA, early testing is an imperative. Testing services earlier in the life cycle is a good investment because the cost of resolving defects in deployed services can be so high. And the testing of client applications that consume services needs to start earlier, too, as defect resolution can take more time when there are more moving parts to manage. July 17, 2006 © 2006, Forrester Research, Inc. Reproduction Prohibited
  3. 3. Best Practices | SOA Raises The Stakes For Software Quality 3 Figure 2 SOA Requires Another Level Of Testing: Service Testing Scope Sample target Responsible parties Unit testing A method within a Java class Developers who write or modify the method in question Integration Multiple units that interact with each other, The development team testing either within a single class or across classes Service A customer information service delivered in The development team that builds the testing isolation of any application service and an independent testing team System An order tracking application that consumes The application development team and testing the customer information service an independent testing organization System Interaction between the application and other An independent testing organization integration elements of the production environment testing 39947 Source: Forrester Research, Inc. · Along more dimensions. IT organizations frequently forget that there’s more to testing than just functional testing. But services also require performance testing, security testing, and interoperability testing. Because services are built for reuse, IT shops need to establish what volumes of usage they’ll be able to support.5 Security is another important dimension for services testing; balancing security and openness is an especially pressing issue for external services. And ongoing challenges around service interoperability mean that IT shops often need to conduct interoperability testing of services across multiple vendor implementations, too.6 SOA REQUIRES A NEW DIVISION OF LABOR To properly test services and the applications that consume them requires a different division of testing labor within and across development teams and independent testing organizations. Forrester recommends the following model: · Delivery teams are responsible for testing services in isolation. No matter whether they are building services or exposing services in the context of building a larger application, the delivery team has primary responsibility for testing the services it exposes at every applicable level. This team is in the best position to ensure that this testing happens early on, and it’s also more likely to have the expertise to be able to perform this testing effectively. Unit and integration testing of services is of course the natural domain of developers, but responsibility for testing how the service functions as a whole can fall to many different parties. At Standard Life, the analysts who define business service interfaces are the ones who devise business service test plans; developers use these plans to create unit test suites, while analysts use them to build service test suites.7 July 17, 2006 © 2006, Forrester Research, Inc. Reproduction Prohibited
  4. 4. Best Practices | SOA Raises The Stakes For Software Quality 4 · An independent group conducts another round of service testing. Because there’s no way to know where or when they will be used, services should be bulletproofed with an additional round of testing by an independent testing team — whether internal or even outsourced.8 This organization should test services in isolation, even when the delivery team has already done so. This is not common practice today. But as the need for independent verification and validation of services becomes more apparent, enterprise IT shops will improve the technical expertise of their centralized testing organizations so that they can better perform this function. · Responsibility for testing applications in their entirety stays where it is today. Even with specialized testers dedicated to service testing, other testers will still feel the impact of SOA; testing at the application level is more complex when an application consumes services. If an online banking application returns incorrect customer data, testers must be able to trace the error to a customer information service so that the owner of that service can diagnose and repair the defect. This requires an understanding of the services an application leverages and how the application interacts with those services. No matter where the testers conducting system testing report today — to application delivery teams or to an independent testing organization — they will be affected by their organization’s adoption of SOA. SOA REQUIRES MORE SPECIALIZED TESTING SKILLS In addition to requiring changes to testing methods, SOA also requires different testing skills. Even outside the world of SOA, testing requires both technical and business acumen. But in an SOA environment, testers need more of each. To properly test services, testers need to: · Be able to navigate application innards. Many testers interact with applications almost exclusively through their graphical user interfaces (GUIs), comparing expected results to observed results and alerting developers when the two do not correspond. In an SOA environment, this is insufficient. To test at the service level, testers have to use test harnesses that directly invoke services, rather than merely invoking them through the GUI. For the many testers who find basic GUI test automation challenging, testing services poses a significant challenge. · Understand how well services embody business processes. Testers must keep an eye on the fidelity of business services to the business process they embody, as well as on services’ interfaces to other and larger business processes.9 To do so requires a strong understanding of business operations. A mortgage rate calculation service has to employ the right algorithms for calculating rates, and it takes a subject matter expert to define test cases that verify the algorithm works properly — and to do so in the context of all the different business processes in which it might be used. July 17, 2006 © 2006, Forrester Research, Inc. Reproduction Prohibited
  5. 5. Best Practices | SOA Raises The Stakes For Software Quality 5 Testers have always been expected to have both technical expertise and business expertise, but in practice, testers often don’t have enough of either. Instead of forcing testers to straddle the two domains, some IT shops pursuing SOA encourage their testers to focus on one or the other. This type of transition takes time, and until testers develop the technical expertise necessary to perform this testing, service delivery teams will often rely on developers and analysts for these two types of testing. R E C O M M E N D AT I O N S DEFINE AND ENFORCE STANDARDS FOR SERVICE TESTING To ensure that quality problems don’t derail their SOA initiatives, IT organizations should: · Add service testing to the published testing methodology. After identifying the delta that SOA will have on their testing methodology, IT shops must communicate service testing requirements to all service delivery teams, as well as to all application delivery or maintenance teams that might potentially expose services. · Reassess current roles and responsibilities. Enterprises pursuing SOA should devise both short- and long-term plans for the division of labor around testing of services and the apps that consume them, basing these plans on the current skills inventory and on realistic plans for reskilling. This should be a collaborative effort between management and affected individual contributors. · Implement governance mechanisms to enforce service testing requirements. Once service testing standards are defined, fulfillment of these requirements should be required before the service can be checked into a software configuration management (SCM) system or a services registry. These repositories should contain pointers to relevant test assets; delivery teams can use these tests to better understand how each service operates. Alternately, IT shops can add service testing criteria to their release checklists to ensure that the appropriate testing activities are performed before deployment. ENDNOTES 1 Forrester’s data shows high levels of dissatisfaction with the quality of custom applications. A fall 2004 Forrester survey of 692 technology influencers — those who hold the IT purse strings — indicated that nearly one-third are dissatisfied with the time it takes their development shops to deliver custom applications, and the same proportion is disappointed by the quality of the apps that are ultimately delivered. One-fifth of respondents are unhappy on both counts. See the April 11, 2005, Trends “Corporate Software Development Fails To Satisfy On Speed Or Quality.” July 17, 2006 © 2006, Forrester Research, Inc. Reproduction Prohibited
  6. 6. Best Practices | SOA Raises The Stakes For Software Quality 6 2 Forrester surveyed 911 North American and European software and services decision-makers in September and October 2005. Source: Business Technographics® November 2005 North American and European Enterprise Software And Services Survey. 3 The type of testing that sits between unit testing and system testing is traditionally called “string testing” or “integration testing.” This form of testing verifies that units that have already been tested in isolation function together in combination. But Forrester has noted that companies have recently begun to use the term “integration testing” to refer to testing that explores how the application integrates with other elements of the operating environment, including, but not limited to, other applications. Why? Likely because of the increased extent of integration in today’s enterprise operating environments and the increased need to test the effectiveness of this integration. Forrester uses the term “integration testing” in its traditional sense and employs the term “system integration testing” to refer to the testing of how an application integrates with the rest of the operating environment. 4 Services must implement 100% of the business rules that are necessary to ensure the integrity of each transaction. This is a shift from the days of client/server applications, in which the business logic often relied on validations that were performed in the user interface. Here, the thought process was, “We checked this when the user entered the data, so why waste time checking it again?” By contrast, a service-oriented application says, “Trust no one.” Aside from facilitating success with service orientation, this principle produces more secure, resilient applications. See the August 8, 2003, Planning Assumption “Service Orientation: Key Application Design Principles.” 5 Services by definition include quality of service characteristics, including data on the performance, scalability, and availability that the service has been architected to deliver. Deeper forms of service management keep track of how much of a service’s capacity is “spoken for” by various consumers. See the March 29, 2005, Trends “Your Strategic SOA Platform Vision.” 6 In defining internal usage standards, make sure to prototype and test usage across multiple vendor implementations. A bank found it particularly important to test the interoperability of security standards (e.g., WS-Security). See the September 15, 2005, Tech Choices “Real-World SOA: SOA Platform Case Studies.” 7 These plans are used throughout the life cycle as a basis for all testing and are maintained and extended as required. Developers use these plans to create unit tests that are incorporated into the project’s automated daily build, which generates test reports and code coverage metrics. Once services have been deployed to test environments, analysts use another utility to capture a suite of service tests. This facilitates subsequent automated regression tests at the service level. 8 It is significantly more difficult for testers to assess the quality of applications that their own teams have developed. Enterprises often separate testing from development to prevent conflicts of interest from arising. Outsourced testing is a natural extension of that trend, as testers working for a different company are even more likely to view software with a critical eye than testers who work for the same company as the software’s developers. See the May 16, 2005, Best Practices “Software Quality Is Everybody’s Business,” and see the March 8, 2006, Trends “How To Benefit From Offshore Testing Services.” July 17, 2006 © 2006, Forrester Research, Inc. Reproduction Prohibited
  7. 7. Best Practices | SOA Raises The Stakes For Software Quality 7 9 Forrester’s taxonomy of service types divides services into three categories: business services, application services, and infrastructure services. Business services — the services that have the most potential to create business-level flexibility, value, and impact — deliver business capabilities like “submit order” and “view customer activity.” Business-level requirements and considerations are the primary design drivers for the semantics and syntax of business services interfaces. Testers must verify that business services fulfill business requirements and deliver business capabilities as the business intended them to. See the October 25, 2005, Best Practices “A Taxonomy Of Service Types For SOA.” Forrester Research (Nasdaq: FORR) is an independent technology and market research company that provides pragmatic and forward-thinking advice about technology’s impact on business and consumers. For 22 years, Forrester has been a thought leader and trusted advisor, helping global clients lead in their markets through its research, consulting, events, and peer-to-peer executive programs. For more information, visit © 2006, Forrester Research, Inc. All rights reserved. Forrester, Forrester Wave, Forrester’s Ultimate Consumer Panel, WholeView 2, Technographics, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. Forrester clients may make one attributed copy or slide of each figure contained herein. Additional reproduction is strictly prohibited. For additional reproduction rights and usage information, go to Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. To purchase reprints of this document, please email 39947