How To Integrate Independent Qa To Shorten Development Cycles

Abstract: Many organizations have turned to outsourcing to cut costs. But how can you gain a strategic advantage from an outsourcing initiative, not just cost savings? This white paper will provide executives and managers of software product and service companies with a ―How To‖ guide and best practices for integrating independent Quality Assurance and testing across the product lifecycle to shorten software development cycles.

Technology companies are challenged to innovate faster than their competition, to deliver their applications to market first, and to re-balance their development priorities in order to accelerate revenue generation. Business leaders often view software quality as a luxury - something that can be sacrificed, if necessary, for more functionality, faster development, or lower costs. However, in practice, software development organizations that have a firm commitment to quality can actually improve development cycles, reduce costs, and add new features with greater ease. Organizations that develop low-quality software, whether for sale or for internal use, are always looking backward, spending time and money on fixing defects in "finished" products. In contrast, an organization that builds-in product quality from the beginning can be forward-looking and innovative; it can A study commissioned by the U.S. spend its resources on pursuing new opportunities. Department of Commerce's National Institute of Standards and Technology According to U.S. Department of Commerce, software bugs cost (NIST) found that software defects the U.S. economy an estimated $59.5 billion per year. $22.2 billion cost the U.S. economy almost $60 could be eliminated with improved testing and earlier identification billion annually. $22.2 billion could be of errors. eliminated with improved testing and earlier identification of errors. Customer expectations of flawless execution have become par for the course. In today’s competitive economy, no software vendor can afford the tarnished reputation or unhappy customers that result from preventable software | defects. But when it comes down to traditional methods for assuring software quality, companies are finding it increasingly difficult to give up the valuable time and strategic focus that get consumed by resource-intensive internal test efforts—a problem that grows larger as development projects advance in both scope and complexity. It’s no surprise that many companies are turning, with increasing popularity, to outsourced methods of software quality assurance as a way to ensure the integrity of their products, the satisfaction of their customers, and the long-term success of their growing development initiatives—all while keeping costs down. As a software company, you face a unique set of development challenges— more platforms and configurations to support, more releases to sustain, more testing to be completed, and more products to integrate. Historically, QA is managed at a discrete project level and not coordinated across the product portfolio. As a result, there are several challenges:  Limited strategic direction for QA at an organizational level  No standardization of tools/processes/test environment  No centralized reporting or service execution  A limited view of function level progress/quality  Product-release delays due to excessive testing time

 Lack of quality testing resources, preventing quick application delivery  Product-release delays due to inability to quickly ramp-up QA team when needed  Absence of real-time data for managers to evaluate progress of testing You need tools and practices that accelerate the development process to maintain and extend existing code as well as to develop and deploy new code—while tightly managing the costs associated with numerous members of your development team spread out across multiple locations. Your bottom line is dependent on your development team’s ability to produce quality software. When it comes to product lifecycle management, there are several primary objectives:  Shorten time-to-market timeframes Software companies should rapidly narrow the gap between what the customer needs and what the software delivers by increasing functionality, adaptability, usability, and manageability.  Deliver increased product feature sets There should be a strategic release management plan that optimizes usage of the product engineering and QA team.  Improve product quality Software companies may use the testing and The emerging technology leaders are validation services of an independent team with constantly improving speed, quality, specialized skills that stays focused on delivering and cost parameters at each step of a zero-defect product. the product lifecycle.  Constantly increase software development efficiencies To succeed in the next economic cycle in the software business, companies have to increase efficiencies and productivity, and create and consolidate software assets into reusable components and blocks.  Increase market share and customer/partner retention byfocusing on the strategic activities and the core business Software companies can leverage a thoughtful and balanced offshore strategy across the entire product lifecycle to keep key people focused on strategic activities such as product management, marketing, sales, and customer service.

The Informatics Computer School's Software Engineering course-text defines Software Quality as:  Conformance to clearly stated functional and performance requirements  Conformance to clearly documented development standards  Conformance to implicit characteristics that are expected of all professionally developed software For any organization, defining what you mean by "quality" is an important first step. Too often, software development organizations operate with a loose, general notion of quality and tolerate defects that most engineering disciplines would not allow. In contrast, a solid definition of quality that all team members On average, developers make 100 to understand and accept promotes thoroughness and attention to 150 errors for every thousand lines of detail. In the domain of business applications, we can best define code they write. Even if only a small quality in terms of the target audience: the software users. fraction—for example 10 %—of these Development organizations with a quality focus know that a errors are serious, then a relatively "quality" application must do more than simply provide correct small application of 20 thousand lines results without crashing. Does the application meet stakeholder of code will have roughly 200 serious requirements? Is it usable? Secure? Scalable? Reliable? Easily coding errors. maintained? Easily extended? Simple to monitor? James Juran's definition provides a good starting point: quality is "fitness for use." Juran goes on to say that a product is not high-quality unless it adds value for both the consumer and the manufacturer. Put slightly differently, quality encompasses added value plus attention to detail. Both a luxury car and an entry-level car will get you from point A to point B. But the luxury car offers features and capabilities that go beyond the essentials of transportation: usability, safety, comfort, reliability, and so on. Product quality also reflects the process behind the product. In the software world, a high-quality process can keep development organizations from losing time reworking, re-factoring, and rewriting software. These organizations can produce more innovative and creative products because they have more time to think about adding value and quality details. Achieving quality results requires applying a high-quality process throughout development, integration, and testing. This applies equally to projects involving packaged or homegrown applications, upgrades or greenfield development work, and extending, integrating, and modernizing legacy applications (see Figure 1).

Figure 1: All modes of software development require the same attention to quality.  Design New  Develop Development  Test  Deploy Integrated  Customize  Extend  Expose Legacy Packaged  Expose  Systems Applications Test  Upgrade  Deploy But the bottom line of software quality is conformance to the user's needs⎯or requirements,, or wants. The point is, any definition of software quality must take into consideration the end-user's perspective. An application may do all that it was designed to do, and do it according to professional standards, but if users hate using it, find it difficult, counterintuitive, or error-prone, then its quality is at best questionable. Being free of defects is not enough. For a product to achieve quality, it must also provide exceptional functionality, usability, and compatibility. Software Quality Assurance cannot be treated by executives, managers, or engineers as one additional "Task" on the way to the final launch, or a thing that can be taken care of as the project proceeds, along the development and release process. Team members closely associated with the application "Rigorous inspections can remove up are less likely to find the problems that an external party would to 90% of errors before the first test find easy to spot and report. Independent testing efforts ensure case is run, but rigorous inspections that your product is tested by individuals who will be just as should not replace testing" - Robert L. objective and unforgiving as your customers in the case of a Glass, "Facts and Fallacies of potential defect. Software Engineering" This is simply a fact, and not a judgment of or reflection on the developers' talents. An individual who spends the most time working on a project intended for an audience cannot simply step back and look at it as though through that audience's eyes. The creator is too familiar with his work to view it objectively.

But this is why Quality Assurance experts exist⎯just asprofessional proofreaders, copy editors, test audiences, and software do. These independent parties bring much-needed objectivity to the appraisal of a software development project. It is crucial that software Quality Assurance be independent of project management for it to be completely effective. Outsourcing software Quality Assurance to an independent, unbiased source is the most effective way to achieve the objectivity that a project needs. When it comes to the practice of independent verification and validation, an independent QA team should be the one final check-off point before allowing a system to go into operation; how it is done systematically and organizationally varies, however. The standardization of software engineering processes, margin compression, and competition are compelling technology-centered companies to face the task of delivering more complex application development with limited and decreasing time frames. Software leaders are looking at outsourcing processes across the entire software development lifecycle, from engineering, to quality assurance and testing, to maintenance and porting. Software Quality Assurance outsourcing can take a number of paths, testing for different capabilities and performing different analyses, depending on the demands of the project, the users, and the software itself. But an outsourced Quality Assurance and Testing initiative carried out by seasoned QA professionals will confer certain benefits: 1. Reduced Development Cycles 2. Improved Customer Satisfaction 3. Reduced Cost of Development 4. Reduced Cost of Maintenance Coupled with an outsourced business model and its benefits, outsourcing software QA and testing became a strategic R&D initiative for many technology companies. The expected benefits include: 5. Improved focus of in-house resources on strategic activities and innovation 6. Reduced Total Cost of Ownership (TCO) 7. Reduced technology risks According to Gartner (2005), “20% of So what is the best way to leverage QA and testing to shorten outsourcing deals do not produce cost development cycles? Build your own QA and testing department? savings, with 50% of all outsourcing Create a hybrid of in-house and outsourced resources? Or projects falling short of delivering the outsource entirely? expected value and being deemed unsuccessful". As a software vendor, innovation is the foundation of your success. The earlier in your software development process that you apply world-class testing and quality assurance practices, the more efficient your process will be. Advancing other critical path activities, such as internationalization and

localization, further upstream in the product lifecycle will also serve to improve efficiency. Software makers who manage their product lifecycles as carefully as the process-mature manufacturers in, say, the automotive industry, are sure to be winners as the high-tech industry moves into the next economic cycle. Outsourcing the testing activity, properly applied, can address many of these issues while relieving management of many of the burdens of process development, staffing, and ongoing control. The reasons for outsourcing are many and varied among potential customers. Depending on the nature and maturity of the company, these could range from the need for limited special purpose testing to taking over quality management of a product/application portfolio. The former might be done to address specific gaps in the testing framework, such as a lack of environments for conducting load testing, or areas of particular risk such as security. The latter may be advisable if your internal resources are unable or have limited capabilities to justify, explain, or quantify existing internal testing activities. As with any business decision, the choice to outsource and/or offshore software testing must be viewed in terms of risk and reward. While this will vary from company to company, a few basic principles should be applied when considering such a solution. The first step should be to articulate the goals of the initiative. The basic rationale for undertaking this should be stated explicitly and frame the entire process. A formal strategy statement may also be valuable here to lay out a roadmap for proceeding, articulating critical risks and assumptions, estimating resource requirements, and determining measures of success. Some of the possible reasons for deciding to outsource include: "...the average enterprise will 1. Deficiencies in the specialized skills and ultimately outsource 60% of processes required to support testing application work offshore (circa 2008/09)."— META Group 2. Problems related to poor test practices such as excessive expenditures on maintenance and critical failures in production 3. Lack of test infrastructure including test environments and test beds 4. Insufficient testing activity to warrant a full-time testing staff because the portfolio is too small or there are only a limited number of releases during a year 5. Lack of facilities or expertise to conduct special purpose testing 6. Independent validation and verification of development work performed by other vendors One other area to note where an organization might gain significant long-term benefits from outsourcing is in implementing test automation. Manual testing and tracking is a complicated enough task and, although using automated test tools can help improve efficiency and control, they are not an immediate panacea. Such tools must be carefully selected to match the technology and environments used by a system. The tools must be set up, scripts should be written and maintained by experienced professionals who know how to configure the package, set up databases, design test artifacts, determine and enforce privileges, establish library control procedures, train staff, and enforce standards. By engaging a service provider who knows how to set up and manage these tools, a

company can gain valuable experience in using them without some of the pain associated with starting up such an initiative internally. The final aspect of making a decision about whether or not to outsource Quality Assurance and testing is to determine the organization’s readiness to support testing. An evaluation of existing processes, skill sets, and infrastructure is recommended in order to set the stage for planning a test outsourcing initiative. An important part of any such assessment is how the existing QA or testing organization will interact with the vendor. The chapter entitled, ―Five Phases of Integrating QA Into the Entire Global Software-development Cycle" provides step-by-step guidelines for how to design and manage your Quality Assurance and testing program. Based on the goals articulated by management, a formal framework for moving forward should be developed. Some potential models for outsourcing that might be considered include: 1. Total off-site outsourcing of all testing functions for the entire portfolio including offloading the existing testing staff to the vendor. 2. Conducting outsourced testing on a pilot basis with the intention of developing internal processes or introducing automation. 3. Partial outsourcing of special purpose testing (e.g. performance or security) or testing level (e.g. outsourcing system integration testing but not unit testing or user acceptance testing). 4. A vendor-managed internal testing program in which all work takes place on site, applications remain in the customer environment, test beds are kept in repositories controlled by the client, but the test management and staffing is provided by the vendor. 5. Third party validation of outsourced products, involving independent testing of applications built by another vendor. Total off-site outsourcing has the advantage of reducing internal staff expenditures while improving the efficiency and effectiveness of the testing function. It leverages the knowledge of the vendor to the largest extent. This model tends to work best for customers who have well-documented requirements and well-educated stakeholders. In the absence of those conditions, there may be a ramp-up period in which the vendor will have to spend time organizing requirements and familiarizing internal staff with their roles. In addition, the relationship between the vendor and developers must be managed carefully; delays in the delivery of software will affect testing schedules and the amount of testing, which can be performed. The final consideration in a pure outsourcing arrangement is the fact that the product will be under the control of the vendor during testing. Arrangements must be made to assure the security and integrity of programs and data during this time. A risk worth mentioning with a pure outsourcing approach is that the client can become out of touch with the quality management of its own products. It is incumbent on the customer to review, interpret, and act on test artifacts and test results even if the entire testing operation is run by an off-site test team.

The major benefits of running an outsourcing pilot are that targets and costs can be controlled selectively and future strategies can be modified based on the outcomes and findings from the pilot. When choosing pilot applications, it is usually wise to find those of enough significance and size to make the activity meaningful but not those which are business-critical. This allows clients to learn important lessons about testing applications in their domains without putting core systems at risk. Establishing predetermined criteria for the success of the pilot is vital to extracting the proper lessons from it. Partial outsourcing of special purpose tests or by testing level can be an extremely effective way for an organization with existing test infrastructure to enhance its quality management program. By engaging vendors with extensive A vendor-managed internal testing program is a hybrid business experience in a particular specialty, model, which can serve multiple needs. In this scenario, all work such as performance testing, a takes place on-site and applications remain in the customer company can leverage its knowledge environment. There is greater control and much less risk of and environments, reduce investment compromising systems or data. The vendor may design and in hard to find skills, minimize the create testing procedures or use existing processes, but they can need for expanding test infrastructure, bring additional value to the organization by working with internal and receive independent verification of best practices and enhancing them. The vendor may also develop results. processes for controlling test beds, however it is recommended that the repositories are designed and managed by the client’s infrastructure team. Since the test management and staffing is provided by the vendor, the customer does not have to invest in finding the skills needed to establish the test infrastructure and staff the test teams. One other approach that is gaining popularity is third-party validation of applications built by another vendor. This trend is pronounced with mid-market clients who have decided to outsource most of their IT function. It can be a cost-effective option for organizations with little experience with software development and testing or for those who have very few or small software projects. While it does not contribute to the client developing its own testing processes, it is a good model for those who do not wish to invest in that infrastructure. In addition, the test cases and test data are the property of the customer and can be reused in the future should the outsourcing relationship be discontinued. One issue that may arise when adopting this approach is contention among vendors. Performance of the testers can be affected negatively by developers who do not provide good test requirements, cannot provide feedback on tests in development, or fail to deliver on time. The client in this situation must maintain control of the process and must take steps to assure that the vendors are working in a manner that supports the company’s goals. At the end of the day, how does an executive determine the value of outsourced testing and whether it is worthwhile to undertake? Potential buyers should realize that the benefits to be derived from test outsourcing may go beyond those gained through outsourcing other functions. Those who have survived in the tough conditions of the recent IT economy are well aware of quality issues and the need to deliver fully tested products. The financial motivation to find and fix bugs before product release is not a new notion, however. A study by noted industry analyst Capers Jones back in 1996 clearly showed the staggering costs

associated with post-release bug detection. Post-release bug detection usually occurs when software companies decide to cut costs by reducing testing schedules in early phases of development. Figure 2. Cost of repairing defects across application lifecycle The fact that closer attention to testing is vital to keeping overall long-term costs down is often obscured by near- term budget constraints. Here we run into a paradox: more testing resources and enhanced QA requirements are paramount to a company's success, but their budgets simply cannot support increased spending. There is a solution to this seeming contradiction. As Steven Brier, senior analyst and managing director of research at Roberson Stephens, notes, the secret to success in the IT landscape these days is to focus on running the core business while effectively using outsourcing options. "It's often cheaper and faster to give it to someone else to do," says Brier {UPSIDE, June 2001). In terms of quality assurance, this means that developers can stay within their core competencies by unit testing with highly specialized debugging tools, such as jtest® from ParaSoft®, while outsourcing functionality, compatibility, and integration testing. This allows software engineering organizations to capitalize on what they do best, i.e. write code, while delegating the task of testing how their product functions in the complete solution stack to an outside team with proven expertise in quality assurance. Additionally, in many companies, because of repetitious test processes, the turnover rate for test engineers is very high. This means that an in-house test lab must continuously spend time and money to seek and recruit skilled workers and train them for its purposes. In comparison, outsource test organizations offer clearly defined career

paths to their employees, reducing turnover. Plus, due to the nature of the business, projects in outsourced testing labs are always in flux, which keeps career test engineers interested in working with and discovering upcoming technologies. As a result, these engineers are highly trained, enthusiastic, possess a wide variety of core competencies, and are efficient in carrying out complex test projects. It is important to understand that the savings realized through more effective testing don’t automatically fall to the bottom line. Even if an outsourcing arrangement does directly reduce the cost of testing, most IT organizations will probably still spend their up to their budget limits. The savings can be and usually are spent on development and other productive tasks in support of core business functions. But some of the payback of instituting an effective testing program is derived in other ways, including:  More efficient staff and resource management  Ability to investigate test automation in a low-risk setting  Reduction of time spent on rework and repair  Minimizing and managing product risk By enabling the software engineering department to make more effective use of the internal resources, both testing and development benefit. Developers can focus on the job of developing the software without worrying about planning, designing, and executing tests. This can shorten lead times to get products to market and result in better systems because developers will be focused on just one task - engineering - which has to be the only core competency. Dedicated, trained testing resources who understand the overall framework of quality management will be much more effective in constructing and implementing an organized approach to validating software. The testing process will be more efficient and effective. First of all, highly trained specialists will be better at this task. Effectiveness will also be enhanced since an outsourcer is more likely to provide an unbiased assessment of product quality simply because they are operating independently from the developers. Finally, the costly and time-consuming task of setting up and managing test environments will no longer be a factor in determining resource allocations. A good provider of testing services should either have the appropriate hardware and software infrastructure in place or be able to generate it at less cost and in less time. The management of special-purpose testing will also be enhanced. Performance and stress testing can be done with more breadth and depth. Regression testing can also be enhanced. The key to effective regression testing is to maintain existing test scripts so that they can be re-run when changes are made. However, most companies don’t have experience in determining which scripts to keep, how to store them in an organized fashion or when to re-run them. A qualified service provider will have these skills on staff or make them available, when needed. Outsourcing to a service provider who has extensive experience with automated test tools also provides a buyer the benefits of working with a partner knowledgeable in this field. They can assist with defining the tool set to be used, installing and configuring the product, managing it on an ongoing basis, and training staff in its use. Automated tools can also be used to improve regression testing, as they provide facilities for storing and managing test scripts and test data. When one considers that a significant proportion of test tools purchased are never used effectively by the buyers due to lack of experience, the return on investment is often gained simply by using the very products they purchase. Then there are the benefits of better product quality. Put simply, it is advantageous and economical in the long run to test software systems effectively to assure that they meet their requirements rather than correcting them later.

The amount of time and money spent by software engineering departments on application ―maintenance‖ can exceed 50% of the annual budget. This calculation also does not take into account the overall cost of poor quality which includes the time and resources required to identify defects, determine their location and cause, repair them, retest them, and promote them again into the production environment. Finally, there is what may be the largest cost of poor quality: business lost and the company’s reputation tainted by exposing end-users and customers to systems, which cannot perform their functions accurately and dependably. This could result in inaccurate order processing and billing, unexecuted transactions, miscalculations, inability of customers to do business during high volume periods, generation of inaccurate reports, etc. Any one of these circumstances can cause customers to discontinue their relationship with a company, and they may very well convince their acquaintances to do so as well. The last major benefit to be derived from working with a vendor to establish a sound testing program is better information to support management decision-making. Prioritizing requirements provides a sound basis for evaluating product risk and determining areas for concentrating quality management or improvement programs. Measurements of test completion and coverage represent necessary information for determining if and when products go live. Root cause analysis of incident reports can lead to significant improvements in the product definition and development process. A vendor can make recommendations for making the development effort more efficient and improving the quality of its deliverables. It is this benefit, which is not immediately quantifiable in monetary terms, which can contribute to the long-term success of the enterprise by helping senior management assess risk, set priorities, establish product improvement programs, and generally provide a real-world basis for evaluating investments in technology. The bottom line? In some cases, development organizations prefer having their own test departments for reasons that go beyond financial considerations. For one, they believe close coordination between test and development resources requires proximity. In these instances, a hybrid solution of outsourced test specialists working on-site, backed by external lab-based testing, may be the optimal solution. In either case, today's budget-minded and high velocity organizations avoid missed time-to-market opportunities and enjoy lower over¬all costs of quality assurance through testing outsourcing. Whether you are a software product company and sell software or develop software to use internally, it is important to begin with the discovery of business opportunities across the application lifecycle. Choosing the testing processes that software companies want to outsource is a critical component of their outsourcing initiatives and can have a tremendous impact on their success. The stakeholders should take the current inventory of development and testing processes, resources, and tools before they start identifying objectives, expectations, and metrics to measure the success of the company.

To forge a successful outsourcing strategy, companies should conduct an outsourcing ―readiness‖ assessment. The assessment should be equally applied to companies considering outsourcing processes across the entire product lifecycle, or to those who wish to outsource QA and testing only. The results of the assessment will provide an organization with much of the information required to make key decisions and to focus management effort. The assessment can help examine the ability of a software development organization to outsource by looking at the processes maturity, experience, tools, and ROI objectives. Data for assessment may be gathered from: 1. Questionnaires 2. Interviews 3. Discussion with product managers and application owners 4. Study of product roadmaps, test plans, test coverage, test scripts inventory, and application/code base inventory Deliverables may include: 1. Outsourcing Readiness Report 2. ROI analysis 3. Application/Project Sequencing Plan 4. Engagement Roadmap When reviewing test plans, test coverage, and test scripts inventory, the following criteria are recommended to assess readiness of specific processes: 1. Skill alignment with current/future quality assurance and testing direction 2. Current cost of testing/quality assurance 3. Application maturity 4. Alignment to business (core, context, utility) 5. Business value 6. Functional completeness 7. Technical completeness 8. Rate of change 9. Test coverage 10. Perceived risk profile

Successful completion of this phase will provide the company with insight to areas of the software quality assurance and engineering, which have room for improvement. Gap analysis naturally flows from the results of the Assessment/Business Case phase and involves determining, documenting, and approving the variance between business objectives and current capabilities. At this stage, companies should analyze gathered materials with the goal of comparing its actual performance with its potential performance, and of finding gaps that represent opportunities for improvement. The objective is to study the differences between the current and desired state/best practices of the quality assurance and testing processes, for the purpose of determining how to get from one state to a new state. Gap analysis is undertaken as a means of bridging that space. Among the various methodologies used to perform gap analysis is IDEF, a group of methods used to create a model of a system, analyze the model, create a model of a desired version of the system, and to aid in the transition from one to the other. With that information in hand, a company can pinpoint and address the quality/process gaps. The ultimate goal is to identify and compare a list of initiatives, which you can begin implementing to improve the speed, quality, and cost parameters at each step of the product lifecycle. In this step, companies define a high-level project plan and a test strategy for reaching the desired QA destination. The following steps outline the major components of a typical outsourced QA project. 1. Overview –Includes the objectives for QA on this project and background information. 2. Scope – Contains what will be covered and what will not be covered in the QA effort. 3. Assumptions – Includes any assumptions made in developing the QA Strategy. 4. Project Organization – Contains the project organization chart and how the QA Team fits within. Roles and responsibilities respective to QA and other stakeholders. Specify Project Team responsibilities for test infrastructures and data. 5. Project Steps – Address how the QA effort will be approached. Include findings from Proof of Concept. Include one or more schematics, along with written descriptions of the proposed QA workflow. Describe types of data and validation methods. 6. Levels of Testing – Define what types of testing will be performed such as Unit, Integration, System, Data Validation, Performance, GUI and the entrance and exit criteria for each. 7. Testing Groups - Testing groups provide a breakdown of major functionality. List the testing groups and clearly state what they include and what they do not include. 8. Technical Assessment – Content depends on current status of development, technical infrastructure, and type of technologies and platforms. In this section, the following should be done: Assess compatibility of data structures along with any code or software specific issues. Address feasibility of test automation with recommendations for using home-grown or vendor tools at certain

stages of the development cycle. Include recommendations regarding what should be automated for testing. 9. Test Infrastructure Requirements - Assess existing infrastructure processes and needs. Discuss test infrastructure considerations and requirements for loading converted data and ongoing data load processes. Include environment sizing considerations and depth of test data. Clearly state how the test infrastructure is and is not analogous to the production infrastructure. Note any exceptions or limitations. 10. Processes and Tools – Describe processes and tools to be used for change control, configuration management, and discrepancy reporting. Describe what test automation will or may be used and how it will or could be performed. 11. High Level QA Project Plan - Schedule with tasks and associated resources for successfully completing testing. After implementation of the project, companies have to return to their initial assessment and gap analysis to compare the results of the project with the initial expectations.

 What to Automate  Quality of Software  What Tool to Use  Impact of Failure  Nature of System  Manual Tools  Likelihood of Failure  Strategic Significance Test Tools Risks Application Test Test Objectives Infrastructure Test Strategy  Demonstrate Usability  Code Control  Intrfaces  Enviroments/Progression  Coverage  Hardware  Stress/Load  Software Review Test Resources Strategy Requirements  Money  Walkthroughs  Completeness  Time  Inspections  Accuracy  People  Desk Checks  Stability  Skills  Priorities  Hardware  Software

During the execution phase the following metrics are used: 1. Planned test execution versus actual: are we on track? (could identify a risk for being on time) 2. Number of successful versus not successful tests (too low means inefficient testing due to low quality of the test object) 3. Trend in outstanding number of defects (going up could mean a risk for being on time) 4. Burn rate of the test hours per phase (too high could identify a budget issue) The metrics #1, #2, and #3 are essential to the success of an outsourced QA process and should be tracked weekly. These metrics are mandatory for Quality Assurance projects with a duration of the execution phase longer than two weeks. By applying criteria to the metrics during a running test project, important signals can be derived as to whether the outsourced testing project is going well and whether the quality of the test object is reasonable. Towards completion of the project it is advisable to add the following metrics: 1. Has the test coverage reached sufficient level? 2. Is the percentage of successfully executed tests high enough? 3. Are the remaining defects below a certain level? These metrics play an important role in the acceptance process. When the execution phase is finished some additional metrics can be determined: 1. Number of hours spent on test execution per test case 2. Number of hours spent on test execution per defect found 3. Number of hours spent on testing as percentage of the whole project When these metrics are collected for a number of comparable projects, they play an important role for supplier management purposes (how efficient and effective is a test supplier) and in general to drive improvements. When the project is over, it is recommended that companies analyze defects found after the deliverables were accepted:

1. Number of defects found after acceptance compared to before acceptance 2. Test specification progress per week (number of test cases written) versus planned 3. Defect solution turn around time (are all showstoppers solved within a week?) 4. Defect root cause analysis in all phases (are there particular weak spots in build?) The following mistakes are the most common when it comes to outsourcing Quality Assurance and testing activities: 1. No ownership of product quality. 2. No overall test program design or goals. 3. Non-existent or ill-defined test plans and cases. 4. Testing that focuses narrowly on functional cases. 5. NO ad hoc, stress, or boundary testing. 6. Use of inconsistent or incorrect testing methodology. 7. Relying on inexperienced testers. 8. Improper use of tools and automation, resulting in lost time and reduced ROI. 9. No meaningful metrics for tracking bugs and driving quality back into development. 10. Incomplete regression cycle before software release. Future State, and Required Roadmap Software vendors often make decisions about what and how to outsource too quickly, without detailed expert assessment of their current state, desired future state, and required path on how to get there. It is critical to plan outsourcing strategy by considering a company’s current software engineering processes maturity, in-house team size, available documentation, and experience with distributed software development, cultural fit, and executive and staff support. It is recommended to access applications, source code base, and business processes, and to lay out a matrix of how and in what sequence applications and processes can and/or should be outsourced, kept in-house, or retired.

The cost expectations that determine the savings that organization can leverage as a result of outsourcing are often over-estimated. Although they can be based on ROI projections, software vendors should consider the learning curve for the offsite team to become productive. For several years now, the business press has been claiming that IT work costing $40-$80 an hour in the United States can be done for $15-$25 an hour in India or Russia. If those figures sound too good to be true, that’s because they usually are. An offsite team often will not become productive (by onshore standards) for at least three to four months, or even longer for more complex projects. IT executives should expect to pay an additional 5 percent to 15 percent on managing an offsite outsourcing program, at least during the first year. The transition phase will add costs, too — planned expenses should cover due diligence, transfer of knowledge, and extensive program management. Once a company has decided to outsource, it usually rushes to get the outsourcing deal done and put the transaction on a fast track. To do that, executives from the company and the outsourcer meet, agree, and then let the next level of management sort out all the details. Since an outsourcing process is a complicated one, it is advisable that various elements of a company remain involved in the original meetings and have a say in the agreement. Top-level executives alone are rarely sufficient to review such a process effectively. The essence of outsourcing is that a company transfers the process of application engineering to the outsourcer, and then buys the results of that process. The quickest and most certain way to destroy an outsourcing process is to dictate to the outsourcer how the process should be done. When a company tells the outsourcer what to do, it removes from the outsourcer the ability to add value to the process. As the result, the outsourcer cannot deliver quality software. Many companies choose not to transfer the ownership of the outsourcing process to the outsourcer, and instead opt to keep the process under company control. When the company assigns someone to oversee the process, this person usually becomes overwhelmed and is unable to manage it, because it is an area that he or she has very little knowledge about. As a result the outsourcing process fails. The longer your contract terms, the more deep you get in it. Information-driven enterprises and ASPs should construct a contract that will last a long time, but that will allow the service term's agreements to change during short periods of time. It is preferable for a company to develop a close relationship with an outsourcing company, but to sign short-term contracts that can be easily negotiated and, if necessary, renegotiated or even broken.

Companies new to the outsourcing process often do not assign the right people to manage the process. They often assume that since all the details of an outsourcing contract are discussed and the contract is signed, that they do not have to worry about the outsourcing process any longer. The right person for the job is someone who understands the process, is able to focus on soley on the process, and who can interpret the results with the best interest of the company in mind. He or she should have a different way of thinking, in respect to most business managers, and the oversight to recognize any problems or mishaps before they surface. Far too often, the company forgets that the outsourcer is a business asset and must be treated as such. Team members across the globe are vital parts of a successful and prosperous business. If the outsourcer is treated differently— for example, like one of many suppliers or one of many customers—then the one thing that the outsourcer adds to a company will be lost.  It is preferred that companies inspect in person the physical premises where the software is to be developed. This is an opportunity to check the security firewalls of their vendor’s buildings and work area, the organization culture, the functioning of their networks, etc.  It is preferable that companies interview the team members who will work on their project. This will help companies to judge the employees' level of reliability.  Companies should also check the offshore development company’s employee retention rate, and also determine if the outsourcing vendor is working with any competing organizations. If this is the case, companies should ensure that the teams working on their competitors’ projects do not have access to their project information.  Any method of information transfer, such as e-mails, fax, electronic file exchange, instant messenger, on-line meetings, paper documentation etc, should have parameters for usage clearly defined.  All activities that will have to accompany the end/termination of the contract should be defined while negotiating the contract terms and conditions. This includes the retrieval of any methods and procedures, documents, source and executable code, company proprietary security or development standards, code libraries, and data stored offsite.  Contracts should be framed, such that the offshore company takes responsibility for the actions of its employees.

 Companies should ensure that any project-related work is not subcontracted without approval. This will help to protect a sensitive application or data within the application.  Companies should ensure that the vendor agrees not to use any of the company’s confidential information for purposes like sales, marketing, or demo without prior approval.  Companies should ensure that only data related to the performance and reliability of the system is transmitted over the Internet. Information from the database should not be disturbed during any part of the project.  Companies should make sure that the system experts make modifications to the system only after obtaining prior permission.  Exchange of passwords and other critical information should be made secure by encrypting them.  Companies should ensure that the data used during testing does not expose the real information of the customers.  Unwanted data should be destroyed.  Companies should ensure that the vendor reports on replacements of team members, if any.  Companies should ensure that an original copy of the source code is maintained. The business benefits of including quality-oriented activities in all phases of your software development lifecycle are both broad and deep. These measures not only facilitate innovation and lower costs by increasing predictability, reducing risk, and eliminating rework, but they can also help to differentiate your business from its competitors. Most importantly, continuously ensuring quality will always cost less than ignoring quality considerations. Renat Khasanshyn is CEO of Altoros Systems, LLC. www.altoros.com

1. James M. Juran and Frank Gryna, Juran's Quality Control Handbook. McGraw-Hill, 1988 2. Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement, Jeff Tian (Wiley-IEEE Computer Society Press, 2005) 3. Managing Successful IT Outsourcing Relationships, Petter Gottschalk and Hans Solli-saether (Jul 2005) 4. "Software Quality Assurance: Principles And Practice" by Nina S. Godbole Alpha Science International, Ltd (2004) 5. Test Outsourcing: Risks and Rewards, 2006 Rich Yannetti 545 Fifth Avenue 4th Floor, Ste 401 New York, NY 10017 6. Metrics for Outsourced Test Activities, Kees Blokland / Polteq IT Services B.V. (NL) 2004 7. Data Warehousing QA Challenges, Denise Copening, Olenick & Associates 8. http://en.wikipedia.org/wiki/Gap_analysis 9. Software Quality Assurance Improvement Plan: EPIcode Gap Analysis; May 2004, U.S. Department of Energy 10. Facts and Fallacies of Software Engineering; Robert L. Glass; Addison-Wesley (2002) 11. See Watts S. Humphrey's article entitled "The Quality Attitude," published in the Software Engineering Institute (SEI ) newsletter, (2004, no.3): http://www.sei.cmu.edu/news-at- sei/columns/watts_new/watts-new.htm

How To Integrate Independent Qa To Shorten Development Cycles

by Altoros Systems, Inc. on Aug 19, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

How To Integrate Independent Qa To Shorten Development Cycles — Document Transcript