Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
                                   Volume 3


                   1 Mobile applications
                     and regression testing


                   1 The challenges
                     of testing wireless
                     mobile appl...
bad news, however, is that these jobs         in 2004, and were able to increase
                 are harder than they use...
Each organization will have to            application, our experiences might be
                 answer these questions in...
was 20 years ago. Let’s see, you have       ing on other partnerships as well,
                 Symbian, Blackberry OS, Wi...
There are also a variety of options         a recent Keynote release.
                 for Quality Assurance (QA) managers...
center within IT, and efficiencies in                   third-party vendors will likely pay off
                 the devel...

                   1 Regression testing
                     hot spots: Coverage,
sion test is the execution of tests to       tial production issues. The following
                 verify that changes in...
refers to the current maturity of the      the Tester. The Tester should accom-
                 solution set. Exists as i...
the tester. The tester should once          test the business functionality
                 again accomplish this by impl...
scripts should be constructed to be         required to be built and, more impor-
                 reusable test artifacts...
Therefore a board needs to:                     and monitoring this use to achieve
to support risk evaluation. The organi-     coverage to satisfy the current IT
                 zation should have already...
set of artifacts to be applied during                           to right size the Regression test
                 each re...
Upcoming SlideShare
Loading in …5



Published on

  • Be the first to comment


  1. 1. July/August 2009 Volume 3 SOFTWARE TESTING This issue of Software Testing delves into mobile application and regression testing. • CHAPTER 1 THE CHALLENGES OF TESTING WIRELESS MOBILE APPLICATIONS • CHAPTER 2 REGRESSION TESTING HOTSPOTS
  2. 2. EDITOR’S LETTER 1 Mobile applications and regression testing a EDITOR’S LETTER p WANT JOB SECURITY? This issue of peculiarities that require great devel- Software Testing explores the rewards opment creativity and more savvy a of being thoroughly-modern, thor- testing. CHAPTER 1 THE oughly-trained and always-diligent In this ezine’s article on testing and CHALLENGES mobile application and regression retesting defects, David W. Johnson OF TESTING WIRELESS testers. connects the dots between mobile MOBILE APPLICATIONS In the mobile application develop- application and regression testing ment market, the role of regression by citing the Blackberry blackouts testing is proving to be mission criti- of 2007 and 2008, which were “an a cal in the Web 2.0, Rich Internet unfortunate example of production CHAPTER 2 REGRESSION Application age. Both mobile apps issues that should have been detected TESTING and regression go better with agile by a regression test suite in combina- HOT SPOTS: COVERAGE, and Software Development Lifecycle tion with other formal testing efforts.” COMMON MISTAKES (SDLC) approaches—as well as some Helping testers avoid common mis- creative thinking—so just being a takes is Johnson’s key goal in this arti- code wizard or pretty good tester cle. Too often, a user reports a bug; won’t get you the gold. regression testing is done; a “fixed” Those who get and stay in the mo- report goes to the user; and the user bile application training circuit could experiences the same problem. John- have job security for many years, son suggests practices that can help according to Enterprise Management end this embarrassing scenario. Associates (EMA) Julie Craig, who Regression testers, like mobile wrote this issue’s lead article. Just application testers, must invest in don’t expect to quit school at any continuing education due to the con- time, because the mobile technolo- stant emergence of new breeds of gies will continue to be complex and software, such as mobile applications. varied, with new devices appearing This issue is a good first step on that faster than bubbles in seltzer water. training trail. I Beyond discussing the great career potential in mobile application test- JAN STAFFORD, Executive Editor ing, Craig lays out the mobile app 2 SOFTWARE TESTING JULY/AUGUST 2009
  3. 3. CHAPTER 1 1 The challenges of testing wireless mobile applications Mobile application testing requires not just skill, a but creativity and resourcefulness. It also requires products EDITOR’S LETTER and services specifically designed for the challenges of mobile technology. BY JULIE CRAIG a CHAPTER 1 THE CHALLENGES OF TESTING WIRELESS MOBILE APPLICATIONS p MOBILE DEVICES ARE hot, hot, hot, Desk software using the Web browser and applications are the reason why. on their Microsoft® Windows Mobile, a The lure of always-available Web RIM® Blackberry®, or Apple® iPhone® CHAPTER 2 REGRESSION browsing, email, document viewing devices. TD Ameritrade offers mobile TESTING and instant messaging has proven stock trading software which enables HOT SPOTS: COVERAGE, to be irresistible to business and per- a customer to execute stock trades COMMON MISTAKES sonal users alike. Not to be outdone, from his or her phone., entertainment-related applications a Software as a Service (SaaS)-based are flourishing as well, with mobile Enterprise Resource Planning (ERP) games, Twitter and Facebook, and provider, recently announced in- photo uploads becoming fundamental development software that will pro- elements of the new mobile lifestyle. vide access to Workday’s Human This is reflected in the economy. Resources (HR) and financial applica- Annual global sales of mobile phones tions from an iPhone. Clearly there is in units are now roughly equal to strong demand for a growing host of sales of laptops, with smartphones mobile applications, and as a result, accounting for more than 10% of the mobile development and testing are total. flourishing. Vendors are capitalizing on this This explosion can be a bonanza growth with a host of mobile-enabled for “mobile literate” development and applications. Numara® Software, Inc. testing professionals. The outlook for recently introduced Numara Foot- growth in this space is so favorable Prints Mobile, which allows support that job security is virtually guaran- personnel to access Numara’s Service teed for the foreseeable future. The 3 SOFTWARE TESTING JULY/AUGUST 2009
  4. 4. bad news, however, is that these jobs in 2004, and were able to increase are harder than they used to be, product time to market by two to thanks to a wide array of handheld three times over the industry norm hardware, mobile-specific software, (go to www.enterprisemanagement. and the sheer force of constant com1 for more information on this change. Mobile applications isn’t your story). grandmother’s COBOL, and testing From the perspective of mobile them requires not just skill, but cre- applications, Agile methodologies are ativity and resourcefulness. It also often also the best choice, primarily a requires products and services specif- because of the ongoing focus on cus- EDITOR’S LETTER ically designed for the challenges of tomer involvement and iterative test- mobile technology. ing. The steps required—requirements gathering, design, development, test- a ing, and deployment—are similar for CHAPTER 1 THE MOBILE APPLICATIONS: SIMILAR both connected and wireless applica- CHALLENGES TO CONNECTED APPLICATIONS tions. However, the lifecycle must be OF TESTING WIRELESS Even without mobile technology, soft- tweaked for mobile applications, and MOBILE APPLICATIONS ware development is a notoriously it is important to answer some key failure-prone activity. I don’t need to questions from the outset. go into the details since everybody a reading this is likely aware of the high They include: CHAPTER 2 percentage of software development IWhich mobile devices and REGRESSION TESTING projects that fail to deliver in terms of operating system versions will this HOT SPOTS: COVERAGE, functionality, budget or schedule. application support? COMMON Mobile applications add to applica- I How do we test applications MISTAKES tion complexity, and in doing so, to make sure they run on those increase development and Quality platforms? Assurance (QA) risks. I What modifications must be Mobile development requires the made to accommodate the differ- same Software Development Life ences among platforms? Cycle (SDLC) approach that adds dis- I How will industry innovations be cipline and governance to traditional supported going forward, since new software development. Agile method- mobile devices, technologies, and ologies are rapidly replacing older applications are constantly being Waterfall processes to become the introduced? “de facto” standard governing the I How will development and testing SDLC. Agile principles include incre- processes accommodate the inherent mental and iterative cycles, testing differences among wireless network early and often, and incorporating protocols and mobile service end-user input throughout the lifecy- providers? cle, and they appear to be paying off. I How do we know how much BMC Software adopted Agile starting testing is enough? 4 SOFTWARE TESTING JULY/AUGUST 2009 1 “Agile Management—What Managers can Learn from Agile Methodologies”
  5. 5. Each organization will have to application, our experiences might be answer these questions internally, very different. Factoring such varia- based on their overall goals and tions into expected testing outcomes objectives. can become quite complex. The primary problem is one of test- ing every possible scenario, given that KEY DIFFERENCES few development organizations own Challenges with mobile testing tend one of every device, version, and car- to center around several key areas. rier connection. And even the most a One is the wide variety of mobile plat- talented QA team will probably not be EDITOR’S LETTER forms and the complexity around able to simulate every combination of testing an application’s compatibility these variables, especially when the with all supported platforms. Another idiosyncrasies of users are factored in. a is the challenge of developing applica- Fortunately, a lively marketplace has CHAPTER 1 THE tions that compensate for the facts arisen around mitigating these chal- CHALLENGES that the mobile device is not always lenges, and some of the products and OF TESTING WIRELESS on and that the wireless connection services detailed below may be help- MOBILE APPLICATIONS may not be as reliable as a wired one. ful to organizations struggling with Another key consideration is the mobile testing. uniqueness of each user. Customers a subscribe to different carrier networks CHAPTER 2 and have different use cases. The “last HETEROGENEITY CHALLENGES REGRESSION TESTING mile” to their device is likely to run at AND VENDORS’ RESPONSES HOT SPOTS: COVERAGE, different speeds and over slightly dif- Say what you will about Microsoft, for COMMON those of us who remember the “old MISTAKES days” when operating systems (OSes) Mobile development were virtually vendor-specific, the requires the same ubiquity of Windows has certainly made it easier to develop and test. Of Software Development course, Apple is, and always has been, Lifecycle (SDLC) an exception to the standardization approach that adds rule, and many Independent Software discipline and gover- Vendors (ISVs) develop applications for both Windows and Mac OS. This nance to traditional standardization has been a boon in software development. terms of limiting platform prolifera- tion. Assuming support for both ven- dors and a given number of operating ferent protocols depending on user system levels for each, the worst case location, roaming, vendor, network, testing scenario is still fairly limited. and device. The bottom line is that, if In contrast, mobile computing is you and I are running the same mobile just about where personal computing 5 SOFTWARE TESTING JULY/AUGUST 2009
  6. 6. was 20 years ago. Let’s see, you have ing on other partnerships as well, Symbian, Blackberry OS, Windows including one with DeviceAnywhere. Mobile, Palm OS, Apple OS X, Linux, DeviceAnywhere markets both cloud- etc., etc. Add to this the fact that based and on-site versions of its there are multiple browsers and little mobile Quality of Service (QoS) solu- standardization across devices, throw tion with support for more than 2000 handsets and 30 mobile operators. Mobile applications I Microsoft: Microsoft is a rich a isn’t your grand- source of testing resources for Win- EDITOR’S LETTER dows mobile applications, with a mother’s COBOL, broad range of tools supporting devel- and testing them opment and testing of wireless appli- a requires not just skill, cations built using Microsoft technol- CHAPTER 1 THE but creativity and ogy. Microsoft’s mobile development CHALLENGES site is a good starting point. OF TESTING WIRELESS resourcefulness. Resources include a set of mobile MOBILE APPLICATIONS emulator images that can be used with or without Microsoft Visual Stu- in 2 or 3 release levels each and the dio and provide a testing platform for a result is a fairly complex testing sce- the Windows Mobile operating sys- CHAPTER 2 nario. tem. REGRESSION TESTING This can be a problem for testing HOT SPOTS: COVERAGE, teams since, unless they have one of I Dexterra™: Dexterra is a different COMMON each combination of device, OS, ver- product entirely, and its flagship prod- MISTAKES sion, etc., it is virtually impossible to uct is the Dexterra Concert mobile ensure that a given application works application development platform. “everywhere”. Multiple vendors are Dexterra is available in two versions. addressing this challenge with a wide “Enterprise Edition” is an open, stan- array of solutions. dards-based development platform Some solutions are aimed at in- designed to be deployed within the house testing: enterprise. “Carrier Edition” is target- ed at service providers who plan to I Hewlett Packard (HP): HP is inte- host the solution as a service. Both grating mobile application testing into platforms support a wide range of its existing Quick Test Professional mobile devices as well as software (QTP) solution by partnering with requirements management. Dexterra vendors that specialize in mobile test- promises to streamline and simplify ing. The first partnership is with Jamo development of mobile applications, Solutions, which offers an end-user and may be well worth evaluating for experience management solution for companies which are heavily invested Windows Mobile devices. HP is work- in mobile development. 6 SOFTWARE TESTING JULY/AUGUST 2009
  7. 7. There are also a variety of options a recent Keynote release. for Quality Assurance (QA) managers Keynote has now expanded into looking for hands-on testers for spe- mobile testing as well, leveraging cific devices or to supplement in- testers from locations worldwide to house teams. This can be a lifesaver deliver an impressive breadth of solu- when testing schedules back up, or to tions. They include: certify solutions against actual devices instead of emulations. Two vendors I Mobile Device Perspective offering such services include: (MDP): Provides feedback on how a effectively mobile devices interact EDITOR’S LETTER I uTest®: uTest is positioned with their service content squarely in the software testing realm, I Mobile Application Perspective and recently announced support for (MAP): Provides information about a mobile applications. The company’s how well mobile sites function during CHAPTER 1 THE business model is a distinctive one. It transmission and delivery CHALLENGES consists of a host of software testers I SITE Test System: Provides OF TESTING WIRELESS (more than 14,000 worldwide) who mobile operators with metrics that MOBILE APPLICATIONS test mobile applications across a vari- quantify the health of the mobile net- ety of operating systems and plat- work forms. Testers get paid “by the bug”. I Global Roamer: Provides an in- a They execute testing scenarios and depth understanding of how applica- CHAPTER 2 report unexpected issues to the cus- tions function from various points REGRESSION TESTING tomer, who then decides whether the across the world, based on testing HOT SPOTS: COVERAGE, issue is, in fact, a defect. If so, the cus- from multiple locales COMMON tomer pays the tester who found the I Web Effective for iPhone: Plat- MISTAKES bug. If not, the customer does not pay. form to run large-scale, task-based iPhone Web usability studies This model can be valuable because human testers actually simulate the kinds of issues that are likely to arise PROBLEMS AND PAY-OFFS once the software is released to the There are obviously many more ven- public. It is also cost effective, in that dors and solutions in the mobile mar- uTest’s customers don’t pay for test- ket than there is space to discuss ing “features” the tester interprets as them. The point to think about is the “bugs”. idea of using automation wherever possible to add efficiency to mobile I Keynote™ Systems, Inc.: Keynote development and testing. is one of the best known Web testing In recent years, the cost of IT has companies in the world, due in part to risen to the point where, for some its frequent press releases detailing companies, it is limiting the ability to performance of well-known online grow and compete. Software develop- vendors. Click here for an example of ment, for many companies, is a cost 7 SOFTWARE TESTING JULY/AUGUST 2009
  8. 8. center within IT, and efficiencies in third-party vendors will likely pay off the development and testing spaces in terms of both testing efficiency are as important to overall cost reduc- and quality. Research has repeatedly tion as efficiencies in the data center. shown that software bugs detected That being said, rising IT costs aren’t due to IT specialists, develop- ers, or testers who waste time. They Mobile applications are due to technology complexity. The represent a “new past 10 years have seen an explosion reality” in terms of a of heterogeneous technology, with EDITOR’S each new technology requiring spe- applications, in that LETTER cialized personnel, tools, and infra- many of the applica- structure. Mobile is no exception. tions currently under a Mobile applications represent a development cannot CHAPTER 1 “new reality” in terms of applications, THE CHALLENGES in that many of the applications cur- be 100% tested by OF TESTING WIRELESS rently under development cannot be in-house personnel MOBILE APPLICATIONS 100% tested by in-house personnel and resources. and resources. They will be used by literally millions of users, and it is a impossible for a testing team with early in the lifecycle are much cheap- CHAPTER 2 finite resources to duplicate every er to remedy than those found in pro- REGRESSION TESTING use case, platform, network, and duction. Delivering quality software is HOT SPOTS: COVERAGE, device. one way that QA teams can directly COMMON Because of this, investments in impact costs, and this has become MISTAKES automated testing products and aug- even more true with the proliferation mentation of in-house test teams with of mobile applications. I ABOUT THE AUTHOR: JULIE CRAIG , a senior analyst at EMA, has over 20 years of experience in software engineer- ing, IT infrastructure engineering and enterprise management. Julie's areas of focus currently include best practices, configuration management, application management, service-oriented architecture (SOA) and Software as a Service (SaaS). Her goal is to interpret industry trends with a common sense approach and a simplified outlook. 8 SOFTWARE TESTING JULY/AUGUST 2009
  9. 9. CHAPTER 2 1 Regression testing hot spots: Coverage, common mistakes This article discusses regression test opportunities, coverage a policies and development, as well as filling you in on five EDITOR’S LETTER common regression testing mistakes. BY DAVID W. JOHNSON a CHAPTER 1 THE CHALLENGES OF TESTING WIRELESS MOBILE APPLICATIONS p THE REGRESSION TEST is one of the reduces remediation costs while miti- most misunderstood software testing gating potential production risks. In a concepts. Ineffective regression test this article, I will walk you through CHAPTER 2 REGRESSION policies lead to increased post- regression test opportunities, regres- TESTING deployment risk and resulting busi- sion test coverage policies and regres- HOT SPOTS: COVERAGE, ness losses. sion test development, as well as fill- COMMON MISTAKES Several recent software failures should have been detected and miti- gated by effective regression test Several recent soft- suites. For example, the Blackberry ware failures should blackouts of 2007 and 2008 are an unfortunate example of production have been detected issues that should have been detected and mitigated by by a regression test suite in combina- effective regression tion with other formal testing efforts. test suites. The convergence of adaptive develop- ment practices (agile, extreme, etc.) and complex integrated application ing you in on five common regression spaces (Internet, cloud computing, testing mistakes. etc.) create a growing risk profile that Regression testing is often referred needs to be mitigated by a disciplined to as a Test Phase or Test Type, but it implementation of effective regres- is really the execution of one or more sion testing policies. tests within a given Test Phase. For Testing early and often significantly the purposes of this article, a regres- 9 SOFTWARE TESTING JULY/AUGUST 2009
  10. 10. sion test is the execution of tests to tial production issues. The following verify that changes in the application table illustrates regression test oppor- landscape or architectural framework tunities as the solution moves from have not negatively impacted existing, being conceptual to a production unchanged functionality. Generally, according to WhatIs, a regression test involves the execution For the most part, of tests in a current Test Phase that validates the changes in the current the IT community has a application landscape—such as soft- come to realize that EDITOR’S LETTER ware, hardware, data, and/or metada- testing early and often ta—have not negatively impacted significantly reduces existing, unchanged functionality. remediation costs while a mitigating potential CHAPTER 1 THE CHALLENGES REGRESSION TEST production issues. OF TESTING WIRELESS OPPORTUNITIES MOBILE In the past, a regression test has been APPLICATIONS closely associated with the Function release/package. The solution set can Test Phase and System Test Phase. include software, hardware, data, a For the most part, the IT community metadata, or a mixture of these and CHAPTER 2 has come to realize that testing early any other aspects of the application REGRESSION TESTING and often significantly reduces reme- landscape. HOT SPOTS: COVERAGE, diation costs while mitigating poten- In the chart below, Solution Set COMMON MISTAKES SOLUTION SET IS… EXISTS AS… REGRESSION OPPORTUNITY… Conceptual (no code) A Model Design Review Phase Under Construction Discrete Units Unit Test Phase • Instrumentation Constructed Functional Function Test Phase Components • Functional Test Sets Functional Integrated System Test Phase System • System Test Sets Confirmed Package User Acceptance or Release Test Phase • UA Test Sets Released Production Production Monitoring Solution 10 SOFTWARE TESTING JULY/AUGUST 2009
  11. 11. refers to the current maturity of the the Tester. The Tester should accom- solution set. Exists as is the current plish this by implementing a suite of deliverables supporting the solution automated scripts that test the func- set. Regression Opportunity notes the tional areas of the application deemed available regression test opportuni- to be in-scope for the current test ties. The primary opportunities are in italics. There is often not IUnit Test Phase: Instrumentation enough time in the a The first opportunity to perform sub- EDITOR’S stantive regression testing occurs in test schedule for LETTER the unit test phase. The primary actor significant manual responsible for regression testing regression testing. a within the Unit Test Phase is the CHAPTER 1 THE developer. The developer accomplish- CHALLENGES es this by implementing the Adaptive engagement—these automated OF TESTING WIRELESS development principles—namely scripts should be constructed to be MOBILE APPLICATIONS agile, extreme and rapid application reusable test artifacts. Remediation of development (RAD)—of continuous defects at this level of solution matu- integration and code instrumentation. rity will reduce remediation costs. a Remediation of defects at this level of Manual regression testing can also CHAPTER 2 solution maturity will yield significant be performed, but the overall return REGRESSION TESTING downstream benefits and significantly on investment is greatly reduced. HOT SPOTS: COVERAGE, reduce remediation cost. Manual tests take longer to execute COMMON I have recently had the pleasure of and are prone to failure over multiple MISTAKES working with several disciplined agile executions. There is often not enough development teams that have fully time in the test schedule to allow for implemented code instrumentation significant manual regression testing. and continuous integration. The results of their efforts were a signifi- I System Test Phase: System cant decrease in defects later in the Test Sets development lifecycle. The third opportunity to perform sub- stantive regression testing occurs in IFunction Test Phase: Functional the System Test Phase. The System Test Sets Test Phase should include several The second opportunity to perform types of testing; from a Regression substantive Regression Testing occurs testing perspective, the two most in the Function Test Phase – this is the important would be Business func- opportunity that is most often lever- tionality and performance. aged by organizations. The primary The primary actor responsible for actor responsible for Regression Test- regression testing business function- ing within the Function Test Phase is ality within the System Test Phase is 11 SOFTWARE TESTING JULY/AUGUST 2009
  12. 12. the tester. The tester should once test the business functionality again accomplish this by implement- deemed to be in scope for the current ing a suite of automated scripts that test engagement; these automated 1 Avoiding Common Regression Test Mistakes a EDITOR’S THE MOST COMMON regression test mistakes come from misplaced LETTER assumption that regression tests: • ARE STATIC a » Regression test maintenance must be ongoing. Regression CHAPTER 1 THE tests need to be maintained as IP (intellectual property) that CHALLENGES meet the current organizational test coverage needs. OF TESTING » I have worked several engagements in the last few years that WIRELESS MOBILE have treated existing regression test suites as static artifacts. APPLICATIONS In one case, the client ended up with over five-thousand test cases that could never be executed within the time-frame pro- vided for testing. Once we did a maintenance sweep of these a manual regression test scripts we had less than one-thousand CHAPTER 2 automated test cases. REGRESSION TESTING HOT SPOTS: • ARE A NICE-TO-HAVE COVERAGE, COMMON » Regression testing is only a nice-to-have if the current organi- MISTAKES zational coverage needs—such as regulatory, IT governance and risk—does not require any supportive regression test met- ric. This would be a rather unusual circumstance, but it does occur; i.e., an in-house reporting tool. » Anyone who works in the testing space has seen the conse- quences of this mindset. In several cases I have seen clients do little or no regression testing and pay the cost in post-produc- tion support; a cost that far exceeds the cost of actually doing the testing. • NEED TO RETEST EVERYTHING » This is usually not a practical goal unless the nature of the application space (i.e. medical devices) and the level of risk, justify a complete end-to-end regression test—the testing team simply runs out of time. Unfortunately this usually means that the “easier” tests are executed first while the “harder” regression tests are deferred. Ensure you create a test sched- ule that fits into the available Regression test time-box and addresses the higher priority coverage items first. 12 SOFTWARE TESTING JULY/AUGUST 2009
  13. 13. scripts should be constructed to be required to be built and, more impor- reusable test artifacts. Ensure these tantly maintained. Regression test tests do not replicate other function policies should be based on three key testing efforts. factors: The primary actor responsible for regression testing performance within 1. Regulatory Space the System Test Phase is the perform- • Rules, Regulations, Laws, ance assurance test engineer. The and Standards that apply to performance assurance test engineer the application space. a should accomplish this task by imple- » i.e. FDA, FAA, SOX, etc. EDITOR’S LETTER menting a regression suite of per- formance tests and associated per- 2. IT Governance formance monitors—these suites • Corporate governance as it a should be constructed to be reusable applies to the regulatory space. CHAPTER 1 THE performance test artifacts. • Corporate governance as it CHALLENGES Manual regression testing of busi- applies to corporate rules, OF TESTING WIRELESS ness functionality can also be per- regulations, and standards. MOBILE APPLICATIONS formed but the overall return on investment is greatly reduced. Manual 3. Risk Profile tests take longer to execute and are • The acceptable level or risk a prone to failure over multiple execu- for any given aspect of the CHAPTER 2 tions. There is often not enough time application space. REGRESSION TESTING in the test schedule to allow for signif- HOT SPOTS: COVERAGE, icant manual regression testing. I Regulatory Space COMMON The regulatory space can have a MISTAKES IUser Acceptance Test Phase significant impact on the scope of The final opportunity to perform Regression testing within any given regression testing occurs during the test phase. Mangers and test leads User Acceptance Test Phase. These responsible for testing need to ensure manual tests are performed by the they understand their legal and fiduci- end-user. The primary purpose of ary responsibilities within the regula- regression testing during this test tory space and the impacts of this phase should be assuring the end- responsibility on the overall scope of user that no unexpected changes the testing effort. have occurred in previously existing functionality. I IT Governance IT governance implies a system in which all stakeholders have the nec- REGRESSION TEST COVERAGE essary input into the IT decision mak- Regression test coverage policies ing process preventing IT from inde- guide the organization in determining pendently making and later being held what regression test artifacts are solely responsible for poor decisions. 13 SOFTWARE TESTING JULY/AUGUST 2009
  14. 14. Therefore a board needs to: and monitoring this use to achieve plans. It includes the strategy and • understand the overall architec- policies for using ICT within an ture of its company’s IT applica- organization.” tions portfolio; • ensure that management knows From a regression testing perspec- what information resources are tive application of IT Governance to out there; the application landscape means the • know what condition information appropriate level of information must a resources are in; be consistently supplied to support EDITOR’S LETTER • understand what role information decision making at the board level. resources play in generating Regression testing provides several revenue. opportunities to collect appropriate a information—as IT Governance prac- CHAPTER 1 THE There are several definitions of IT tices mature the amount of informa- CHALLENGES governance: tion collected needs to be assessed OF TESTING WIRELESS for completeness. MOBILE APPLICATIONS • Weill and Ross focus on “Specify- ing the decision rights and I Risk Profile accountability framework to Risk is the net negative impact of a a encourage desirable behavior in vulnerability taking into consideration CHAPTER 2 the use of IT.” both the probability and impact/cost REGRESSION TESTING if the risk becomes an issue/event. HOT SPOTS: COVERAGE, • IT Governance Institute expands Risk management is the process of COMMON the definition to include founda- identifying risk, assessing risk, and MISTAKES tional mechanisms: “… the leader- ship and organizational structures and processes that ensure that the Regression tests organization’s IT sustains and should be consistently extends the organization’s strate- gies and objectives.” reviewed and adapted to meet • AS8015, the Australian Standard the current coverage for Corporate Governance of needs. Information and Communication Technology (ICT) defines Corpo- rate Governance of ICT as “The taking steps to reduce risk to an system by which the current and acceptable level. Risk management future use of ICT is directed and often encompasses three main controlled. It involves evaluating processes: risk assessment, risk miti- and directing the plans for the use gation, and risk evaluation—Regres- of ICT to support the organization sion testing deals with supplying data 14 SOFTWARE TESTING JULY/AUGUST 2009
  15. 15. to support risk evaluation. The organi- coverage to satisfy the current IT zation should have already considered Governance requirements. what level of risk is acceptable for a 3. Regression test coverage in particular application. It is the respon- combination with other formal test- sibility of the organization to under- ing efforts shall provide sufficient stand the basis for the current risk test coverage to satisfy the current assessment and apply the appropriate acceptable Risk Profile. level of Regression testing rigor, to meet the specified level of acceptable a risk. REGRESSION TEST EDITOR’S LETTER Acceptable levels of risk can be DEVELOPMENT categorized as: Regression tests should address all meaningful: a Tier 1: Critical Application Space CHAPTER 1 THE • Minimal or no interruptions • Code Branches: Regression CHALLENGES in service are acceptable testing at the level of Unit Test OF TESTING WIRELESS • Functional Business events: MOBILE APPLICATIONS Tier 2: Important Application Space Regression testing at the level • Short uncommon interruptions of Function Test in service are acceptable • End-to-End Business events: a Regression testing at the level CHAPTER 2 Tier 3: Normal Application Space of System Test REGRESSION TESTING • Short common interruptions • User Experience: Regression HOT SPOTS: COVERAGE, in service are acceptable testing at the level of User COMMON Acceptance Test MISTAKES Tier 4: Non-Critical Application Space In each case appropriate tests need • Interruptions in service are to be designed, built, and constructed. acceptable These tests need to be easy to main- • Application can be pulled tain, with definitive pass/fail condi- tions, and when possible automated. Regarding regression test automa- I Regression Test Policies: tion, the User Acceptance test should Coverage be defined and executed by represen- 1. Regression test coverage in com- tatives of the user community - these bination with other formal testing types are rarely automated. All other efforts shall provide sufficient test levels of regression testing should be coverage to satisfy the Regulatory either automated using test automa- Space. tion tools or instrumented using 2. Regression test coverage in Adaptive development techniques. combination with other formal testing On the maintenance front, regres- efforts shall provide sufficient test sion tests are often treated as a static 15 SOFTWARE TESTING JULY/AUGUST 2009
  16. 16. set of artifacts to be applied during to right size the Regression test each release cycle. Regression tests suite to meet the current cover- should be consistently reviewed and age needs, such as regulation, adapted to meet the current coverage governance and risk. needs (regulation, governance, and risk). This review should include • Regression test dependencies— adding to, removing, and merging plans, schedules, test data, test regression tests in the regression test environments, tools, etc.—must suite. If test case maintenance is not be considered part of the regres- a incorporated into Regression testing, sion test suite. EDITOR’S LETTER the tendency is to end up with an unmanageable number of tests that Regression testing needs to be may not address the coverage need. incorporated into the overall testing a In software development, there are strategy. The goal should be to miti- CHAPTER 1 THE some key regression test policies that gate risks to production caused by CHALLENGES you can’t overlook without bad conse- unexpected impacts against un- OF TESTING WIRELESS quences. From my experience, these changed functionality; creating re- MOBILE APPLICATIONS are the most important: gression test “saves” early in the sys- tem development life cycle. A “save” • Regression tests shall be auto- is the detection of a defect before it a mated using test automation reaches production thus providing the CHAPTER 2 tools or instrumented into the opportunity to remediate the defect REGRESSION TESTING solution using Adaptive develop- before it is released. The regression HOT SPOTS: COVERAGE, ment techniques. testing effort also needs to be right- COMMON sized to meet the corporate coverage MISTAKES • Regression tests shall undergo requirements and fit into the available scheduled maintenance designed regression testing time-box. I ABOUT THE AUTHOR: DAVID (DJ) W. JOHNSON is a senior test architect with over 22 years of experience in Infor- mation Technology across several industries having played key roles in business needs analy- sis, software design, software development, testing, training, implementation, organizational assessments, and support of business solutions. Developed specific expertise over the past 12 years on implementing “Test ware” including - test strategies, test planning, test automation (functional and performance), and test management solutions. 16 SOFTWARE TESTING JULY/AUGUST 2009