World Quality Report 2012-13


Published on

World Quality Report 2012-13

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

World Quality Report 2012-13

  1. 1. fourth EdItIon2012-13worldQualiTyreporT
  2. 2. 4 Introduction 6 Executive SummaryThe State ofQuality 201212 Quality Budgets: Stepping up18 Testing Centers of Excellence:A growing priority24 Mobile Testing:Behind the curve30 Cloud: Gathering pace36 Quality Assurance Resources:A question of valuesectoranalysis42 Consumer Products,Retail, and Distribution:Mobile and internet commercedrives sector transformation44 Energy & Utilities: Commitmentto improve quality of testingthrough increased investment46 Financial Services:Evolving QA – Riding thetransformation wave48 High Tech: Increasing focuson data-driven customerservice delivery50 Public Sector: Renewedfocus on efficiency gainslinked to budget reductions52 Telecoms, Media, andEntertainment: Maximizingassets, unleashing growth,and transforming to succeedCountry andRegional Analysis56 Australia and New Zealand58 Brazil60 China62 France64 Germany66 The Netherlands68 The Nordic Region70 North America72 United Kingdom74 About the Study10 40 54 WORLD QUALITY REPORT 2012-13 3CONTENTS
  3. 3. Introduction4 WORLD QUALITY REPORT 2012-13 Introduction
  4. 4. Welcome to the fourth edition of the World Quality Report, which, as in previousyears, has been designed to provide the most comprehensive assessment of thecurrent state of enterprise application quality and testing practices availablefrom around the world.To enhance the quality and relevance of thedata gathered, we increased the number ofinterviews to more than 1,550 CFOs, CIOs,IT directors, and quality assurance (QA)directors around the globe, all of which werecarried out by phone. We also broadenedour geographical sample, focused moreintently on the ‘enterprise’ market, andincluded additional questions reflectingthe significant changes in our technologylandscape, such as the almost ubiquitousuptake of mobile communications.Changes taking place in the IT marketincreasingly require scalable, robust, andreliable applications and infrastructure.Further, the disruptive effects ofcloud computing adoption, increasedmobility, and device proliferation requireorganizations to develop applications thatcan seamlessly interface with an ever-broader range of distribution channels androutes-to-market. Our research is evidencethat a focus on quality management isincreasingly regarded as a strategic value.While this year’s research confirms manyof the trends from previous editions ofthe World Quality Report, there are also anumber of new emerging developmentsin the market and an increased paceof change. For example, since last year,many readers will have seen corporateannouncements about a growing numberof large-scale enterprise-wide managedtesting service contracts being awarded,and an increased uptake of Testing as aService (TaaS) solutions.Similarly, many organizations are lookingto develop Testing Centers of Excellence(TCOEs), which use a standardized testingmethodology, best practices, and tools,together with a flexible pool of professionalresources to ensure high levels of qualityand risk mitigation across all applications.Over the last four years, the World QualityReport, published by the CapgeminiGroup and HP, has established itself asthe largest annual survey of applicationquality and testing practices, with a growingreputation as a valuable benchmark fororganizations around the world. The reportis the result of close collaboration betweenCapgemini, Sogeti, and HP, both leaders inour respective fields of outsourced testingand Application Lifecycle Managementtools, services, and products developed toimprove the efficiency and effectiveness ofenterprise applications.We hope you find that the research,analysis, and commentary containedin this year’s report both informs yourown testing and quality assurancedecisions and perhaps challenges someof your current thinking. We also valueyour comments and ideas, and inviteyou to contact either of us or any of thecontributors to this report regarding anyquestion you might have concerning testingand quality assurance.Finally, we would like to thank all of theindividuals who participated in the researchand gave generously of their time. Withoutthem, it would not have been possible toproduce the World Quality Report 2012-13.You can also reach us by sending an emailto: de MeijerSenior Vice President, Leader Global Service LineTesting, Capgemini & SogetiMatthew MorganVice President, Hybrid IT and Cloud ProductMarketing, Software, HP WORLD QUALITY REPORT 2012-13 5Introduction
  5. 5. Executive SummaryThis is the fourth year that Capgemini, Sogeti, and HP have produced theWorld Quality Report. Our aim remains to examine the current trends inenterprise application quality across different industries and geographies forthe purposes of providing actionable insight for decision making. Our hope isthat readers will learn from our report and apply their new understanding toimprove the effectiveness of Quality Assurance (QA) and testing within theirown organizations.6 WORLD QUALITY REPORT 2012-13Executive Summary
  6. 6. Murat AksuVice-President, Global Alliance Executive for HPCapgeminiCharlie LiVice-President, Global Service Line TestingCapgeminiOur global survey was carried out againsta challenging economic landscape fororganizations of all sizes and sectors.Many countries continue to experience low,stagnant, or negative rates of economicgrowth, and many governments haveapplied harsh austerity measures aimed atreducing public expenditure. Some sectors,particularly public and financial services,are under greater scrutiny to get IT rightthe first time and ensure improved valuefor money. Social media and the speed bywhich information is exchanged are alsochanging the way consumers and providersof applications interact with one another.Consumers of applications enjoy anenhanced scrutiny over service providers,whereas the providers are looking forways to respond to the resulting pressure,while also keeping an eye on a potentialmarket uptick.We have focused our research on theimpact of this complex environment ontesting and QA, a discipline that is itselfalso undergoing a series of technologicaland cultural changes.Rapidly evolving IT landscapenecessitates innovation in theQA disciplineQuality Assurance is undergoing a quietbut steady evolution from in-housetesting generalists to a structuredand efficient discipline, with a greaterinfluence within the overall applicationdevelopment lifecycle. We observefrom our survey the emergence of amultifaceted discipline with an increasingrange of operating models at its disposal,that enables a streamlined and cost-effective output that is also aligned withbusiness needs.The level of QA and testing investmenthas proved to be resilient in adverseeconomic environments in most markets,and has stretched to accommodate anever-increasing workload. But now testingresources need to prove themselves moreeffectively and QA teams need to stayalert to the new disruptive technologiesand ensure their skill levels are attuned toboth market and internal expectations. Weexplore these findings in more detail below.Has QA been caught off guardby mobility?As mobile adoption has become almostubiquitous in developed markets, thebusiness imperative for mobile businessis clear. But our study indicates that speedof adoption and proliferation of handhelddevices, coupled with use of social media,seem to have caught enterprise testingby surprise. Organizations may not be givingmobile the priority it warrants. Only 31%of respondents across the world currentlytest mobile applications – a figure that doesnot deviate much from region to region, andthose surveyed readily admit to being illequipped for mobile testing. This suggeststhat QA has fallen behind the mainstreammobile curve.Reasons given are multiple: many reportthat they don’t have the ability to test oreffectively certify mobile applicationsbecause of the lack of appropriate tools,processes, or expertise, and limited accessto the necessary devices. Moreover, thefocus is firmly on efficiency of performance,cited by 64% of firms, rather thanfunctionality, usability, or security. Overall,this argues for an underestimation of theinfrastructure challenges posed by themobile era, or an inability to address them.As today’s mobile users – customers andemployees – expect interaction at theirfingertips, anytime, anywhere, making abusiness mobile and “always on” should behigher up the corporate and IT agenda.New operating models such ascloud-based models and TestingCenters of Excellence havecome of ageCloud-based models for testing have had arelatively slow adoption rate, but evidencefrom our study of sustained wider-scaleadoption and falling barriers indicates thatorganizations are on the verge of muchmore extensive use. Testing in a cloudenvironment, for example, is growing at ahealthy rate. Some 28% of our respondentscurrently use this infrastructure model, aproportion forecast to rise to 39% by 2015.Moreover, only 4% say that they will notuse the cloud in some way over the nextthree years, down from 31% only two yearsago, a clear indication that testing in thecloud is set to become a new norm, asissues surrounding cloud, such as security,are addressed.Equally significant is the adoption of othercloud-based services. Software as aService (SaaS) continues to grow; almosta third (31%) of firms’ testing software iscurrently provided on a SaaS basis – upfrom 25% last year, and by 2015 we predictthat almost half of firms will use theon-demand model. WORLD QUALITY REPORT 2012-13 7Executive Summary
  7. 7. Testing as a Service (TaaS) – which canbe a combination of cloud-based testingenvironment and tools, and on-demand testexecution – is a cost-effective operatingmodel for ad hoc quality projects and alsofor longer-term testing provisioning thatdemonstrates similar rates of adoption. Wefound that 78% of respondents are planningto move to TaaS in the next two years, risingto 89% by 2015. This rapid take-up of TaaSis fueled by cost-reduction drivers andits flexibility as a complementary modelto more comprehensive outsourcing orin-house Testing Centers of Excellence(TCOEs). So while some still see the securityof the cloud as a continuing threat, theirnumbers are dwindling as the advantagesmore clearly outweigh the drawbacks.For TCOEs that have been underutilizedin the past, their time too has arrived.Undoubtedly an efficient approach toindustrializing testing activity, TCOEsact as a virtual command center, using astandardized approach and a flexible poolof available resources. Our data revealsthat, this year, 60% of respondents planon – or are – developing a TCOE, up from45% last year. In our experience, we seethe concept is now sufficiently advancedand its benefits more clearly understood:a TCOE enables organizations to meet thechallenges of the business – speed-to-market and doing “more for less” – head on.Can we put a price on quality?Technological innovations and initiatives areincreasing the workload for ever-stretchedQA teams, but the corresponding QA andtesting budgets appear to be weathering theeconomic storms. In 2011, these budgetswere not supporting the challenges facedin the marketplace, but this year’s surveyfinds testing budgets growing at a strongerrate than last year, with 42% reporting thatbudgets had risen over the past 12 months.While 18% forecast a fall when lookingahead to 2015, some 53% optimisticallyexpect budgets to rise, indicating a newdegree of confidence that the business ismore committed to investing in QA.Moreover, the focus appears to haveshifted from “business as usual” tasksto investment in transformationalwork, to drive enhancements, and59% of the budget is now spent oncustomer-facing applications. So prioritiesare changing to support the core areas ofthe business that will require an optimizedplatform for efficient delivery of testing,which we regard as a positive step.For the first time, the World Quality Reporthas established a worldwide benchmarkfor testing hourly rates, internallyand externally. Firms are paying a globalaverage hourly rate of around $55 per hourfor in-house testers, compared to $53for external support – a differential of $2per hour. Obviously, rates vary greatly fromregion to region and can be affected by thescarcity of available skills in local markets.Does this represent the price of quality?Only an individual business can make thatdecision, when evaluated against otherprime resources.Confidence in QA resourcesis a concernWhile budgets might be in reasonableshape, confidence in testing resources isnot resounding. A majority of organizationscharacterize their internal teams as“average” at best, in their knowledge ofcore testing processes and methodologies,and not necessarily up to speed with thelatest testing tools and technologies. Theirassessment of external testers is slightlybetter, with a third of organizations scoringtheir external testers’ knowledge andabilities as “above average”. But less than5% of firms are fully confident that theirtesters (internal or external) are “best inclass”. Despite initiatives and investment,especially over the last decade or so, thereare clearly lessons to be learned in terms ofeither real or perceived quality of output forboth providers and users of QA.Simply put, as the competitive landscapeforces organizations to update and optimizetheir testing resources and drive downcosts and time-to-market, the overallquality of testing resources needs tokeep pace or change, to satisfy or exceedthe perceived and increasingly complexrequirements of organizations in the future.The emergence of a more globaland uniform industryIn our previous surveys, we have focusedattention on the differences between themajor countries or regions surveyed. Thisyear we notice that while variances stillexist, there is a greater harmonization,with fewer major differences evidencedacross the globe, at least within theenterprise markets. One example isthat the proportion of an organization’ssoftware development lifecycle budgetthat is invested in testing, averaged acrossthe regions, is 18%, with remarkably fewregional variations.Our perspective is that testing is becomingless of a regionally facing discipline andthat, led by globally active companies, thegap between emerging markets and moretraditionally mature markets is narrowing.This is no doubt a reflection of the rapidinvestment in skilled resources andtools, led by India, that has taken place inoffshore and nearshore countries – China,South East Asia, South America, and,more recently, Eastern Europe, CentralAmerica, and parts of Africa. All pointto testing emerging as a more uniformglobal discipline.World Quality ReportrecommendationsFor providers and users of QA and testing, weoffer these suggestions on how to apply thefindings of this year’s World Quality Report.QA needs to be a formalized stepin the application lifecycleWhile QA’s status within the IT organizationhas improved compared to a decade ago,we still see instances of the testing functionbeing considered as an afterthoughtor viewed as a roadblock preventingan application from being deployedto production. IT can only be as agile orefficient as its weakest component.As IT continues to reinvent itself with newprocesses, paradigms, and platforms, theseinnovations rarely include aspects of QA.The fact that testing is not being broughtalong the journey of reinvention often resultsin QA being the bottleneck. Organizationsneed to understand the critical role of QA inthe application lifecycle and treat it with thesame respect as its peers.Mobility should become fullyintegrated into testing prioritiesWe were surprised by the relatively lowlevel of proactive structured testingin this increasingly essential area ofbusiness connectivity. We believe that8 WORLD QUALITY REPORT 2012-13Executive Summary
  8. 8. mobile testing needs to be a fully integratedelement of the QA discipline, so that themobile strategy of the enterprise takestesting into account right from the start.The strategy should consider the objectivesof the business owner, how the mobileapp is delivered, and the target userfor the app whether that be customers,suppliers, or employees. Organizationsneed to accept the paradigm shift broughtabout by mobility and embrace the newnotion of quality for mobile apps, whichis a departure from traditional standardsapplied to desktop applications.No doubt, standardization of devices willease the complexity over time, but in theshort term, proliferation of smartphoneand mobile devices, the roll-out of 4G,and use of social media will only continueto exacerbate the situation. As will theneed to focus on the user experienceand functionality testing as wellas performance. If organizations are toturn the mobile opportunity into a businessadvantage, some will need to “skill up” or“skill out”.Evaluate the breadth of cost-effectiveoperating models to see if it is a good fitIn the quest for greater testing maturitybalanced against return on investment,organizations have an increasing array ofoperating models from which to chooseto achieve their goals. Companies shouldfirst define the business goals they wishto achieve and then evaluate the myriadof models against these objectives. Eachmodel has its advantages and shortcomings,and corporate culture, appetite for risk, andneed for control versus flexibility all play apart in determining whether a new operatingmodel should be adopted and which one.On the journey from in-house testinggeneralists and developers to aprofessionalized discipline wanting toadopt “Shift Left” principles that embedQA earlier in the traditional softwaredevelopment lifecycle, organizations shouldcritically appraise their current set-up andreview how testing in a cloud environment,cloud-based TaaS, and TCOEs can providethose critical incremental efficiency andquality improvements.Pay closer attention to demonstratingQA’s business value for the moneyValue for money is, of course, subjective,but what does constitute a good returnon your QA investment? One measure isinvestment in QA/testing as a proportionof the total IT budget, averaging at 18%across the world. Below 18%, organizationsshould examine if they are investing enoughto achieve or sustain the quality required.Those spending more should review theiroperational efficiency.But more precisely, companies need toestablish specific measures to evaluatethe return on their overall investment,and testing departments and serviceproviders need to act “smarter” in their useof tools, models, and communication tostakeholders to demonstrate the businessvalue they provide in order to justify theproportional spend on QA.Re-evaluate the diversity of skills requiredThe perceived quality of testing resourceshas a bearing on value for money, andthe unexceptional assessment of abilityneeds examination. Over time, we expectthat businesses will be even moreexacting in their expectations of any andall testing resources when it comes tocomplex business-critical applications. Allproviders, in-house, contract, and external,should take notice.QA has made great strides in becoming aprofessional discipline with a distinct careerpath and specific skills. But this is a movingtarget; testing training at the individual andteam level needs to keep pace with the fast-changing landscape, and should be properlyfunded, rather than being sacrificed underthe pressure of deadlines.Budgets are currently at a strong level, aclear recognition of the value of the testingfunction to the organization, yet this isin contrast to the perceived value of theactual resources. This gap can certainlybe bridged by clearly targeting those skillareas that are essential to the business.ConclusionOur findings in this report indicate thatQA continues to make steady progresson repositioning itself further up thebusiness value chain on the journey ofreinvention, emerging from the shadows ofdevelopment teams to a value-for-moneyand mature discipline. Looking ahead, weanticipate further strides in the adoptionof “Shift Left” initiatives and greater use ofnew models such as crowd sourcing whichleverages knowledge and manpower froman undefined pool of resources, to improvevalue-for-money ratios and timelines. Butgreater improvement will not come aboutwithout significant investment – not onlyfinancially, in terms of budgets and skills,but also culturally.But organizations will need to exert firmbut flexible control to work across a widerrange of delivery models and technologies.Budgets will need to work harder to effectthe changes required to respond to a rangeof demands from time-to-market, costefficiency, and improved quality. And QA willneed to keep up with market expectations,remaining focused on providing real,measurable business value. WORLD QUALITY REPORT 2012-13 9Executive Summary
  9. 9. TheState ofQuality201212 Quality Budgets: Stepping up18 Testing Centers of Excellence:A growing priority24 Mobile Testing:Behind the curve30 Cloud: Gathering pace36 Quality Assurance Resources:A question of value10 WORLD QUALITY REPORT 2012-13
  10. 10. WORLD QUALITY REPORT 2012-13 11The State of Quality 2012
  11. 11. Quality Budgets:Stepping up“Investment is going to befocused on those elementsthat support our widerbusiness objectives, asthere is a need for [testing]and the business side to bemuch closer. Decisions needto be made more quickly andproduct development timeis shrinking – we need thetools and skills to be ableto answer that need.”A telecommunications business,The NetherlandsOrganizations are trying to balance market uncertainty with the need toposition for eventual growth, as well as tackling the challenges of social mediaand mobile technologies. At the operational level, a complex web of factors isforcing organizations to update and optimize their testing resources and internalprocesses to drive reduced time-to-market and lower costs, as well as improvequality using new beneficial delivery models, such as cloud and TCOEs.As a result, the QA function and QA budgets, in particular, are under the spotlight,because to achieve these stretch business targets, organizations need to leveragetheir QA capabilities more effectively. This requires a step-up in investment over theshort and medium term, even in times of economic uncertainty.Indeed, this year’s survey finds testing budgets are expanding at a healthier ratethan the incremental upturn seen in 2011-12, when growth was failing to keep pacewith an increasing workload, due to cost-cutting measures. This positive steppoints to a desire among businesses to invest for the future and meet internal andexternal demands quickly and effectively.An upward curveIn general, the survey provides good news regarding QA budgets, given theeconomic backdrop. Close to half (42%) of QA budgets have increased over the lastyear, with only 11% reporting a decrease. The greatest proportion (44%) saw nomovement either way, but in a time of uncertainty, maintaining the same level ofinvestment in the quality function is, in itself, a testimony to the importance of QAto the organization. Overall, this is perhaps a more positive situation than mighthave been predicted.Moreover, this upward curve is expected to continue over the coming three years,indicating a degree of optimism. A healthy majority (53%) expect their budgets torise between now and 2015, while just a fifth (18%) forecast a fall. This seems to bea universal trend with little variation by region or sector.The State of Quality 2012Quality Budgets12 WORLD QUALITY REPORT 2012-13
  12. 12. INCREASEDSIGNIFICANTLYINCREASEDSLIGHTLYSTAYED THE SAMEDECREASEDSLIGHTLYDECREASEDSIGNIFICANTLYNO BUDGETTO TESTBase: 1553 Respondents2015NOW8%3%8%45%10%26%36%6%3%4%7%44%CHANGE IN TESTING BUDGETS FROM LAST YEAR, ASCOMPARED TO NOW AND IN 2015 FIGURE 1The majority of organizationsexpect testing budgets to risebetween now and 2015The State of Quality 2012Quality Budgets WORLD QUALITY REPORT 2012-13 13
  13. 13. SPEND ON EMPLOYEE- VERSUS CUSTOMER-FACING APPLICATIONS,AND TRANSFORMATIONAL VERSUS MAINTENANCE WORK FIGURE 2Base: 1,221 respondentsCUSTOMER FACINGEMPLOYEE FACING41%59%VSMAINTENANCE WORKNEW TRANSFORMATIONAL WORK41%59%VSIncreasing pressure todemonstrate resultsBudgets may be on the upward trend,but there is no room for complacency. Anincrease in investment will intensify thefocus on QA productivity, a function that isviewed primarily as a necessary cost base.Even as testing spend rises, companiesstill need to maintain downward pressureon costs, as the move to consolidationin TCOEs and on-demand testing via thecloud both illustrate. Any investmentneeds to achieve a return, which meansthat continued or intensified scrutiny bythe business units who pay for QA servicesis likely, in order to ensure that newtesting resources are optimally deployedand managed.One way to ensure budgets are beingspent productively is to effectivelymeasure the added value of testingand clearly communicate this to thebusiness stakeholders. The good newsis that only 15% of organizations do nothave a process to consistently collect andpresent their quality KPIs to the business.The vast majority of organizations arecollecting this data using QA managementor business intelligence products inthe market, or using common officeapplications such as Microsoft Excel. Infact, 56% indicate that they automaticallygather and share metrics using Excel, and21% collect data manually and use Excelas a presentation layer. However, whileExcel is universally available and a low-costsolution, it does not support the richness infunctionality that a professional reportingand QA management product can provide.In times of greater business scrutiny, QAorganizations’ best interests will be servedby providing constantly updated businessKPIs that track return on investment.The magic number forquality budgetsTesting budgets tend to average out ataround 18% of total spend on organizations’software development lifecycle, and thisfigure seems to extend across all globalregions and industry sectors surveyed.On balance, this would seem to be thebenchmark proportion of the softwaredevelopment budget that organizationsneed to commit to testing.Spending much below this level runs therisk of developing applications that areinadequate from a quality perspective,or applications that are outdated beforetheir release when the QA process takestoo long to bring a product to marketahead of the competition. Spend toofar above this level, and organizationsmay need to question the efficiency oftheir QA operations, or ask whether theadditional spend is justified because thesuperior application quality contributesto a commercial competitive edge,or because it needs to meet highlystringent governmental regulationsor safety requirements, e.g. publictransportation systems.Increased focus onexternal activitiesThe areas in which testing investmentis being focused reflect the competitivepressures organizations feel in anunpredictable climate. A robust two-fifths(41%) of testing spend is being dedicatedto transformational work – testing newapplications, as opposed to maintenanceof existing software. This is a positiveindication that, even amid continued globaleconomic turbulence, firms are continuingto invest in developing new productsand services.Similarly, the greater proportion oftesting budgets (59%) are being spent oncustomer-facing applications rather thanon internal organizational applications(41%), a clear focus on external applicationsthat have a direct impact on sales andrevenue growth. Firms are also boostingtheir investment in TCOEs in orderto compress the time-to-market fornew products. Companies, no matter whatsector or region, are investing in innovationin order to maintain and capture marketshare and ultimately drive revenue.The State of Quality 2012Quality Budgets14 WORLD QUALITY REPORT 2012-13
  14. 14. “The problem is increasingpressure to get thingsdone yesterday. We havethis conflict betweentesting adequately toour satisfaction and thetesting time expected bythose on the [business] side.Investment has increasedbut then expectation inspeed of delivery hasalso increased. It’s noteasy, as the organizationneeds to be able to jumpinto a much faster testingmode while maintainingquality standards.”A manufacturingbusiness, CzechRepublicThe State of Quality 2012Quality Budgets WORLD QUALITY REPORT 2012-13 15
  15. 15. MORE INVESTMENT INEXISTING TESTING TOOLSBase: 1553 RespondentsAREAS OF GREATEST INCREASE IN SPEND: NOW AND IN 2015 FIGURE 3INTERNAL PROFESSIONALTESTING RESOURCESEXTERNAL OUTSOURCEDRESOURCESTEST ENVIRONMENTS/INFRASTRUCTURENEW TESTING TOOLSQA BUDGET IS NOTINCREASINGNOW201555%44%NOW201544%46%NOW201546%43%NOW201539%39%NOW201519%22%NOW201512%13%Investment priorities:current and futureQA investment priorities are largely shapedby the new market landscape whereorganizations must achieve more with fewerresources, find ways to reduce cost, anddeal with increased competition from bothbricks-and-mortar and online channels.“My investment going forward will [focus on] bringing my internal teams upto standard, increasing the level of automation used, and developing greaterefficiencies throughout. Although our testing budget is not separate fromthe wider IT budget, the level of monitoring and performance measuring hasincreased – we need to justify our costs.”An automotive business, Brazil As a result, organizations are seeing animperative to upgrade existing suitesof testing tools. More than half (55%) ofrespondents describe the upgrade processas the budgetary line item receiving thegreatest increase in spend, and in ouropinion this is driven by a desire to move tonewer application lifecycle platforms andsoftware-on-demand type models.Looking ahead to 2015, however, prioritiesare set to shift from upgrading existingtools to buying external resource support.A smaller proportion (44%) of firms stillexpect existing tools to be their number oneinvestment focus three years from now,but it is secondary to purchasing externalresources to supplement their internal QAteam (46%).The State of Quality 2012Quality Budgets16 WORLD QUALITY REPORT 2012-13
  16. 16. world are radically reducing spend tocope with the fallout from the worldwideeconomic crisis. Almost half (48%) of publicorganizations’ testing budgets increasedfrom the previous year, the highestproportion of all sectors studied, with 56%also predicting an increase between nowand 2015.Our observation is that public bodies areinvesting in consolidating their testingoperations to drive cost efficiencies inthe face of austerity. With a lingeringlegacy of failed major IT projects, QA hasto be stepped up to derive more fromexisting capabilities and improve testing.Investment in existing testing tools is seenas one of the solutions to remedy the failedIT projects issue. Over half (58%) of PublicSector entities identify this as the greatestincrease in spend.Manufacturing firms are the mostoptimistic among the sectors, anticipatingan increase in testing budgets three yearsfrom now: 58% expect spend to growbetween now and 2015, against a mean of53%. Although our experience has shownthat manufacturing firms tend to spend lesson IT overall compared to other sectors, itis an increase nonetheless. Some 30% ofmanufacturers predict that new testingtools will be the greatest spend in 2015,compared to 22% on average. CPRD firmsare similarly optimistic in terms of budgetoutlook in 2015, with 57% anticipating anincrease, and the sector places the highestpriority on innovation, committing 45% ofspend to new transformational work (all-sample average 41%).Energy and Utilities (E&U) is an intriguingexception in terms of current investmentpriorities, spending notably more on bothexisting (65%) and new (23%) tools, andtransportation firms are the most keenlyfocused on outsourcing. More than half(52%) see this as the fastest rising area ofspend, compared to a mean of 43%.In summary, against a gloomy backdrop ofdepressed economic activity and relativefinancial instability affecting most regionssurveyed, QA budgets have held theirplace in the IT overall spend. We believethis is a somewhat late recognition of theinherent value of testing and QA in ensuringrobust business-critical applications asan enabler of commercial growth andpublic reputation.New product development will generate theneed for new testing skills, tools, and devices,prompting organizations to seek supportfrom external partners. Investment in newtesting tools will also rise, as new productsand services are introduced to the market.Some 22% see this as the top spend priorityfor 2015, up from 19% at present.Budgets relatively consistentacross regionsThe greatest degree of consensus betweenglobal regions in this year’s report is aroundQA budgets. Budget levels hover consistentlyaround the all-sample mean of 18% of totalsoftware development spend across all butone of the geographic areas surveyed. Theexception is South America, where testingtakes up some 25% of the budget.We believe that this is an indication oftargeted South American investment. AsBrazil, in particular, continues to boomeconomically and expectations aroundapplication needs are raised, so testinginvestment is being enhanced.The pattern is similarly consistent for budgetincreases, with the exception of Asia. Asianfirms appear to benefit from the mostaggressive budget expansion compared toother regions, as was the case in 2011. Halfof Asian respondents saw an increase inbudget from last year, compared to the globalaverage of 42%. More still (60%) expect anincrease over the next three years, against anaverage of 53%.The balance of spend between new andmaintenance work also presents a fairlyuniform picture. Here again, South Americais the outlier, where firms are spendingalmost half (48%) of testing budget ontransformational work (average 41%).The same goes for the split between spend oncustomer- and employee-facing applications.South America again goes against the globaltrend, being the least externally focusedregion, spending only 52% on customerapplications (compared to 59% on average).Investment prioritiesvary by regionWhen it comes to investment priorities,there are greater variations betweenthe regions. Eastern Europe currently hasinvested far more in new testing tools (33%)when compared to other regions and to theglobal average (19%). The Nordics and NorthAmerica are at the other end of the scale, withjust 15% of firms giving top priority to buyingnew tools, although both regions indicate afocus on upgrading existing tool sets, whichmight suggest that they are looking to betterleverage their shelfware and make theirexisting investments go further. SouthernEuropean firms are more heavily focusedon upgrading existing tools (63% comparedto 55% on average), perhaps suggestingthat they are maturing from a more manual-intensive process to incorporate automation.Looking ahead, Southern Europe will beincreasing its investment in outsourcedtesting, with a majority (57%) of firms inthis region viewing this as their main QAinvestment priority for 2015, compared to46% overall. Just over half of respondentsin the UK and Ireland, North America,and Eastern Europe also expect externalsuppliers to receive the lion’s share of theirtesting spend three years from now, furtherevidence that the global move towardsoutsourcing will continue to grow.Public Sector leads the wayfor increased QA budgetThe pattern of uniformity around testingbudgets seen between the regions isrepeated across the industries surveyed,with QA budgets in most sectors fairlyevenly balanced around the average 18%of total software development spend. Thegreatest disparities are in the ConsumerProducts, Retail, and Distribution (CPRD)space (22%), reflecting a higher focus on thequality of end-user consumer applications,and Financial Services (FS) at 20%, atraditionally heavy investor in QA.The Public Sector, surprisingly, isexperiencing unexpected budget upliftsat a time when governments around the48% of publicorganizations’testing budgetsincreased lastyear - the highestproportion of allsectors studiedThe State of Quality 2012Quality Budgets WORLD QUALITY REPORT 2012-13 17
  17. 17. Testing Centersof Excellence: Agrowing priorityConsolidation and standardization are continuingthemes for all organizations around the world andthe testing function within these companies is notimmune to these trends. Aided by the emergenceof new delivery models, technologies, and bestpractices, organizations are looking to streamlinethe way in which their QA function is structuredand run. TCOEs are undoubtedly an efficient approachto industrializing testing activity. They act as avirtual command center that uses a standardizedtesting methodology, best practices, automation,metrics, and tools, while managing a flexible poolof available resources, both internal and external,to ensure high levels of quality across applicationsbefore deployment and during production. TCOEsalso provide visibility into the level of quality for anysoftware system or project, helping IT managementmake deployment decisions based on business risk.Growing from a low baseWe found that only 6% of companies around theworld have developed QA into a fully functional TCOE.This number is a significant increase, in terms ofpercentage, from last year’s findings (4%), but thesingle digit number still reflects a long road ahead forcompanies to universally adopt the TCOE model.One of the key attributes of a TCOE is a centralizedpool of resources that can be leveraged acrossmultiple projects in an organization. We found thatclose to 70% of organizations still rely on elementsof decentralized testing, while more than a quarter(27%) of them describe their testing function as“highly decentralized”.Testing automation, another aspect of a TCOE, isalso scarce, with just 32% of test cases supportedby automation. These figures indicate that companiescontinue to perform QA as siloed projects withminimal opportunity for standardization and sharingof best practices across the enterprise. Of note, thegreater proportion of automated testing is conductedoffshore, a testament to the growing confidence inmoving this type of testing beyond national borders,with only 40% being performed onshore.Flexible resource pools that can ramp up and downwith the changing needs of the company is anotherattribute of a TCOE. Our research shows that themajority of organizations (51%) still run testing asan in-house function, and only 13% have moved to aservice fully managed by an external provider. Withthe exception of North America, QA organizationshave yet to truly embrace the cost benefitsof offshoring. The majority of testing resources (52%)are still based on home territory.Reasons for this global discrepancy vary from regionto region. In South America, government-imposedcustoms taxes and new rules governing the useof offshore services discourage the use of thisflexible pool of resources. In Europe, particularly incontinental Europe, the lack of available offshoreresources skilled in the native language hinders theadoption of an offshore model, although we are nowseeing the UK and Nordics embrace this model ata faster pace. In other areas of the world, the laborarbitrage does not yield adequate amounts of returnon investment to warrant an offshoring strategy.The State of Quality 2012Testing Centers of Excellence18 WORLD QUALITY REPORT 2012-13
  18. 18. “[A TCOE] is something I’mbuilding here; it representsa significant change in ourapproach but one that Ipersonally believe tobe vital to our businesssince I’ve come from anorganization with anoperating testing center.We know that it can reducecosts and overall time oftesting practices, and giventhe competitive nature ofthe market we operate in,it’s critical to get speedand costs down whilstgetting quality up. Webelieve a TCOE can do this.”A manufacturingbusiness, UnitedKingdomThe State of Quality 2012Testing Centers of Excellence WORLD QUALITY REPORT 2012-13 19
  19. 19. Base: 924 RespondentsPLANS FOR A TESTING CENTER OF EXCELLENCE FIGURE 4PLAN TO DEVELOP INTERNALLY MANAGEDTCOE WITHIN THE NEXT 2 YEARSPLAN TO USE A THIRD-PARTYCOMPANY WITH A TCOE CAPABILITYIN-HOUSE TCOE STARTED WITHIN LAST 2YEARS, BUT NOT YET FULLY OPERATIONALIN-HOUSE TCOE IS FULLYOPERATIONALNO PLANS IN PLACE34%24%21%15%6%The percentage of organizations reportingthat they have no plans to set up a TCOEhas fallen significantly from last year (45%to 34%), and we predict that this will fallfurther over the coming few years as theTCOE model gains recognition for deliveringperformance improvement. This trend begsthe question – what is driving the surge ininterest in industrializing testing activity?Less cost, more speedFirms are centralizing their QAinfrastructure in order to reduce time-to-market, bring down the cost of testing, andmanage testing resources more effectively.Compressing time-to-market is the primaryreason for TCOE implementation, citedby 36% of respondents, and especiallythose in what might be called the moreAn appetite for changeHowever, we are witnessing a growinginterest in centralizing QA operations.Survey respondents indicated a significanttrend towards new investment in TCOEs,guided by the imperatives to reduce time-to-market and costs, increase quality,and gain better control of the wholelifecycle process.Whilst only 6% of organizations currentlyhave a fully fledged operational center,this presents a 50% increase from 2011(4%) – a small but significant step. On apositive note, almost two-thirds (60%)of firms are currently in the process ofbuilding or planning a TCOE, while a third ofthese companies (21%) plan to leverage anoutsourced partner.mature economies. Add to this the 17%aiming to enhance the agility of their QAoperations, and it is clear that speed is akey driver of growing TCOE adoption.In an unpredictable global economy, andwith market and business expectationsfocused on producing better productsfaster, time-to-market has becomeessential in gaining or maintainingcompetitive edge and maximizing profits.As a result, development cycles arebeing squeezed, and testing teams haveto do more with less. A TCOE offers theopportunity to meet these challengeshead on.Reducing the cost of testing activity isnext on the priority list, identified by justover a quarter (27%) of organizations,The State of Quality 2012Testing Centers of Excellence20 WORLD QUALITY REPORT 2012-13
  20. 20. Base: 1221 RespondentsEXPECTED BENEFITS FROM IMPLEMENTING A TESTINGCENTER OF EXCELLENCE FIGURE 5REDUCED COSTBETTER RESOURCE MANAGEMENT(HW/SW AND PEOPLE)BETTER QUALITYSTANDARDIZED TRACKING OFPROJECTS THROUGH METRICSREDUCED TIME-TO-MARKETIMPROVED QA AGILITY TOSUPPORT BUSINESS OBJECTIVES36% 27% 25%22% 19% 17%“For us, the move towards a TCOE is part of thewider union between the business and IT side ofthe organization. We now have a much strongerlink between the two, and while that meanswe feel more able to understand and answerdemands, it does increase pressure on testingto be more efficient as a whole – decentralizedtesting teams just couldn’t provide thatefficiency and it meant that walls existedbetween the teams, whereas shared experienceand best practice are far easier within aTCOE framework.”A financial services business, Finlandclosely followed by the need to manage hardware,software, and human resources more efficiently andproductively (25%).It is interesting to observe that cost – at least foreconomies such as North America – is not theoverriding priority. Most organizations have beenfocused on cost cutting for many years, and costshave been pared down to a minimum, with strategiessuch as outsourcing delivering benefits. It shouldnot be a surprise, therefore, that companies in theseregions are now seeking advantages beyond thepurely financial.These objectives for moving to a TCOE model promptus to ask: why has efficiency in meeting time-to-market deadlines risen to the top of firms’ agendas?What is different in today’s market environmentthat has rendered the decentralized approach lessdesirable for the QA function in some instances?The State of Quality 2012Testing Centers of Excellence WORLD QUALITY REPORT 2012-13 21
  21. 21. Based on the survey responses, togetherwith the experiences of our clients, webelieve there are three key factors at play:the market environment, rising technologyexpectations, and diversityof delivery models.1. The market environmentContinuing market unpredictability meansthat organizations can ill afford to takethe emphasis away from cost reduction.Many remain firmly in cost-saving modewhere QA is concerned: some still aim toshave as much as 30-40% off their currenttesting budgets.This report also notes that QA budgetsare increasing but that more is beingdemanded of teams and more complexdevelopments are requested. So whilebudgets are increasing, there is still a needto control costs.In parallel, there is considerable pressureto develop applications and bring them tomarket as rapidly as possible. At the sametime, firms also need to focus on innovationto prepare for an eventual upturn in theglobal economy.2. Rising technology expectationsIntensifying this pressure on applicationdevelopment is an increasinglycompetitive landscape, fuelled bygrowing customer demands. Customers– both internal and external – expect toaccess services and purchase productswhenever they want, on whatever devicethey choose. To meet this demand for 24/7availability, there is mounting pressure todeliver applications as quickly, efficiently,and cheaply as possible – and this isparticularly evident in the mobile spacewhere expectations are well advanced andare most likely going to increase over time.3. Diversity of delivery modelsIn response, QA departments arecapitalizing on the efficiency gainsoffered by cloud technology, increasinglyoutsourcing testing to cloud environments,and buying resources on demand, toreduce cost and time to bring applicationsto market.A further step to speed up the developmentprocess is the wider adoption ofAgile methodology. This is currently beingused by over a third of testers (37%),analysts (34%), and developers (29%). Andencouragingly, three-quarters (75%) ofAgile testing is already being performedoffshore, a situation that seemed unlikely afew years ago.Offshore adoption: a keydeterminant for TCOEimplementationCentralizing and professionalizing the QAfunction undoubtedly presents a valuableopportunity to save costs in a tougheconomic climate. However, motives drivingconsolidation are, in part, a function of howfar different regions have already leveragedthe cost benefits of offshoring.As a rule, markets with greater levels ofoffshoring – primarily North America andthe UK – are less focused on cost reduction.Offshoring has been considered, executed,and reviewed, and there is little remainingscope to further squeeze labor arbitrage.Therefore, other models are being exploited,such as TCOE, for achieving better quality,being quicker to market, and moreeffectively using resources. Cost is not aneasy win for the more mature markets –they have to look at other key benefits todrive more efficient and focused QA.Conversely, other regions with lower levelsof offshoring remain more keenly focusedon cost reduction. This is understandable,given that many are in the early stages ofoffshoring and thus have more resourcecosts to reduce. This difference is reflectedin our survey results.North America and the UK are the only tworegions where the majority of testers arelocated either offshore or nearshore (55%and 50% respectively). And they are the twoleast cost-focused markets: only 15% and16% of companies cite cost reduction as areason for establishing a TCOE.In South America, by contrast, offshoringhas been far less exploited, and (it shouldbe noted) in Brazil is actively discouragedby government policy. Only 38% of testersin the region are offshored, compared to48% on average. Here, cost reduction is afar greater priority for TCOE operations thanelsewhere: more than half of respondents(54%) identify cost reduction as the primarydriver (compared to just 27% on average).So as markets mature, cost becomes lessof a driver, and the focus moves towardsreducing time-to-market.SectorsPublic organizations seem to be leadingthe way in TCOE adoption, because, alongwith Telecoms, Media, and Entertainment(TME), they can claim the lowest proportionof organizations with no plans to implementa TCOE (30%). They have also experiencedthe steepest drop in this proportion – downmarkedly from 59% last year.This is likely being driven by the costreduction imperative that is behind manyorganizations’ TCOE plans. With spendingausterity being implemented by manygovernments, public bodies are being askedto do more with less, while maintainingservice levels and quality. As an efficiencydriver, an onshore TCOE is a compellingmodel, particularly as offshoring is apolitical anathema to many countries.For other sectors, the imperativeis responding faster to consumerexpectations, changing market demands,and time-to-market. Two-thirds (66%) ofCPRD respondents now operate a testingcenter, or plan to – significantly up from41% last year. Streamlining of supply chainsand application testing would appear tobe complementary. And this upturn is alsotrue of the FS industry, where adoption is upfrom 58% to 69% .Finally the High Tech sector, which includesaerospace and defense companies,appears to be closing down internal centersin favor of an outsourced model, as time-to-market is a major issue in this sector. Thesector had a remarkably high proportionof in-house facilities up and running lastyear (17% compared to an overall averageof just 4%), while only 8% planned towork with a third party. This year, thetables have turned. Just 6% now havetheir own TCOE, while 17% plan to use anoutsourced partner.This year’s data indicates that TCOEs aregaining traction, with growing levels ofadoption across the board. As a flexiblemodel, it offers significant advantagesand opportunities to fully leverageexisting resources.The State of Quality 2012Testing Centers of Excellence22 WORLD QUALITY REPORT 2012-13
  22. 22. Base: 1553 RespondentsPercentage of offshoring compared to costreduction as A reason for TCOE FIGURE 6NORTH AMERICA UK AND IRELAND NORDICS55% 50% 50%15% 16% 44%WESTERN EUROPE SOUTHERN EUROPE EASTERN EUROPE49% 49% 43%29% 25% 32%ASIA SOUTH AMERICA AUSTRALIA AND NEW ZEALAND36% 38% 42%29% 54% 20%GLOBAL AVERAGEPercentage of testersoffshore or nearshorePercentage of respondents citing costreduction as reason for TCOE48% 27%Organizations are turning to TCOEs toconsolidate their QA operations in responseto market pressures, technology demands,and emerging models of testing deliveryThe State of Quality 2012Testing Centers of Excellence WORLD QUALITY REPORT 2012-13 23
  23. 23. Mobile Testing:Behind the curveMobile has certainly changed the gamefor enterprise IT. Organizations aroundthe world now need to deliver continuousaccess, anytime and anywhere, tothousands of employees and millionsof customers, over a bewildering arrayof devices.Due to cost pressures, firms are no longerwilling to pay for, and thus control, theiremployees’ mobile-access devices, leadingto a bring-your-own-device culture. Thesituation is further exacerbated becauseorganizations are unwilling or unableto support the proliferation of personaldevices such as smartphones and tabletsbeing used at work by employees. However,as the prevalence of non-companyapproved devices becomes mainstreamand employees demand access to theflexibility and productivity that mobileoffers, IT departments are now beingforced to embrace this as the new realityand address the unique challenges of thisparadigm shift.Mobile has also turned IT securityinside out. A firm’s entire infrastructurecan no longer be tucked safely behind avirtual private network and firewall. What’smore, employees are prepared to downloadand try out apps on mobile devices with alatitude they would not consider acceptableon a desktop or laptop.In addition, the need for mobility hasreset expectations of what constitutesapplication quality. With traditionalsoftware, users expect flawlessfunctionality first and foremost, but mobileusers are seeking convenience. They expectrobust performance and usability on themove, and are more inclined to tolerate theoccasional glitch along the way, as long asthe application performs well and is user-friendly. QA teams may need to rethinktheir testing strategies and prioritiesdesktop application. Performance is farahead of functionality (cited by 48%)on firms’ priority list for mobile testing.Security, a perennial concern where mobileis concerned, is a priority for less thana fifth (18%) of organizations. This doesnot mean that companies are no longerconcerned with security, but rather thatthe perspective on security has changedwith the bring-your-own-device cultureand method of disseminating apps. Unlikewith traditional desktops, companiescannot control what apps are installed onpersonal devices, so the concept of securityis shifting from a denial of access to theprotection of sensitive data and ability towipe clean certain apps or data.Even lower on the radar is the certificationof applications, cited by just 14%, despitehigh levels of inconsistency in the quality ofmobile apps on different operating systems.Specialist partners areplugging the gapThe number one criterion when selectingexternal partners to assist with mobiletesting is the capacity to test across severalnetworks, identified by almost two-thirds(62%) of organizations. This reflects theneed to ensure coverage, as businessesseek partners who can deliver testing in avariety of environments.Interestingly, cost reduction is less of aconsideration (cited by only 25%), despitebeing high on the list of factors driving bothcloud and TCOE adoption. This may be dueto a number of factors: mobile applicationsare relatively cheap to produce, the focusis on managing risk rather than cost,and firms’ need to address their lack ofreadiness and expertise where mobile QAis the mobile era adds a further level ofcomplexity to users’ needs, demands,and expectations.Underprepared and ill equippedOur research suggests that organizationsmay not be giving mobile the priorityit deserves. Only 31% of respondentscurrently test mobile applications, andthose surveyed readily admit to being illequipped for mobile testing. This suggeststhat QA has fallen behind the mobile curve.A troubling two-thirds of organizations(65%) do not have the right tools to testmobile applications, and equally concerningis the 52% who do not have access to therequired devices. It would appear thatorganizations either do not have a goodgrasp of the infrastructure challengesposed by the mobile era or are unable toaddress these new demands. Meanwhile,a third of organizations lack the testingmethodologies and processes (34%) andspecialist expertise (29%) necessary toeffectively certify mobile applications.Performance is the keyto successOrganizations that are conducting mobiletesting seem to be working towards adifferent set of quality standards ascompared to traditional testing. Theprimary focus is on the application’sperformance rather than its functionality.Efficiency of performance was identified asa focus for mobile testing activity by almosttwo-thirds (64%) of firms.The usual priorities – such as functionalityand security – are being pushed lowerdown the scale as companies place moreemphasis on the demands of end-userson mobile platforms, which are differentfrom the demands of a traditionalThe State of Quality 2012Mobile Testing24 WORLD QUALITY REPORT 2012-13
  25. 25. Percentage of respondents testing mobileapplications, by region FIGURE 9Base: 1553 Respondents24%38%22%38%37%21%26%30%39%EASTERN EUROPESOUTHERN EUROPE SOUTH AMERICANORDICSWESTERN EUROPEAUSTRALIA AND NEW ZEALANDNORTH AMERICA ASIAUK AND IRELANDRegional variationsWe might expect the strongest emphasison QA in the mobile arena to be seen inAsia in particular, as smartphone andtablet penetration are accelerating at ablistering pace in China. But the distinctionbetween East and West economies is notclear-cut. Indeed, in Asia, the proportionof organizations conducting mobile testingsits at 30%, close to the all-sample averageof 31%.Regions with traditionally high levels ofmobile adoption, with the exception of Asia,have – unsurprisingly – the highest levelsof mobile testing. Over one-third of firmsin Australia and New Zealand, the Nordics,and Southern and Western Europe currentlytest mobile applications.Regions displaying lower levels of mobiletesting include Eastern Europe and the UKand Ireland, where only a fifth of businessescarry out mobile testing, and just a quarterdo so in North and South America.The State of Quality 2012Mobile Testing26 WORLD QUALITY REPORT 2012-13
  26. 26. 18%65%52%29%26%79%58%34%Greatest challenge whentesting mobile applicationsand devicesFIGURE 10Base: 485 RespondentsDO NOT HAVE THERIGHT TOOLS TO TESTDO NOT HAVE THE DEVICESREADILY AVAILABLENO MOBILE TESTINGEXPERTS AVAILABLENOT ENOUGHTIME TO TESTALLRESPONDENTS ASIAAsian challengesThe explosion of the mobile market in Asia isreflected in the QA challenges and priorities facedby organizations in the area. A greater percentageof Asian firms confirm they faced the commonmobile challenges as compared to all the companieson average.Asian firms also place much more emphasis onportability and compatibility across devices whenconducting mobile testing. Portability is identifiedas a priority by 58% of businesses in this region,compared to less than half (46%) on average.Compatibility is cited by almost as many (55%),compared to less than a third (31%) overall.This focus might be driven by the far wider selectionof devices available to users in the region. In China,for example, mobile customers can choose fromjust about any handset available worldwide. In theUS and Europe, networks offer only the handsetsfor which they have contractual arrangementswith manufacturers. As a result, the choice availableto customers is far more limited.Performance versus functionalityElsewhere, the emphasis on application performanceis particularly acute in the UK and Ireland (75%),against an average of 64%. This could be due toits being a more mature market where end-usersdemand more from their mobile apps. Consider thelandscape a decade ago, when having a browser on amobile device was touted as a new feature, whereastoday these same users want apps that are tailoredfor specific needs. The opposite is the case in SouthAmerica and Eastern Europe, where only around halfof organizations focus on performance.Instead, South American firms are focused primarilyon functionality, and more than four-fifths identifythis as a priority, compared to just under half overall.Mobile security is also a heightened concern inthe region, cited by 30% against an 18% average.Eastern European businesses are even more worriedabout security (36%), and are equally concernedwith application certification (compared to just 14%on average).The State of Quality 2012Mobile Testing WORLD QUALITY REPORT 2012-13 27
  27. 27. Sector perspectivesA look at mobile testing by industry throws up somesurprising results. Firstly, it is Manufacturing firms,rather than those from more consumer-facingindustries, that are most likely to conduct mobiletesting (37% compared to 31% on average).We might expect the TME sector to be most at easewith mobile technology. However, their QA demandsare uniquely complex, comprising the need to testnetworks, equipment, and billing systems, amongothers that are applicable only to TME firms. As aresult, just 30% of firms in the technology-intensiveTME space currently test mobile applications,equal to the all-sample mean. Some 62% lack thenecessary tools to test mobile applications. This isonly slightly better than average (65%), and almostone-third (32%) lack in-house expertise, comparedto just 29% overall. Perhaps this is due to the TMEproviders focusing their QA efforts on the mobileinfrastructure and leaving the testing of mobiledevices to the handset manufacturers.Public Sector organizations (27%) are the slowestto adopt mobile testing. This may not be quite sosurprising in a sector not traditionally viewed asa leader in technological innovation. But it doespresent a major issue in an era when citizensincreasingly expect to interact while on the movewith public service providers, just as they do withcommercial brands.“It’s an area of our business that typifies how things are changing,in my opinion. The guys in marketing say ‘we need a mobile app’; salesor product execs agree and then send it across to development andtesting to push it through in half the time we might actually need.It’s not going away either. More and more of our business is doneonline [in general] so it’s something we need to get hold of andunderstand quickly.”A retail business, United KingdomPublic bodies are also the least focused onperformance, although this is still their greatest areaof priority, cited by more than half of organizations(53% compared to 64% average). The sector placesa higher emphasis than average on security and userexperience, in line with their public responsibilities.Despite its reputation for innovation, the FSindustry faces some of the most acute mobiletesting challenges. Almost three-quarters (72%)of FS firms lack the right tools, compared to 65%on average, and 38% lack mobile testing expertise,against 29% overall. Financial firms are the mostacutely concerned about application performancewith 72% citing this as a priority.User-friendliness is an understandable priority forthe CPRD space. Ease of use and the user interfaceare a focus for 50% of CPRD firms (compared to 36%on average).Mobile is certainly having a significant impact on theQA function and it seems that QA has been slow toreact to the expectations from within the business.We would expect swift reassessment of testingpriorities and increased investment over the comingyear to handle the rush of mobile applications eitherin-house or with specialist providers.The State of Quality 2012Mobile Testing28 WORLD QUALITY REPORT 2012-13
  28. 28. Base: 485 Respondentsgreatest challenge when testing mobileapplications or devices, by sector FIGURE 12CONSUMER PRODUCTS,RETAIL, AND DISTRIBUTION TRANSPORTATION ENERGY AND UTILITIES FINANCIAL SERVICESMANUFACTURING HIGH TECH HEALTH AND LIFE SCIENCESTELECOMS, MEDIA, ANDENTERTAINMENTPUBLIC SECTOR63% 50% 27% 23% 27% 20%63% 43% 40% 20% 26% 31% 62% 35% 50% 12% 23% 12% 66% 64% 22% 26% 22% 17% 62% 62% 30% 32% 12% 16%64% 51% 40% 30% 13% 19%59% 56% 41% 22% 7% 26% 68% 42% 32% 29% 29% 23% 72% 50% 36% 38% 21% 14%DO NOT HAVE THE DEVICESREADILY AVAILABLEDO NOT HAVE THE RIGHTTESTING PROCESS/METHOD Not enough time to testNo mobile testing expertsavailableDo not have in-house testingenvironmentDO NOT HAVE THE RIGHTTOOLS TO TESTBase: 1553 RespondentsPercentage of respondents who test mobileapplications or devices, by sector FIGURE 1127%PUBLIC SECTOR37%MANUFACTURING35%ENERGY ANDUTILITIES35%HEALTH ANDLIFE SCIENCES34%CONSUMERPRODUCTS, RETAIL,AND DISTRIBUTION31%TRANSPORTATION31%FINANCIAL SERVICES30%HIGH TECH30%TELECOMS, MEDIA,AND ENTERTAINMENTThe State of Quality 2012Mobile Testing WORLD QUALITY REPORT 2012-13 29
  29. 29. Cloud: Gathering pace“The cloud is something that has become a significantpriority for our organizational direction in theInformation Technology Department. With regardsto testing, it’s opening up a way of turning on andoff a stream of new skills and abilities in a way thatis both fast and cost-effective when we need them.We can reach out and get exactly what we needwithout delay.”A financial services business, The NetherlandsCloud computing continues to generate asignificant amount of attention as almostevery market forecast indicates exponentialgrowth in this area. Within this climate,cloud adoption in QA is also gatheringmomentum, because the business case isresoundingly clear. As more applicationsare being hosted in the cloud, the necessityfor testing in cloud environments isgrowing apace. And organizations areseizing the benefits that the cloud offersby buying services on demand and gettingaccess to physical resources more quicklyand cheaply to cut QA spend.Cloud coverage on the riseThe extent to which businesses and theirQA function are now embracing the cloudmarks a step up from the evolutionaryprogress revealed in last year’s report.Our research shows that almost a quarter(22%) of software applications are nowhosted in the cloud. This is expected to riseeven further to a third (32%) by 2015, anindication of the acceptance of cloud aspart of the mainstream of IT infrastructure.Testing in the cloud is also expanding at acorrespondingly healthy rate. Some 28% oftesting now occurs in a cloud environment,a proportion forecast to rise to 39% by 2015.Only two years ago, some 31% of companiesdeclared they would not test via the cloudover the next three years, and this hasnow dwindled to just 4%, a clear indicationthat testing in the cloud is becoming thenew norm.28% of testing currentlyoccurs in a cloudenvironment, andis set to rise to 39%by 2015The State of Quality 2012Cloud30 WORLD QUALITY REPORT 2012-13
  30. 30. PERCENTAGE OF APPLICATIONS HOSTED IN THE CLOUD,NOW AND 2015 FIGURE 13Base: 1,553 respondentsPERCENTAGE OF TESTING CONDUCTED IN THE CLOUD,NOW AND 2015 FIGURE 14Base: 1,553 respondentsNOW 201522% 32%NOW 201528% 39%The State of Quality 2012Cloud WORLD QUALITY REPORT 2012-13 31
  31. 31. “We experience peaks and troughs in our testing program, so TaaSprovides a certain degree of flexibility. While some applications maybe run of the mill, some may have more ad hoc requirements thatpush us to meet unique or temporary needs. TaaS allows us to dothis without the burden of great cost or adding to project time.”A retail business, DenmarkRising demand for on-demand servicesAs QA professionals become more comfortable with the cloud as atesting platform, organizations are increasingly capitalizing on theon-demand benefits it provides.Companies that are accustomed to using cloud-basedinfrastructures are now demanding similarly flexible offerings fortheir software licenses in the form of Software as a Service (SaaS),where costs incurred are matched against actual usage.Organizations areincreasingly capitalizingon the on-demandbenefits of the cloudAlmost a third (31%) of firms’ testing software is currently providedon a SaaS basis – compared to only a quarter (25%) last year. Thisis expected to reach close to half (43%) by 2015. Furthermore,some 30% of firms had no SaaS arrangements in place for testingsoftware in 2011; this has fallen to just 8% this year.Throwing caution to the cloud?Conventional wisdom dictates that lingering doubts over the cloudand, in particular, its perceived lack of security would discourageorganizations from migrating their most business-critical ITinfrastructure to the cloud. Contrary to this assumption, oursurvey reveals a growing tendency to test critical applications incloud environments.In fact, organizations are actually testing more critical applications(61%) – both internal and external – than non-critical software(39%) in the cloud. However, it is worth noting that internal softwareaccounts for two-thirds of these critical applications, and externalsoftware just a third.Testing as a ServiceAlthough starting from a low base, firms are also increasinglytreating the testing activity as an on-demand utility, giving rise tothe term Testing as a Service (TaaS) – the delivery of applicationtesting services on a pay-per-use model, using cloud-based testenvironments and software tools. While only 11% of businessescurrently use TaaS arrangements, almost four-fifths (78%) plan tointroduce the concept, and almost half (47%) will do so within thenext 12 months. By 2015, only 11% will have no TaaS arrangements, apointer to how ubiquitous this delivery model will have become.An opportunity to reduce costWhen asked about the benefits of TaaS, respondents were emphaticin their enthusiasm. The most highly-anticipated benefits from TaaSarrangements were reduced cost, identified by 58% of businesses,and more effective management of testing resources (cited by 49%).By contrast, reduced time-to-market (the main driver for TCOEconsolidation) only ranks third (42%), and improved quality is evenless of a focus, mentioned by just 24%.The cost driver is clearly important when assessing TaaS services,as the focus is on how to reduce, for example, licensing costs, to payonly for what is used, as opposed to improvement in quality that canbe achieved through other means.So why would organizations view cloud-based TaaS as a greatercost-saving opportunity than consolidation, as provided by TCOEs?TaaS offers the chance to bring down costs that have alreadybeen heavily optimized and the greater portion of testing spendgoes towards resource costs. In mature markets, this element hasalready been drastically reduced by successive waves of offshoringand restructuring. And in developing markets, low labor costs area given.So, adopting a cloud utility service model presents a rareopportunity to further squeeze costs by paying only for what isneeded when it is needed, as opposed to high investment in oftendormant technology. TaaS also provides increased flexibility inservicing and pricing and the ability to quickly scale up or down asrequired for ad hoc or ongoing projects.The State of Quality 2012Cloud32 WORLD QUALITY REPORT 2012-13
  33. 33. Low hurdles to entryEncouragingly, as adoption has increased,the cloud in general is now seen as a fargreater opportunity than a risk. The potentialchallenges associated with cloud-basedoperations provoked far less responseamong respondents than the benefits, sothe gains clearly outweigh the pitfalls.The benefits ofcloud-basedQA activityclearly outweighthe pitfallsThis can be seen most sharply in the halvingof levels of concern over last year’s biggestworry – security. Predictably, back in 2011,more than half (55%) of organizations sawsecurity as the most acute risk associatedwith using a cloud environment. This year,taking TaaS as an example of a generalincreased ease with using cloud-baseddelivery, security came only third on thelist of the challenges anticipated whenadopting TaaS. Even the starkest currentchallenge – loss of control over total testingspend – is troubling the minds of less than athird (32%) of companies.Yet, despite the overall enthusiasm, theseconcerns about loss of control over spendsignal a continuing concern and confusionabout the cloud generally. In our experience,there is little reason to lose control oftesting spend, as this can be preventedthrough robust governance, spend metrics,and service-level agreements.Perhaps this anxiety is a legacy of pooroutsourcing experiences, where contractshave exceeded cost estimates. Or it mayreflect persistent economic instability:unable to reliably forecast market demand,businesses are struggling to calculate anaccurate assessment of the scale of theirtesting requirements. Whatever the reason,we would encourage organizations to lookto the benefits of this new model of testingservice provision, as the concerns canbe managed.Mature markets driveon-demand provisionLast year’s report found emergingeconomies had the most aggressiveplans to migrate QA activity to the cloud.In 2012, we see this leveling off, as cloudpenetration in developing regions begins tocatch up with that of mature markets.As indicated earlier, as an example of cloudadoption, companies are increasinglylooking to TaaS as a means of costreduction and process efficiency. How isthis reflected at a regional level? Plansshow little variation between the globalregions in the immediate term. For the mostpart, the more mature markets are drivingon-demand uptake, with a high percentageof firms already adopting or planning to useTaaS services. Only 3% of North Americanfirms have no plans to use TaaS, with asimilar figure of just 5% in Western Europe.In Asia, only 7% do not anticipate TaaS intheir future.Yet this jumps to almost half (46%) inEastern Europe and nearly a third (29%)in South America with no TaaS plans.However, looking ahead, South Americais predicting a significant change, withrespondents expecting almost half (46%)of testing to be cloud based by 2015,compared to the 39% global average.For all regions, reduced cost is the primarybenefit of TaaS, and taking into accountbetter resource management, efficiencyis the secondary driver for TaaS adoption.Emerging markets are particularly focusedon cost reduction – seven in 10 Asian andSouth American businesses identify this asthe primary benefit of TaaS, compared to58% across the whole survey.Sector leaders and laggardsTurning to the sectors, testing in thecloud is being led by the IT-intensive TMEsector, where 31% of testing activity iscurrently cloud-based, followed by FS at30%, compared to the 28% sector average.Both the TME and FS sectors are drivenby technology and have traditionally beenearly adopters. Late adopters, with onlya quarter of testing in the cloud, includeE&U (25%) and Manufacturing industries(24%). These two sectors tend to view ITas an enabler rather than a driver, andtraditionally have spent less on IT.The pattern is repeated when it comesto leveraging TaaS. Only 6% of TMErespondents and 8% of FS companies haveno plans to use TaaS, compared to 11%overall. Public Sector organizations are alsoembracing on-demand testing, with only 7%having no TaaS plans in place.By contrast, some 30% of manufacturersare disregarding the on-demand testingprovision altogether, as are 16% of E&Ufirms, and, as in last year’s survey, CPRDbusinesses are slow adopters – currently17% have no plans for TaaS.Small firms reluctant to adoptOrganizations of all sizes are integratingcloud technology into their QA operations.Businesses are investing in the cloud tomodernize and optimize IT infrastructures,while small businesses look to takeadvantage of its flexibility in order tocircumvent the need for costly IT set-up.Large, medium, and small firms are allrunning approximately 28% of testingactivity in cloud environments. Likewise, allthree groups forecast this to rise to around39% by 2015.However, smaller firms are much morereluctant to conduct testing on demand.Almost a quarter (23%) have no plans touse TaaS, compared to just 6% of large andmedium-sized organizations. Again, this maybe driven by the fear that testing costs willspiral out of control or by a lack of awarenessof the true benefits that TaaS can offer.More small businesses (35%) identify thisas a challenge than their corporate (30%) ormedium-sized (31%) counterparts.Our research shows that the move towardsa cloud-based infrastructure, supported byofferings such as TaaS, continues to growacross companies of all sizes. Organizationsare primarily looking to achieve a morecost-effective and efficient means for QAthrough a flexible, agile, on-demand model.We expect that this trend will accelerateas the cloud matures and competitivepressures force non-adopters to reconsidertheir position.The State of Quality 2012Cloud34 WORLD QUALITY REPORT 2012-13
  34. 34. “We are moving moreof our testing intothe cloud, but it’salways going to beproblematic for us.National legislationrestricts the natureof data that can bestored and accessedoutside the country;therefore we arecurrently limited asto what exactly canbe done via the cloud.”A manufacturingbusiness, GermanyThe State of Quality 2012Cloud WORLD QUALITY REPORT 2012-13 35
  35. 35. Quality AssuranceResources: A questionof valueIn an increasingly digitized era, the ability tomonitor and assess the quality of softwareapplications is critical to an organization’soperations – not to mention its reputation.So confidence in the quality of QA resourcesto deliver on this priority is essential. Ourdata shows that this, unfortunately, is notalways the case.Our study finds that organizations considertheir internal QA resources to be “average”overall, which we believe is of some concerngiven the increasing importance beingplaced on testing. On balance, companiesbelieve that they can secure slightly betterquality resources via external partners thanby hiring internally.One contributing factor could be that,historically, testing has not been viewed asa well-defined professional career acrossthe world, although it is gratifying to seemuch evidence of this now changing. Indeveloped countries, QA is becoming arecognized and dedicated profession,commanding increased salariescommensurate with the level of expertiseand experience. In the less-developedmarkets, there is often a limited number oftrained resources available, which is drivingthe need to use external expertise, often ata premium to internal cost structures.A fine balancing act betweeninternal and externalThe majority of organizations (59%)characterize their internal QA teams as“average” in their QA knowledge and notnecessarily up to speed with the latesttesting tools and technologies. Externalresources fare better with slightly fewercompanies (52%) describing externaltesters in the same way. More significantly,only 4-5% of firms are confident that theirresources (internal or external) are bestin class.However, when apportioning high praise,companies express more confidencein outsourced resources than theirown employees. A third of firms (33%)score their external teams’ knowledge andabilities a 4 or 5 (with 5 being the highest);less than a quarter (24%) score internalresource as highly.This may reflect the fact that many externaltesters work for service providers whosebusiness model is to offer large-scaleprofessional teams of resources designedto elevate a company’s existing capabilities.It may also reflect an increasing desireamong certain economies to increasethe level of QA skills they can offer to theglobal community.Our study found minimal regionaldifferences in confidence levels, whichmay reflect the fact that in the moremature markets, expectations are higherand the need to outsource greater, whilein the emerging markets, expectations arelower and the available talent is startingto come through. Whichever the case, it isclear that there is room for improvementin the skill levels of QA resources, if theyare to meet the exacting demands beingplaced on them.A question of valueInternal QA resources, while rated lower interms of knowledge, are marginally moreexpensive, which is to be expected, asorganizations seek external outsourcedskills predominantly to reduce costs. Firmsare paying a global average hourly rate ofUS$54.58 for internal resources, comparedto US$52.62 for external support.Although this weighted average is notdesigned to be exact, the comparisonis revealing, as it would indicate thatorganizations are getting more valuefrom outsourced support, because theyare paying less per hour for a resourcethat is more highly regarded, althoughthe difference is small. This is potentiallythe case in economies where there is agreater number of available testers, andoutsource providers compete on costagainst in-house teams.However, these figures need to be viewedin context. Hourly rates naturally varygreatly by geography, and price needs to beconsidered alongside the value that firmsperceive they are getting for their investment.Regional perceptionsIndeed, regional comparisons paint acontrasting picture, divided broadly alongmarket maturity lines. While there isgeneral agreement across the regions thatexternal resources are more effective, thereare wide disparities between how mucheach region is paying for both external andinternal support.In developed regions, there is a pricepremium on internal testing capacity. NorthAmerican firms, for example, are payingaround 20% per hour more for internal thanexternal resources. We believe this reflectsthe increased standing of QA resourcesin North American organizations and thesalaries they can command.The State of Quality 2012Quality Assurance Resources36 WORLD QUALITY REPORT 2012-13
  36. 36. Base: 1553 RespondentsINTERNALRanking of external and internal testing resources(scale of 1 to 5, where 5 is highest) FIGURE 17EXTERNAL5%0% 19%59%17%4%29%52%14%1%THEY ARE KNOWLEDGEABLE INTESTING METHODOLOGY. THEYARE BEST IN CLASS ACROSS THETESTING INDUSTRY AND THETESTING ACTIVITIES RUN SMOOTHLYTHEY ARE AVERAGE IN THEIRKNOWLEDGE. THEY KNOW ENOUGHTO PERFORM THE JOB, BUT THEYMAY NOT BE UP TO SPEED ON THELATEST TESTING TECHNOLOGIESTHEY DO NOT EXECUTE THETESTING ACTIVITIES EFFICIENTLYAND MANY DEFECTS ESCAPE THEIRDETECTION. THEY DO NOT HAVEDEFINED TESTING PROCESSESAND ARE NOT USING TESTINGTECHNOLOGY EFFECTIVELY54321“Outsourced testing will always be an importantoption, but how we source those testers doestend to depend on project or cost, rather thanany specific predefined rules. We used to useIndia-based suppliers a lot more than we dotoday, but the cost benefits were not balancedby our internal client satisfaction feedback.”A financial services business, AustraliaThe State of Quality 2012Quality Assurance Resources WORLD QUALITY REPORT 2012-13 37
  37. 37. $47Average hourly rate (US$) for external and internal testers FIGURE 18NORTH AMERICA UK AND IRELAND NORDICSWESTERN EUROPE SOUTHERN EUROPE EASTERN EUROPEASIA SOUTH AMERICA AUSTRALIA AND NEW ZEALANDGLOBAL AVERAGE$32Base: 1553 Respondents$41$30$23$48$41$68$68$40$51$48$58$57$85$44$36$55$53INTERNALEXTERNAL$66The State of Quality 2012Quality Assurance Resources38 WORLD QUALITY REPORT 2012-13
  38. 38. As we have seen, North America is the mosthighly outsourced region, with businesseshaving transferred large proportions of theirQA to lower-cost, offshore labor markets.As a result, they are paying well below theaverage for external support, and balancingthis by paying a relative premium for highquality in-house experts.Yet interestingly the quality rating doesnot differ by much between the twosets of resources. This may indicate apragmatic approach from North Americanorganizations in assessing both sets ofresources on a relative value-for-moneybasis, balanced by availability constraints.This indicates a mature and measuredapproach in North America.In emerging economies such as SouthAmerica and Eastern Europe, however, thereverse is true. In these areas, QA is a less-developed, less-professionalized field; insome organizations, it may barely exist asa recognized function at all. As a result,dedicated external resources are morehighly prized, and so come at a premium.South American companies pay around30% more for external than internal teams;in Eastern Europe, the disparity is around25%. The conclusion is clear – emergingmarkets may not immediately be savingcosts by using external expertise, butjudging by the recognition of the skills ofexternal resources, companies are buyingan enhanced quality of output.Sector perceptionsOne industry that does recognize theprice premium and value of experiencedQA resources is clearly FS. The industryis paying a hefty premium for internalsupport: $62.18 per hour compared to the$54.58 average. The FS sector is, of course,a unique, complex, and highly regulateddomain, requiring very specific functionalityexpertise in addition to generic QA skills.Not surprisingly, this rare combination ofskills pushes up the cost of QA capabilityfor the industry.Manufacturing companies are the mostsatisfied overall with the quality of theirQA resource. Just over a third (34%)of manufacturers score their internalresource a 4 or 5, compared to the overallaverage of 23%. Almost two in five (38%)rate their external support highly (average33%). It may be that the importance thesector places on ensuring the qualityof its production output extends to itssoftware applications, resulting in a greaterappreciation of technical ability.At the other end of the scale, only around aquarter of Public Sector organizations ratetheir internal (21%) or external (28%) QAresources highly. This might be a reflectionof constrained local pay scales, orcomplexity of large-scale legacy systems,meaning that quality may be more difficultto achieve.Two sectors – where a breakdown in qualitycould have dramatic consequences – rely onexternal expertise to compensate for a lackof confidence in their in-house resource.The Health and Life Sciences businessesand E&U have the lowest confidence in theirinternal teams: only a fifth of firms in theseindustries rate them highly, yet, by contrast,external resources in both are given arelative vote of confidence. Around twice asmany organizations in these industries rateexternal providers highly.Paying the price?Experienced QA experts – both in-houseand third party – possess a rarecombination of highly technical abilities:business domain expertise, businessprocess analysis skills and development,scripting, and testing capabilities, andup-to-date QA knowledge, to name a few.In our experience, when it comes to payingthe price for that expertise, organizationsoften have unrealistic expectations. Somemay be unwilling to pay the higher cost forhighly qualified, experienced professionals;others may be restrained from doing so bybudgetary considerations.Value for money is undoubtedly subjective.Getting the balance right between in-houseand external teams, factoring in technicaland business knowledge and relativecosts, is challenging and requires constantmonitoring and investment.“I’d love to be able to professionalize our testing function – give people acareer and the pay to match and they’ll stick with you. But the reality is quitethe opposite. Testing is seen as a ‘necessary evil’ in many organizations. Thismeans organizations never give [testers] the proper attention [they] deserveand we end up buying external experts who may have the skills, but notnecessarily the intimate knowledge of our business.”A financial services business, United KingdomThe State of Quality 2012Quality Assurance Resources WORLD QUALITY REPORT 2012-13 39
  39. 39. sectoranalysis42 Consumer Products,Retail, and Distribution:Mobile and internet commercedrives sector transformation44 Energy & Utilities: Commitmentto improve quality of testingthrough increased investment46 Financial Services:Evolving QA – Riding thetransformation wave48 High Tech: Increasing focuson data-driven customerservice delivery50 Public Sector: Renewedfocus on efficiency gainslinked to budget reductions52 Telecoms, Media, andEntertainment: Maximizingassets, unleashing growth,and transforming to succeed40 WORLD QUALITY REPORT 2012-13
  40. 40. WORLD QUALITY REPORT 2012-13 41sector analysis
  41. 41. The Consumer Products, Retail, and Distribution (CPRD) sectorrepresents a large and diverse group of companies, includingconsumer products manufacturers, retailers, distributors (such aslogistics companies, postal systems, trucking), and transportation(including airlines, airports, and rail operators). Organizationsin each of these sub-sectors have distinct technology needs.However, they are all part of the broader consumer value chain andtherefore need to be extremely responsive to the changes in demandcoming from the end consumer. And that consumer is increasinglyempowered by technology.Given the growing importance of technology enablement duringthe shopping journey – from awareness and choosing, through thetransaction, and to delivery and service – the CPRD sector todayis more focused on mobile and internet commerce. As a result, weexpect to see an increase in testing needs for transformationalprojects, cloud testing, and Testing as a Service (TaaS), as well asgrowth in adoption of Testing Centers of Excellence (TCOEs).CPRD companies report investing more in QA/testing than the overallmarket and other sectors, with 22% of total IT budget going towardthe testing function. Last year, 35% of respondents in the CPRDsector said their QA budgets had increased over the previous twoyears; this year, 41% say it has increased and 57% say they expect tosee this increase again by 2015.This indicates that companies are continuing to invest in QA/testing in order to stay competitive and meet the high expectationsof consumers. In the growing, interconnected world of consumersand customers, failure is quickly and easily communicated, socompanies cannot afford to provide less than perfect products/services to customers.A high 45% of CPRD respondents say their testing activities arewith new or transformational projects, which reflects the need forinnovation in this sector. Asked in which areas of QA the greatestincrease in spending has occurred, 51% of companies say it is oninternal professional testing resources, 45% on more investment inexisting tools, and 44% on external outsourced resources.ConsumerProducts,Retail, andDistributionMobile and internetcommerce drives sectortransformationBy Bernard HeldersGlobal LeaderConsumer Products & Retail SectorCapgemini Etienne CartignySenior Test ManagerCapgemini Rashmi SinghSenior ManagerCapgemini42 WORLD QUALITY REPORT 2012-13sector analysisConsumer Products, Retail, and Distribution
  42. 42. Interestingly, when asked to look ahead to 2015 and where theythought the greatest area of investment would be, the top answer(48%) was in using external outsourced resources – perhapsreflecting the need for more specialized and specific resourcesrequired to test leading-edge services and products, such asmobile commerce.Across all sectors and respondents this year, our survey saw thecontinuing growth of the adoption of TCOEs. This is also reflected inthe CPRD sector, but it is clear that organizations in other sectors aretaking a slightly different path. While only 6% of CPRD respondentssay their in-house TCOE is already fully functional (on par withthe overall figure study-wide), a further 25% say they are alreadydeveloping or planning to develop an internal TCOE over the nexttwo years. This compares to 40% of the overall survey.The difference is apparent in the percentage of organizationsplanning to use a third-party company with a TCOE capability;35% of all surveyed CPRD firms (compared to 21% overall) say theyare taking this TCOE route. The scope of their ideal TCOE is broad,but executing performance testing (50%) and providing ProgramManagement Office services (44%) are top capabilities. The keybenefit in adopting a TCOE for CPRD sector firms is reducing time-to-market and reducing costs.The growth in the impact of the cloud continues to change the wayorganizations approach multiple areas of their operations, andthis is no different for CPRD firms. Respondents say 21% of theirapplications are currently cloud based, with an expected figure of32% by 2015. Indeed, our survey suggests that the percentage oftesting in a cloud-based test environment will rise by almost 50%by 2015 – suggesting CPRD firms are keen to exploit this growingmethodology for their testing function.Asked about TaaS offered by a third party, just 9% of CPRDrespondents say they already use it, but 45% say they will within thenext year, and a further 28% within the next two years. With 59% ofrespondents seeing reduced costs and 48% seeing better resourcemanagement as key benefits, TaaS appears to deliver distinctadvantages for an organization’s testing. The survey also sees apredicted increase in the use of Software as a Service (SaaS) for QA/testing software licenses between now and 2015, no doubt part ofthe broader move towards the cloud.Some 34% of CPRD respondents say that their organizationscurrently test mobile applications and devices, slightly higher thanthe overall figure of 31%. Given the growing importance of mobileand internet commerce, consumer/customer mobile communicationand associated services, it is hardly surprising that this is an area ofexpected growth, particularly for the CPRD sector. However, it comeswith new and additional QA/testing requirements, which manyorganizations are finding challenging.So while organizations that test mobile applications focus onefficiency, performance, and functionality, there are shortcomingsin those testing processes. Some 63% of CPRD firms say they don’thave the right tools to test, and a further 50% say they don’t havethe devices readily available to test. Indeed, when asked about theirkey criteria for selecting an outsourced mobile testing resource,60% of respondents say the capacity to test their application onseveral networks would be most important. A further 40% saythe key criterion is enabling their development team to focuson development.With the economy likely to remain challenging over the nextyear or so, CPRD companies will continue to explore new waysto stay competitive, while remaining innovative and attractive tobudget-minded customers. New developments in mobile and webcommerce will continue to offer new routes for the sector to exploit,but it must do so from a position of assurance and knowledge, asconsumers are unlikely to be loyal should they be disappointed.The rising and positive investment in QA/testing is no guarantee ofsuccess, of course. Investing in the right skills, tools, and experiencewill be essential to ensure that the challenges of remaininginnovative and competitive are balanced with those of ensuring thelevels of quality demanded by customers. WORLD QUALITY REPORT 2012-13 43sector analysisConsumer Products, Retail, and Distribution