• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
World Quality Report 2012-13
 

World Quality Report 2012-13

on

  • 249 views

World Quality Report 2012-13

World Quality Report 2012-13

Statistics

Views

Total Views
249
Views on SlideShare
249
Embed Views
0

Actions

Likes
0
Downloads
11
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    World Quality Report 2012-13 World Quality Report 2012-13 Document Transcript

    • fourth EdItIon2012-13worldQualiTyreporT
    • 4 Introduction 6 Executive SummaryThe State ofQuality 201212 Quality Budgets: Stepping up18 Testing Centers of Excellence:A growing priority24 Mobile Testing:Behind the curve30 Cloud: Gathering pace36 Quality Assurance Resources:A question of valuesectoranalysis42 Consumer Products,Retail, and Distribution:Mobile and internet commercedrives sector transformation44 Energy & Utilities: Commitmentto improve quality of testingthrough increased investment46 Financial Services:Evolving QA – Riding thetransformation wave48 High Tech: Increasing focuson data-driven customerservice delivery50 Public Sector: Renewedfocus on efficiency gainslinked to budget reductions52 Telecoms, Media, andEntertainment: Maximizingassets, unleashing growth,and transforming to succeedCountry andRegional Analysis56 Australia and New Zealand58 Brazil60 China62 France64 Germany66 The Netherlands68 The Nordic Region70 North America72 United Kingdom74 About the Study10 40 54 WORLD QUALITY REPORT 2012-13 3CONTENTS
    • Introduction4 WORLD QUALITY REPORT 2012-13 Introduction
    • Welcome to the fourth edition of the World Quality Report, which, as in previousyears, has been designed to provide the most comprehensive assessment of thecurrent state of enterprise application quality and testing practices availablefrom around the world.To enhance the quality and relevance of thedata gathered, we increased the number ofinterviews to more than 1,550 CFOs, CIOs,IT directors, and quality assurance (QA)directors around the globe, all of which werecarried out by phone. We also broadenedour geographical sample, focused moreintently on the ‘enterprise’ market, andincluded additional questions reflectingthe significant changes in our technologylandscape, such as the almost ubiquitousuptake of mobile communications.Changes taking place in the IT marketincreasingly require scalable, robust, andreliable applications and infrastructure.Further, the disruptive effects ofcloud computing adoption, increasedmobility, and device proliferation requireorganizations to develop applications thatcan seamlessly interface with an ever-broader range of distribution channels androutes-to-market. Our research is evidencethat a focus on quality management isincreasingly regarded as a strategic value.While this year’s research confirms manyof the trends from previous editions ofthe World Quality Report, there are also anumber of new emerging developmentsin the market and an increased paceof change. For example, since last year,many readers will have seen corporateannouncements about a growing numberof large-scale enterprise-wide managedtesting service contracts being awarded,and an increased uptake of Testing as aService (TaaS) solutions.Similarly, many organizations are lookingto develop Testing Centers of Excellence(TCOEs), which use a standardized testingmethodology, best practices, and tools,together with a flexible pool of professionalresources to ensure high levels of qualityand risk mitigation across all applications.Over the last four years, the World QualityReport, published by the CapgeminiGroup and HP, has established itself asthe largest annual survey of applicationquality and testing practices, with a growingreputation as a valuable benchmark fororganizations around the world. The reportis the result of close collaboration betweenCapgemini, Sogeti, and HP, both leaders inour respective fields of outsourced testingand Application Lifecycle Managementtools, services, and products developed toimprove the efficiency and effectiveness ofenterprise applications.We hope you find that the research,analysis, and commentary containedin this year’s report both informs yourown testing and quality assurancedecisions and perhaps challenges someof your current thinking. We also valueyour comments and ideas, and inviteyou to contact either of us or any of thecontributors to this report regarding anyquestion you might have concerning testingand quality assurance.Finally, we would like to thank all of theindividuals who participated in the researchand gave generously of their time. Withoutthem, it would not have been possible toproduce the World Quality Report 2012-13.You can also reach us by sending an emailto: worldqualityreport.s.uk@capgemini.comor worldqualityreport.s.uk@sogeti.comMichel de MeijerSenior Vice President, Leader Global Service LineTesting, Capgemini & SogetiMatthew MorganVice President, Hybrid IT and Cloud ProductMarketing, Software, HP WORLD QUALITY REPORT 2012-13 5Introduction
    • Executive SummaryThis is the fourth year that Capgemini, Sogeti, and HP have produced theWorld Quality Report. Our aim remains to examine the current trends inenterprise application quality across different industries and geographies forthe purposes of providing actionable insight for decision making. Our hope isthat readers will learn from our report and apply their new understanding toimprove the effectiveness of Quality Assurance (QA) and testing within theirown organizations.6 WORLD QUALITY REPORT 2012-13Executive Summary
    • Murat AksuVice-President, Global Alliance Executive for HPCapgeminiCharlie LiVice-President, Global Service Line TestingCapgeminiOur global survey was carried out againsta challenging economic landscape fororganizations of all sizes and sectors.Many countries continue to experience low,stagnant, or negative rates of economicgrowth, and many governments haveapplied harsh austerity measures aimed atreducing public expenditure. Some sectors,particularly public and financial services,are under greater scrutiny to get IT rightthe first time and ensure improved valuefor money. Social media and the speed bywhich information is exchanged are alsochanging the way consumers and providersof applications interact with one another.Consumers of applications enjoy anenhanced scrutiny over service providers,whereas the providers are looking forways to respond to the resulting pressure,while also keeping an eye on a potentialmarket uptick.We have focused our research on theimpact of this complex environment ontesting and QA, a discipline that is itselfalso undergoing a series of technologicaland cultural changes.Rapidly evolving IT landscapenecessitates innovation in theQA disciplineQuality Assurance is undergoing a quietbut steady evolution from in-housetesting generalists to a structuredand efficient discipline, with a greaterinfluence within the overall applicationdevelopment lifecycle. We observefrom our survey the emergence of amultifaceted discipline with an increasingrange of operating models at its disposal,that enables a streamlined and cost-effective output that is also aligned withbusiness needs.The level of QA and testing investmenthas proved to be resilient in adverseeconomic environments in most markets,and has stretched to accommodate anever-increasing workload. But now testingresources need to prove themselves moreeffectively and QA teams need to stayalert to the new disruptive technologiesand ensure their skill levels are attuned toboth market and internal expectations. Weexplore these findings in more detail below.Has QA been caught off guardby mobility?As mobile adoption has become almostubiquitous in developed markets, thebusiness imperative for mobile businessis clear. But our study indicates that speedof adoption and proliferation of handhelddevices, coupled with use of social media,seem to have caught enterprise testingby surprise. Organizations may not be givingmobile the priority it warrants. Only 31%of respondents across the world currentlytest mobile applications – a figure that doesnot deviate much from region to region, andthose surveyed readily admit to being illequipped for mobile testing. This suggeststhat QA has fallen behind the mainstreammobile curve.Reasons given are multiple: many reportthat they don’t have the ability to test oreffectively certify mobile applicationsbecause of the lack of appropriate tools,processes, or expertise, and limited accessto the necessary devices. Moreover, thefocus is firmly on efficiency of performance,cited by 64% of firms, rather thanfunctionality, usability, or security. Overall,this argues for an underestimation of theinfrastructure challenges posed by themobile era, or an inability to address them.As today’s mobile users – customers andemployees – expect interaction at theirfingertips, anytime, anywhere, making abusiness mobile and “always on” should behigher up the corporate and IT agenda.New operating models such ascloud-based models and TestingCenters of Excellence havecome of ageCloud-based models for testing have had arelatively slow adoption rate, but evidencefrom our study of sustained wider-scaleadoption and falling barriers indicates thatorganizations are on the verge of muchmore extensive use. Testing in a cloudenvironment, for example, is growing at ahealthy rate. Some 28% of our respondentscurrently use this infrastructure model, aproportion forecast to rise to 39% by 2015.Moreover, only 4% say that they will notuse the cloud in some way over the nextthree years, down from 31% only two yearsago, a clear indication that testing in thecloud is set to become a new norm, asissues surrounding cloud, such as security,are addressed.Equally significant is the adoption of othercloud-based services. Software as aService (SaaS) continues to grow; almosta third (31%) of firms’ testing software iscurrently provided on a SaaS basis – upfrom 25% last year, and by 2015 we predictthat almost half of firms will use theon-demand model. WORLD QUALITY REPORT 2012-13 7Executive Summary
    • Testing as a Service (TaaS) – which canbe a combination of cloud-based testingenvironment and tools, and on-demand testexecution – is a cost-effective operatingmodel for ad hoc quality projects and alsofor longer-term testing provisioning thatdemonstrates similar rates of adoption. Wefound that 78% of respondents are planningto move to TaaS in the next two years, risingto 89% by 2015. This rapid take-up of TaaSis fueled by cost-reduction drivers andits flexibility as a complementary modelto more comprehensive outsourcing orin-house Testing Centers of Excellence(TCOEs). So while some still see the securityof the cloud as a continuing threat, theirnumbers are dwindling as the advantagesmore clearly outweigh the drawbacks.For TCOEs that have been underutilizedin the past, their time too has arrived.Undoubtedly an efficient approach toindustrializing testing activity, TCOEsact as a virtual command center, using astandardized approach and a flexible poolof available resources. Our data revealsthat, this year, 60% of respondents planon – or are – developing a TCOE, up from45% last year. In our experience, we seethe concept is now sufficiently advancedand its benefits more clearly understood:a TCOE enables organizations to meet thechallenges of the business – speed-to-market and doing “more for less” – head on.Can we put a price on quality?Technological innovations and initiatives areincreasing the workload for ever-stretchedQA teams, but the corresponding QA andtesting budgets appear to be weathering theeconomic storms. In 2011, these budgetswere not supporting the challenges facedin the marketplace, but this year’s surveyfinds testing budgets growing at a strongerrate than last year, with 42% reporting thatbudgets had risen over the past 12 months.While 18% forecast a fall when lookingahead to 2015, some 53% optimisticallyexpect budgets to rise, indicating a newdegree of confidence that the business ismore committed to investing in QA.Moreover, the focus appears to haveshifted from “business as usual” tasksto investment in transformationalwork, to drive enhancements, and59% of the budget is now spent oncustomer-facing applications. So prioritiesare changing to support the core areas ofthe business that will require an optimizedplatform for efficient delivery of testing,which we regard as a positive step.For the first time, the World Quality Reporthas established a worldwide benchmarkfor testing hourly rates, internallyand externally. Firms are paying a globalaverage hourly rate of around $55 per hourfor in-house testers, compared to $53for external support – a differential of $2per hour. Obviously, rates vary greatly fromregion to region and can be affected by thescarcity of available skills in local markets.Does this represent the price of quality?Only an individual business can make thatdecision, when evaluated against otherprime resources.Confidence in QA resourcesis a concernWhile budgets might be in reasonableshape, confidence in testing resources isnot resounding. A majority of organizationscharacterize their internal teams as“average” at best, in their knowledge ofcore testing processes and methodologies,and not necessarily up to speed with thelatest testing tools and technologies. Theirassessment of external testers is slightlybetter, with a third of organizations scoringtheir external testers’ knowledge andabilities as “above average”. But less than5% of firms are fully confident that theirtesters (internal or external) are “best inclass”. Despite initiatives and investment,especially over the last decade or so, thereare clearly lessons to be learned in terms ofeither real or perceived quality of output forboth providers and users of QA.Simply put, as the competitive landscapeforces organizations to update and optimizetheir testing resources and drive downcosts and time-to-market, the overallquality of testing resources needs tokeep pace or change, to satisfy or exceedthe perceived and increasingly complexrequirements of organizations in the future.The emergence of a more globaland uniform industryIn our previous surveys, we have focusedattention on the differences between themajor countries or regions surveyed. Thisyear we notice that while variances stillexist, there is a greater harmonization,with fewer major differences evidencedacross the globe, at least within theenterprise markets. One example isthat the proportion of an organization’ssoftware development lifecycle budgetthat is invested in testing, averaged acrossthe regions, is 18%, with remarkably fewregional variations.Our perspective is that testing is becomingless of a regionally facing discipline andthat, led by globally active companies, thegap between emerging markets and moretraditionally mature markets is narrowing.This is no doubt a reflection of the rapidinvestment in skilled resources andtools, led by India, that has taken place inoffshore and nearshore countries – China,South East Asia, South America, and,more recently, Eastern Europe, CentralAmerica, and parts of Africa. All pointto testing emerging as a more uniformglobal discipline.World Quality ReportrecommendationsFor providers and users of QA and testing, weoffer these suggestions on how to apply thefindings of this year’s World Quality Report.QA needs to be a formalized stepin the application lifecycleWhile QA’s status within the IT organizationhas improved compared to a decade ago,we still see instances of the testing functionbeing considered as an afterthoughtor viewed as a roadblock preventingan application from being deployedto production. IT can only be as agile orefficient as its weakest component.As IT continues to reinvent itself with newprocesses, paradigms, and platforms, theseinnovations rarely include aspects of QA.The fact that testing is not being broughtalong the journey of reinvention often resultsin QA being the bottleneck. Organizationsneed to understand the critical role of QA inthe application lifecycle and treat it with thesame respect as its peers.Mobility should become fullyintegrated into testing prioritiesWe were surprised by the relatively lowlevel of proactive structured testingin this increasingly essential area ofbusiness connectivity. We believe that8 WORLD QUALITY REPORT 2012-13Executive Summary
    • mobile testing needs to be a fully integratedelement of the QA discipline, so that themobile strategy of the enterprise takestesting into account right from the start.The strategy should consider the objectivesof the business owner, how the mobileapp is delivered, and the target userfor the app whether that be customers,suppliers, or employees. Organizationsneed to accept the paradigm shift broughtabout by mobility and embrace the newnotion of quality for mobile apps, whichis a departure from traditional standardsapplied to desktop applications.No doubt, standardization of devices willease the complexity over time, but in theshort term, proliferation of smartphoneand mobile devices, the roll-out of 4G,and use of social media will only continueto exacerbate the situation. As will theneed to focus on the user experienceand functionality testing as wellas performance. If organizations are toturn the mobile opportunity into a businessadvantage, some will need to “skill up” or“skill out”.Evaluate the breadth of cost-effectiveoperating models to see if it is a good fitIn the quest for greater testing maturitybalanced against return on investment,organizations have an increasing array ofoperating models from which to chooseto achieve their goals. Companies shouldfirst define the business goals they wishto achieve and then evaluate the myriadof models against these objectives. Eachmodel has its advantages and shortcomings,and corporate culture, appetite for risk, andneed for control versus flexibility all play apart in determining whether a new operatingmodel should be adopted and which one.On the journey from in-house testinggeneralists and developers to aprofessionalized discipline wanting toadopt “Shift Left” principles that embedQA earlier in the traditional softwaredevelopment lifecycle, organizations shouldcritically appraise their current set-up andreview how testing in a cloud environment,cloud-based TaaS, and TCOEs can providethose critical incremental efficiency andquality improvements.Pay closer attention to demonstratingQA’s business value for the moneyValue for money is, of course, subjective,but what does constitute a good returnon your QA investment? One measure isinvestment in QA/testing as a proportionof the total IT budget, averaging at 18%across the world. Below 18%, organizationsshould examine if they are investing enoughto achieve or sustain the quality required.Those spending more should review theiroperational efficiency.But more precisely, companies need toestablish specific measures to evaluatethe return on their overall investment,and testing departments and serviceproviders need to act “smarter” in their useof tools, models, and communication tostakeholders to demonstrate the businessvalue they provide in order to justify theproportional spend on QA.Re-evaluate the diversity of skills requiredThe perceived quality of testing resourceshas a bearing on value for money, andthe unexceptional assessment of abilityneeds examination. Over time, we expectthat businesses will be even moreexacting in their expectations of any andall testing resources when it comes tocomplex business-critical applications. Allproviders, in-house, contract, and external,should take notice.QA has made great strides in becoming aprofessional discipline with a distinct careerpath and specific skills. But this is a movingtarget; testing training at the individual andteam level needs to keep pace with the fast-changing landscape, and should be properlyfunded, rather than being sacrificed underthe pressure of deadlines.Budgets are currently at a strong level, aclear recognition of the value of the testingfunction to the organization, yet this isin contrast to the perceived value of theactual resources. This gap can certainlybe bridged by clearly targeting those skillareas that are essential to the business.ConclusionOur findings in this report indicate thatQA continues to make steady progresson repositioning itself further up thebusiness value chain on the journey ofreinvention, emerging from the shadows ofdevelopment teams to a value-for-moneyand mature discipline. Looking ahead, weanticipate further strides in the adoptionof “Shift Left” initiatives and greater use ofnew models such as crowd sourcing whichleverages knowledge and manpower froman undefined pool of resources, to improvevalue-for-money ratios and timelines. Butgreater improvement will not come aboutwithout significant investment – not onlyfinancially, in terms of budgets and skills,but also culturally.But organizations will need to exert firmbut flexible control to work across a widerrange of delivery models and technologies.Budgets will need to work harder to effectthe changes required to respond to a rangeof demands from time-to-market, costefficiency, and improved quality. And QA willneed to keep up with market expectations,remaining focused on providing real,measurable business value. WORLD QUALITY REPORT 2012-13 9Executive Summary
    • TheState ofQuality201212 Quality Budgets: Stepping up18 Testing Centers of Excellence:A growing priority24 Mobile Testing:Behind the curve30 Cloud: Gathering pace36 Quality Assurance Resources:A question of value10 WORLD QUALITY REPORT 2012-13
    • WORLD QUALITY REPORT 2012-13 11The State of Quality 2012
    • Quality Budgets:Stepping up“Investment is going to befocused on those elementsthat support our widerbusiness objectives, asthere is a need for [testing]and the business side to bemuch closer. Decisions needto be made more quickly andproduct development timeis shrinking – we need thetools and skills to be ableto answer that need.”A telecommunications business,The NetherlandsOrganizations are trying to balance market uncertainty with the need toposition for eventual growth, as well as tackling the challenges of social mediaand mobile technologies. At the operational level, a complex web of factors isforcing organizations to update and optimize their testing resources and internalprocesses to drive reduced time-to-market and lower costs, as well as improvequality using new beneficial delivery models, such as cloud and TCOEs.As a result, the QA function and QA budgets, in particular, are under the spotlight,because to achieve these stretch business targets, organizations need to leveragetheir QA capabilities more effectively. This requires a step-up in investment over theshort and medium term, even in times of economic uncertainty.Indeed, this year’s survey finds testing budgets are expanding at a healthier ratethan the incremental upturn seen in 2011-12, when growth was failing to keep pacewith an increasing workload, due to cost-cutting measures. This positive steppoints to a desire among businesses to invest for the future and meet internal andexternal demands quickly and effectively.An upward curveIn general, the survey provides good news regarding QA budgets, given theeconomic backdrop. Close to half (42%) of QA budgets have increased over the lastyear, with only 11% reporting a decrease. The greatest proportion (44%) saw nomovement either way, but in a time of uncertainty, maintaining the same level ofinvestment in the quality function is, in itself, a testimony to the importance of QAto the organization. Overall, this is perhaps a more positive situation than mighthave been predicted.Moreover, this upward curve is expected to continue over the coming three years,indicating a degree of optimism. A healthy majority (53%) expect their budgets torise between now and 2015, while just a fifth (18%) forecast a fall. This seems to bea universal trend with little variation by region or sector.The State of Quality 2012Quality Budgets12 WORLD QUALITY REPORT 2012-13
    • INCREASEDSIGNIFICANTLYINCREASEDSLIGHTLYSTAYED THE SAMEDECREASEDSLIGHTLYDECREASEDSIGNIFICANTLYNO BUDGETTO TESTBase: 1553 Respondents2015NOW8%3%8%45%10%26%36%6%3%4%7%44%CHANGE IN TESTING BUDGETS FROM LAST YEAR, ASCOMPARED TO NOW AND IN 2015 FIGURE 1The majority of organizationsexpect testing budgets to risebetween now and 2015The State of Quality 2012Quality Budgets WORLD QUALITY REPORT 2012-13 13
    • SPEND ON EMPLOYEE- VERSUS CUSTOMER-FACING APPLICATIONS,AND TRANSFORMATIONAL VERSUS MAINTENANCE WORK FIGURE 2Base: 1,221 respondentsCUSTOMER FACINGEMPLOYEE FACING41%59%VSMAINTENANCE WORKNEW TRANSFORMATIONAL WORK41%59%VSIncreasing pressure todemonstrate resultsBudgets may be on the upward trend,but there is no room for complacency. Anincrease in investment will intensify thefocus on QA productivity, a function that isviewed primarily as a necessary cost base.Even as testing spend rises, companiesstill need to maintain downward pressureon costs, as the move to consolidationin TCOEs and on-demand testing via thecloud both illustrate. Any investmentneeds to achieve a return, which meansthat continued or intensified scrutiny bythe business units who pay for QA servicesis likely, in order to ensure that newtesting resources are optimally deployedand managed.One way to ensure budgets are beingspent productively is to effectivelymeasure the added value of testingand clearly communicate this to thebusiness stakeholders. The good newsis that only 15% of organizations do nothave a process to consistently collect andpresent their quality KPIs to the business.The vast majority of organizations arecollecting this data using QA managementor business intelligence products inthe market, or using common officeapplications such as Microsoft Excel. Infact, 56% indicate that they automaticallygather and share metrics using Excel, and21% collect data manually and use Excelas a presentation layer. However, whileExcel is universally available and a low-costsolution, it does not support the richness infunctionality that a professional reportingand QA management product can provide.In times of greater business scrutiny, QAorganizations’ best interests will be servedby providing constantly updated businessKPIs that track return on investment.The magic number forquality budgetsTesting budgets tend to average out ataround 18% of total spend on organizations’software development lifecycle, and thisfigure seems to extend across all globalregions and industry sectors surveyed.On balance, this would seem to be thebenchmark proportion of the softwaredevelopment budget that organizationsneed to commit to testing.Spending much below this level runs therisk of developing applications that areinadequate from a quality perspective,or applications that are outdated beforetheir release when the QA process takestoo long to bring a product to marketahead of the competition. Spend toofar above this level, and organizationsmay need to question the efficiency oftheir QA operations, or ask whether theadditional spend is justified because thesuperior application quality contributesto a commercial competitive edge,or because it needs to meet highlystringent governmental regulationsor safety requirements, e.g. publictransportation systems.Increased focus onexternal activitiesThe areas in which testing investmentis being focused reflect the competitivepressures organizations feel in anunpredictable climate. A robust two-fifths(41%) of testing spend is being dedicatedto transformational work – testing newapplications, as opposed to maintenanceof existing software. This is a positiveindication that, even amid continued globaleconomic turbulence, firms are continuingto invest in developing new productsand services.Similarly, the greater proportion oftesting budgets (59%) are being spent oncustomer-facing applications rather thanon internal organizational applications(41%), a clear focus on external applicationsthat have a direct impact on sales andrevenue growth. Firms are also boostingtheir investment in TCOEs in orderto compress the time-to-market fornew products. Companies, no matter whatsector or region, are investing in innovationin order to maintain and capture marketshare and ultimately drive revenue.The State of Quality 2012Quality Budgets14 WORLD QUALITY REPORT 2012-13
    • “The problem is increasingpressure to get thingsdone yesterday. We havethis conflict betweentesting adequately toour satisfaction and thetesting time expected bythose on the [business] side.Investment has increasedbut then expectation inspeed of delivery hasalso increased. It’s noteasy, as the organizationneeds to be able to jumpinto a much faster testingmode while maintainingquality standards.”A manufacturingbusiness, CzechRepublicThe State of Quality 2012Quality Budgets WORLD QUALITY REPORT 2012-13 15
    • MORE INVESTMENT INEXISTING TESTING TOOLSBase: 1553 RespondentsAREAS OF GREATEST INCREASE IN SPEND: NOW AND IN 2015 FIGURE 3INTERNAL PROFESSIONALTESTING RESOURCESEXTERNAL OUTSOURCEDRESOURCESTEST ENVIRONMENTS/INFRASTRUCTURENEW TESTING TOOLSQA BUDGET IS NOTINCREASINGNOW201555%44%NOW201544%46%NOW201546%43%NOW201539%39%NOW201519%22%NOW201512%13%Investment priorities:current and futureQA investment priorities are largely shapedby the new market landscape whereorganizations must achieve more with fewerresources, find ways to reduce cost, anddeal with increased competition from bothbricks-and-mortar and online channels.“My investment going forward will [focus on] bringing my internal teams upto standard, increasing the level of automation used, and developing greaterefficiencies throughout. Although our testing budget is not separate fromthe wider IT budget, the level of monitoring and performance measuring hasincreased – we need to justify our costs.”An automotive business, Brazil As a result, organizations are seeing animperative to upgrade existing suitesof testing tools. More than half (55%) ofrespondents describe the upgrade processas the budgetary line item receiving thegreatest increase in spend, and in ouropinion this is driven by a desire to move tonewer application lifecycle platforms andsoftware-on-demand type models.Looking ahead to 2015, however, prioritiesare set to shift from upgrading existingtools to buying external resource support.A smaller proportion (44%) of firms stillexpect existing tools to be their number oneinvestment focus three years from now,but it is secondary to purchasing externalresources to supplement their internal QAteam (46%).The State of Quality 2012Quality Budgets16 WORLD QUALITY REPORT 2012-13
    • world are radically reducing spend tocope with the fallout from the worldwideeconomic crisis. Almost half (48%) of publicorganizations’ testing budgets increasedfrom the previous year, the highestproportion of all sectors studied, with 56%also predicting an increase between nowand 2015.Our observation is that public bodies areinvesting in consolidating their testingoperations to drive cost efficiencies inthe face of austerity. With a lingeringlegacy of failed major IT projects, QA hasto be stepped up to derive more fromexisting capabilities and improve testing.Investment in existing testing tools is seenas one of the solutions to remedy the failedIT projects issue. Over half (58%) of PublicSector entities identify this as the greatestincrease in spend.Manufacturing firms are the mostoptimistic among the sectors, anticipatingan increase in testing budgets three yearsfrom now: 58% expect spend to growbetween now and 2015, against a mean of53%. Although our experience has shownthat manufacturing firms tend to spend lesson IT overall compared to other sectors, itis an increase nonetheless. Some 30% ofmanufacturers predict that new testingtools will be the greatest spend in 2015,compared to 22% on average. CPRD firmsare similarly optimistic in terms of budgetoutlook in 2015, with 57% anticipating anincrease, and the sector places the highestpriority on innovation, committing 45% ofspend to new transformational work (all-sample average 41%).Energy and Utilities (E&U) is an intriguingexception in terms of current investmentpriorities, spending notably more on bothexisting (65%) and new (23%) tools, andtransportation firms are the most keenlyfocused on outsourcing. More than half(52%) see this as the fastest rising area ofspend, compared to a mean of 43%.In summary, against a gloomy backdrop ofdepressed economic activity and relativefinancial instability affecting most regionssurveyed, QA budgets have held theirplace in the IT overall spend. We believethis is a somewhat late recognition of theinherent value of testing and QA in ensuringrobust business-critical applications asan enabler of commercial growth andpublic reputation.New product development will generate theneed for new testing skills, tools, and devices,prompting organizations to seek supportfrom external partners. Investment in newtesting tools will also rise, as new productsand services are introduced to the market.Some 22% see this as the top spend priorityfor 2015, up from 19% at present.Budgets relatively consistentacross regionsThe greatest degree of consensus betweenglobal regions in this year’s report is aroundQA budgets. Budget levels hover consistentlyaround the all-sample mean of 18% of totalsoftware development spend across all butone of the geographic areas surveyed. Theexception is South America, where testingtakes up some 25% of the budget.We believe that this is an indication oftargeted South American investment. AsBrazil, in particular, continues to boomeconomically and expectations aroundapplication needs are raised, so testinginvestment is being enhanced.The pattern is similarly consistent for budgetincreases, with the exception of Asia. Asianfirms appear to benefit from the mostaggressive budget expansion compared toother regions, as was the case in 2011. Halfof Asian respondents saw an increase inbudget from last year, compared to the globalaverage of 42%. More still (60%) expect anincrease over the next three years, against anaverage of 53%.The balance of spend between new andmaintenance work also presents a fairlyuniform picture. Here again, South Americais the outlier, where firms are spendingalmost half (48%) of testing budget ontransformational work (average 41%).The same goes for the split between spend oncustomer- and employee-facing applications.South America again goes against the globaltrend, being the least externally focusedregion, spending only 52% on customerapplications (compared to 59% on average).Investment prioritiesvary by regionWhen it comes to investment priorities,there are greater variations betweenthe regions. Eastern Europe currently hasinvested far more in new testing tools (33%)when compared to other regions and to theglobal average (19%). The Nordics and NorthAmerica are at the other end of the scale, withjust 15% of firms giving top priority to buyingnew tools, although both regions indicate afocus on upgrading existing tool sets, whichmight suggest that they are looking to betterleverage their shelfware and make theirexisting investments go further. SouthernEuropean firms are more heavily focusedon upgrading existing tools (63% comparedto 55% on average), perhaps suggestingthat they are maturing from a more manual-intensive process to incorporate automation.Looking ahead, Southern Europe will beincreasing its investment in outsourcedtesting, with a majority (57%) of firms inthis region viewing this as their main QAinvestment priority for 2015, compared to46% overall. Just over half of respondentsin the UK and Ireland, North America,and Eastern Europe also expect externalsuppliers to receive the lion’s share of theirtesting spend three years from now, furtherevidence that the global move towardsoutsourcing will continue to grow.Public Sector leads the wayfor increased QA budgetThe pattern of uniformity around testingbudgets seen between the regions isrepeated across the industries surveyed,with QA budgets in most sectors fairlyevenly balanced around the average 18%of total software development spend. Thegreatest disparities are in the ConsumerProducts, Retail, and Distribution (CPRD)space (22%), reflecting a higher focus on thequality of end-user consumer applications,and Financial Services (FS) at 20%, atraditionally heavy investor in QA.The Public Sector, surprisingly, isexperiencing unexpected budget upliftsat a time when governments around the48% of publicorganizations’testing budgetsincreased lastyear - the highestproportion of allsectors studiedThe State of Quality 2012Quality Budgets WORLD QUALITY REPORT 2012-13 17
    • Testing Centersof Excellence: Agrowing priorityConsolidation and standardization are continuingthemes for all organizations around the world andthe testing function within these companies is notimmune to these trends. Aided by the emergenceof new delivery models, technologies, and bestpractices, organizations are looking to streamlinethe way in which their QA function is structuredand run. TCOEs are undoubtedly an efficient approachto industrializing testing activity. They act as avirtual command center that uses a standardizedtesting methodology, best practices, automation,metrics, and tools, while managing a flexible poolof available resources, both internal and external,to ensure high levels of quality across applicationsbefore deployment and during production. TCOEsalso provide visibility into the level of quality for anysoftware system or project, helping IT managementmake deployment decisions based on business risk.Growing from a low baseWe found that only 6% of companies around theworld have developed QA into a fully functional TCOE.This number is a significant increase, in terms ofpercentage, from last year’s findings (4%), but thesingle digit number still reflects a long road ahead forcompanies to universally adopt the TCOE model.One of the key attributes of a TCOE is a centralizedpool of resources that can be leveraged acrossmultiple projects in an organization. We found thatclose to 70% of organizations still rely on elementsof decentralized testing, while more than a quarter(27%) of them describe their testing function as“highly decentralized”.Testing automation, another aspect of a TCOE, isalso scarce, with just 32% of test cases supportedby automation. These figures indicate that companiescontinue to perform QA as siloed projects withminimal opportunity for standardization and sharingof best practices across the enterprise. Of note, thegreater proportion of automated testing is conductedoffshore, a testament to the growing confidence inmoving this type of testing beyond national borders,with only 40% being performed onshore.Flexible resource pools that can ramp up and downwith the changing needs of the company is anotherattribute of a TCOE. Our research shows that themajority of organizations (51%) still run testing asan in-house function, and only 13% have moved to aservice fully managed by an external provider. Withthe exception of North America, QA organizationshave yet to truly embrace the cost benefitsof offshoring. The majority of testing resources (52%)are still based on home territory.Reasons for this global discrepancy vary from regionto region. In South America, government-imposedcustoms taxes and new rules governing the useof offshore services discourage the use of thisflexible pool of resources. In Europe, particularly incontinental Europe, the lack of available offshoreresources skilled in the native language hinders theadoption of an offshore model, although we are nowseeing the UK and Nordics embrace this model ata faster pace. In other areas of the world, the laborarbitrage does not yield adequate amounts of returnon investment to warrant an offshoring strategy.The State of Quality 2012Testing Centers of Excellence18 WORLD QUALITY REPORT 2012-13
    • “[A TCOE] is something I’mbuilding here; it representsa significant change in ourapproach but one that Ipersonally believe tobe vital to our businesssince I’ve come from anorganization with anoperating testing center.We know that it can reducecosts and overall time oftesting practices, and giventhe competitive nature ofthe market we operate in,it’s critical to get speedand costs down whilstgetting quality up. Webelieve a TCOE can do this.”A manufacturingbusiness, UnitedKingdomThe State of Quality 2012Testing Centers of Excellence WORLD QUALITY REPORT 2012-13 19
    • Base: 924 RespondentsPLANS FOR A TESTING CENTER OF EXCELLENCE FIGURE 4PLAN TO DEVELOP INTERNALLY MANAGEDTCOE WITHIN THE NEXT 2 YEARSPLAN TO USE A THIRD-PARTYCOMPANY WITH A TCOE CAPABILITYIN-HOUSE TCOE STARTED WITHIN LAST 2YEARS, BUT NOT YET FULLY OPERATIONALIN-HOUSE TCOE IS FULLYOPERATIONALNO PLANS IN PLACE34%24%21%15%6%The percentage of organizations reportingthat they have no plans to set up a TCOEhas fallen significantly from last year (45%to 34%), and we predict that this will fallfurther over the coming few years as theTCOE model gains recognition for deliveringperformance improvement. This trend begsthe question – what is driving the surge ininterest in industrializing testing activity?Less cost, more speedFirms are centralizing their QAinfrastructure in order to reduce time-to-market, bring down the cost of testing, andmanage testing resources more effectively.Compressing time-to-market is the primaryreason for TCOE implementation, citedby 36% of respondents, and especiallythose in what might be called the moreAn appetite for changeHowever, we are witnessing a growinginterest in centralizing QA operations.Survey respondents indicated a significanttrend towards new investment in TCOEs,guided by the imperatives to reduce time-to-market and costs, increase quality,and gain better control of the wholelifecycle process.Whilst only 6% of organizations currentlyhave a fully fledged operational center,this presents a 50% increase from 2011(4%) – a small but significant step. On apositive note, almost two-thirds (60%)of firms are currently in the process ofbuilding or planning a TCOE, while a third ofthese companies (21%) plan to leverage anoutsourced partner.mature economies. Add to this the 17%aiming to enhance the agility of their QAoperations, and it is clear that speed is akey driver of growing TCOE adoption.In an unpredictable global economy, andwith market and business expectationsfocused on producing better productsfaster, time-to-market has becomeessential in gaining or maintainingcompetitive edge and maximizing profits.As a result, development cycles arebeing squeezed, and testing teams haveto do more with less. A TCOE offers theopportunity to meet these challengeshead on.Reducing the cost of testing activity isnext on the priority list, identified by justover a quarter (27%) of organizations,The State of Quality 2012Testing Centers of Excellence20 WORLD QUALITY REPORT 2012-13
    • Base: 1221 RespondentsEXPECTED BENEFITS FROM IMPLEMENTING A TESTINGCENTER OF EXCELLENCE FIGURE 5REDUCED COSTBETTER RESOURCE MANAGEMENT(HW/SW AND PEOPLE)BETTER QUALITYSTANDARDIZED TRACKING OFPROJECTS THROUGH METRICSREDUCED TIME-TO-MARKETIMPROVED QA AGILITY TOSUPPORT BUSINESS OBJECTIVES36% 27% 25%22% 19% 17%“For us, the move towards a TCOE is part of thewider union between the business and IT side ofthe organization. We now have a much strongerlink between the two, and while that meanswe feel more able to understand and answerdemands, it does increase pressure on testingto be more efficient as a whole – decentralizedtesting teams just couldn’t provide thatefficiency and it meant that walls existedbetween the teams, whereas shared experienceand best practice are far easier within aTCOE framework.”A financial services business, Finlandclosely followed by the need to manage hardware,software, and human resources more efficiently andproductively (25%).It is interesting to observe that cost – at least foreconomies such as North America – is not theoverriding priority. Most organizations have beenfocused on cost cutting for many years, and costshave been pared down to a minimum, with strategiessuch as outsourcing delivering benefits. It shouldnot be a surprise, therefore, that companies in theseregions are now seeking advantages beyond thepurely financial.These objectives for moving to a TCOE model promptus to ask: why has efficiency in meeting time-to-market deadlines risen to the top of firms’ agendas?What is different in today’s market environmentthat has rendered the decentralized approach lessdesirable for the QA function in some instances?The State of Quality 2012Testing Centers of Excellence WORLD QUALITY REPORT 2012-13 21
    • Based on the survey responses, togetherwith the experiences of our clients, webelieve there are three key factors at play:the market environment, rising technologyexpectations, and diversityof delivery models.1. The market environmentContinuing market unpredictability meansthat organizations can ill afford to takethe emphasis away from cost reduction.Many remain firmly in cost-saving modewhere QA is concerned: some still aim toshave as much as 30-40% off their currenttesting budgets.This report also notes that QA budgetsare increasing but that more is beingdemanded of teams and more complexdevelopments are requested. So whilebudgets are increasing, there is still a needto control costs.In parallel, there is considerable pressureto develop applications and bring them tomarket as rapidly as possible. At the sametime, firms also need to focus on innovationto prepare for an eventual upturn in theglobal economy.2. Rising technology expectationsIntensifying this pressure on applicationdevelopment is an increasinglycompetitive landscape, fuelled bygrowing customer demands. Customers– both internal and external – expect toaccess services and purchase productswhenever they want, on whatever devicethey choose. To meet this demand for 24/7availability, there is mounting pressure todeliver applications as quickly, efficiently,and cheaply as possible – and this isparticularly evident in the mobile spacewhere expectations are well advanced andare most likely going to increase over time.3. Diversity of delivery modelsIn response, QA departments arecapitalizing on the efficiency gainsoffered by cloud technology, increasinglyoutsourcing testing to cloud environments,and buying resources on demand, toreduce cost and time to bring applicationsto market.A further step to speed up the developmentprocess is the wider adoption ofAgile methodology. This is currently beingused by over a third of testers (37%),analysts (34%), and developers (29%). Andencouragingly, three-quarters (75%) ofAgile testing is already being performedoffshore, a situation that seemed unlikely afew years ago.Offshore adoption: a keydeterminant for TCOEimplementationCentralizing and professionalizing the QAfunction undoubtedly presents a valuableopportunity to save costs in a tougheconomic climate. However, motives drivingconsolidation are, in part, a function of howfar different regions have already leveragedthe cost benefits of offshoring.As a rule, markets with greater levels ofoffshoring – primarily North America andthe UK – are less focused on cost reduction.Offshoring has been considered, executed,and reviewed, and there is little remainingscope to further squeeze labor arbitrage.Therefore, other models are being exploited,such as TCOE, for achieving better quality,being quicker to market, and moreeffectively using resources. Cost is not aneasy win for the more mature markets –they have to look at other key benefits todrive more efficient and focused QA.Conversely, other regions with lower levelsof offshoring remain more keenly focusedon cost reduction. This is understandable,given that many are in the early stages ofoffshoring and thus have more resourcecosts to reduce. This difference is reflectedin our survey results.North America and the UK are the only tworegions where the majority of testers arelocated either offshore or nearshore (55%and 50% respectively). And they are the twoleast cost-focused markets: only 15% and16% of companies cite cost reduction as areason for establishing a TCOE.In South America, by contrast, offshoringhas been far less exploited, and (it shouldbe noted) in Brazil is actively discouragedby government policy. Only 38% of testersin the region are offshored, compared to48% on average. Here, cost reduction is afar greater priority for TCOE operations thanelsewhere: more than half of respondents(54%) identify cost reduction as the primarydriver (compared to just 27% on average).So as markets mature, cost becomes lessof a driver, and the focus moves towardsreducing time-to-market.SectorsPublic organizations seem to be leadingthe way in TCOE adoption, because, alongwith Telecoms, Media, and Entertainment(TME), they can claim the lowest proportionof organizations with no plans to implementa TCOE (30%). They have also experiencedthe steepest drop in this proportion – downmarkedly from 59% last year.This is likely being driven by the costreduction imperative that is behind manyorganizations’ TCOE plans. With spendingausterity being implemented by manygovernments, public bodies are being askedto do more with less, while maintainingservice levels and quality. As an efficiencydriver, an onshore TCOE is a compellingmodel, particularly as offshoring is apolitical anathema to many countries.For other sectors, the imperativeis responding faster to consumerexpectations, changing market demands,and time-to-market. Two-thirds (66%) ofCPRD respondents now operate a testingcenter, or plan to – significantly up from41% last year. Streamlining of supply chainsand application testing would appear tobe complementary. And this upturn is alsotrue of the FS industry, where adoption is upfrom 58% to 69% .Finally the High Tech sector, which includesaerospace and defense companies,appears to be closing down internal centersin favor of an outsourced model, as time-to-market is a major issue in this sector. Thesector had a remarkably high proportionof in-house facilities up and running lastyear (17% compared to an overall averageof just 4%), while only 8% planned towork with a third party. This year, thetables have turned. Just 6% now havetheir own TCOE, while 17% plan to use anoutsourced partner.This year’s data indicates that TCOEs aregaining traction, with growing levels ofadoption across the board. As a flexiblemodel, it offers significant advantagesand opportunities to fully leverageexisting resources.The State of Quality 2012Testing Centers of Excellence22 WORLD QUALITY REPORT 2012-13
    • Base: 1553 RespondentsPercentage of offshoring compared to costreduction as A reason for TCOE FIGURE 6NORTH AMERICA UK AND IRELAND NORDICS55% 50% 50%15% 16% 44%WESTERN EUROPE SOUTHERN EUROPE EASTERN EUROPE49% 49% 43%29% 25% 32%ASIA SOUTH AMERICA AUSTRALIA AND NEW ZEALAND36% 38% 42%29% 54% 20%GLOBAL AVERAGEPercentage of testersoffshore or nearshorePercentage of respondents citing costreduction as reason for TCOE48% 27%Organizations are turning to TCOEs toconsolidate their QA operations in responseto market pressures, technology demands,and emerging models of testing deliveryThe State of Quality 2012Testing Centers of Excellence WORLD QUALITY REPORT 2012-13 23
    • Mobile Testing:Behind the curveMobile has certainly changed the gamefor enterprise IT. Organizations aroundthe world now need to deliver continuousaccess, anytime and anywhere, tothousands of employees and millionsof customers, over a bewildering arrayof devices.Due to cost pressures, firms are no longerwilling to pay for, and thus control, theiremployees’ mobile-access devices, leadingto a bring-your-own-device culture. Thesituation is further exacerbated becauseorganizations are unwilling or unableto support the proliferation of personaldevices such as smartphones and tabletsbeing used at work by employees. However,as the prevalence of non-companyapproved devices becomes mainstreamand employees demand access to theflexibility and productivity that mobileoffers, IT departments are now beingforced to embrace this as the new realityand address the unique challenges of thisparadigm shift.Mobile has also turned IT securityinside out. A firm’s entire infrastructurecan no longer be tucked safely behind avirtual private network and firewall. What’smore, employees are prepared to downloadand try out apps on mobile devices with alatitude they would not consider acceptableon a desktop or laptop.In addition, the need for mobility hasreset expectations of what constitutesapplication quality. With traditionalsoftware, users expect flawlessfunctionality first and foremost, but mobileusers are seeking convenience. They expectrobust performance and usability on themove, and are more inclined to tolerate theoccasional glitch along the way, as long asthe application performs well and is user-friendly. QA teams may need to rethinktheir testing strategies and prioritiesdesktop application. Performance is farahead of functionality (cited by 48%)on firms’ priority list for mobile testing.Security, a perennial concern where mobileis concerned, is a priority for less thana fifth (18%) of organizations. This doesnot mean that companies are no longerconcerned with security, but rather thatthe perspective on security has changedwith the bring-your-own-device cultureand method of disseminating apps. Unlikewith traditional desktops, companiescannot control what apps are installed onpersonal devices, so the concept of securityis shifting from a denial of access to theprotection of sensitive data and ability towipe clean certain apps or data.Even lower on the radar is the certificationof applications, cited by just 14%, despitehigh levels of inconsistency in the quality ofmobile apps on different operating systems.Specialist partners areplugging the gapThe number one criterion when selectingexternal partners to assist with mobiletesting is the capacity to test across severalnetworks, identified by almost two-thirds(62%) of organizations. This reflects theneed to ensure coverage, as businessesseek partners who can deliver testing in avariety of environments.Interestingly, cost reduction is less of aconsideration (cited by only 25%), despitebeing high on the list of factors driving bothcloud and TCOE adoption. This may be dueto a number of factors: mobile applicationsare relatively cheap to produce, the focusis on managing risk rather than cost,and firms’ need to address their lack ofreadiness and expertise where mobile QAis concerned.as the mobile era adds a further level ofcomplexity to users’ needs, demands,and expectations.Underprepared and ill equippedOur research suggests that organizationsmay not be giving mobile the priorityit deserves. Only 31% of respondentscurrently test mobile applications, andthose surveyed readily admit to being illequipped for mobile testing. This suggeststhat QA has fallen behind the mobile curve.A troubling two-thirds of organizations(65%) do not have the right tools to testmobile applications, and equally concerningis the 52% who do not have access to therequired devices. It would appear thatorganizations either do not have a goodgrasp of the infrastructure challengesposed by the mobile era or are unable toaddress these new demands. Meanwhile,a third of organizations lack the testingmethodologies and processes (34%) andspecialist expertise (29%) necessary toeffectively certify mobile applications.Performance is the keyto successOrganizations that are conducting mobiletesting seem to be working towards adifferent set of quality standards ascompared to traditional testing. Theprimary focus is on the application’sperformance rather than its functionality.Efficiency of performance was identified asa focus for mobile testing activity by almosttwo-thirds (64%) of firms.The usual priorities – such as functionalityand security – are being pushed lowerdown the scale as companies place moreemphasis on the demands of end-userson mobile platforms, which are differentfrom the demands of a traditionalThe State of Quality 2012Mobile Testing24 WORLD QUALITY REPORT 2012-13
    • EFFICIENCY/PERFORMANCE(INCLUDING NETWORK-RELATED)Base: 485 RespondentsPRIORITIES WHEN TESTING MOBILE APPLICATIONS FIGURE 8FUNCTIONALITYPORTABILITYUSER INTERFACE/EASE OFUSECOMPATIBILITY/REGRESSION TESTINGSECURITY (PROTECTION OF SENSITIVEDATA OVER THE PHONE OR AIR)CERTIFICATION OF APPLICATION64%48%46%36%31%18%14%DO NOT HAVE THE RIGHTTOOLS TO TESTBase: 485 RespondentsCHALLENGES TO MOBILE TESTING FIGURE 7DO NOT HAVE THE DEVICESREADILY AVAILABLEDO NOT HAVE THE RIGHTTESTING PROCESS/METHODNO MOBILE TESTINGEXPERTS AVAILABLEDO NOT HAVE IN-HOUSETESTING ENVIRONMENTNOT ENOUGH TIME TO TEST65%52%34%29%19% 18%The State of Quality 2012Mobile Testing WORLD QUALITY REPORT 2012-13 25
    • Percentage of respondents testing mobileapplications, by region FIGURE 9Base: 1553 Respondents24%38%22%38%37%21%26%30%39%EASTERN EUROPESOUTHERN EUROPE SOUTH AMERICANORDICSWESTERN EUROPEAUSTRALIA AND NEW ZEALANDNORTH AMERICA ASIAUK AND IRELANDRegional variationsWe might expect the strongest emphasison QA in the mobile arena to be seen inAsia in particular, as smartphone andtablet penetration are accelerating at ablistering pace in China. But the distinctionbetween East and West economies is notclear-cut. Indeed, in Asia, the proportionof organizations conducting mobile testingsits at 30%, close to the all-sample averageof 31%.Regions with traditionally high levels ofmobile adoption, with the exception of Asia,have – unsurprisingly – the highest levelsof mobile testing. Over one-third of firmsin Australia and New Zealand, the Nordics,and Southern and Western Europe currentlytest mobile applications.Regions displaying lower levels of mobiletesting include Eastern Europe and the UKand Ireland, where only a fifth of businessescarry out mobile testing, and just a quarterdo so in North and South America.The State of Quality 2012Mobile Testing26 WORLD QUALITY REPORT 2012-13
    • 18%65%52%29%26%79%58%34%Greatest challenge whentesting mobile applicationsand devicesFIGURE 10Base: 485 RespondentsDO NOT HAVE THERIGHT TOOLS TO TESTDO NOT HAVE THE DEVICESREADILY AVAILABLENO MOBILE TESTINGEXPERTS AVAILABLENOT ENOUGHTIME TO TESTALLRESPONDENTS ASIAAsian challengesThe explosion of the mobile market in Asia isreflected in the QA challenges and priorities facedby organizations in the area. A greater percentageof Asian firms confirm they faced the commonmobile challenges as compared to all the companieson average.Asian firms also place much more emphasis onportability and compatibility across devices whenconducting mobile testing. Portability is identifiedas a priority by 58% of businesses in this region,compared to less than half (46%) on average.Compatibility is cited by almost as many (55%),compared to less than a third (31%) overall.This focus might be driven by the far wider selectionof devices available to users in the region. In China,for example, mobile customers can choose fromjust about any handset available worldwide. In theUS and Europe, networks offer only the handsetsfor which they have contractual arrangementswith manufacturers. As a result, the choice availableto customers is far more limited.Performance versus functionalityElsewhere, the emphasis on application performanceis particularly acute in the UK and Ireland (75%),against an average of 64%. This could be due toits being a more mature market where end-usersdemand more from their mobile apps. Consider thelandscape a decade ago, when having a browser on amobile device was touted as a new feature, whereastoday these same users want apps that are tailoredfor specific needs. The opposite is the case in SouthAmerica and Eastern Europe, where only around halfof organizations focus on performance.Instead, South American firms are focused primarilyon functionality, and more than four-fifths identifythis as a priority, compared to just under half overall.Mobile security is also a heightened concern inthe region, cited by 30% against an 18% average.Eastern European businesses are even more worriedabout security (36%), and are equally concernedwith application certification (compared to just 14%on average).The State of Quality 2012Mobile Testing WORLD QUALITY REPORT 2012-13 27
    • Sector perspectivesA look at mobile testing by industry throws up somesurprising results. Firstly, it is Manufacturing firms,rather than those from more consumer-facingindustries, that are most likely to conduct mobiletesting (37% compared to 31% on average).We might expect the TME sector to be most at easewith mobile technology. However, their QA demandsare uniquely complex, comprising the need to testnetworks, equipment, and billing systems, amongothers that are applicable only to TME firms. As aresult, just 30% of firms in the technology-intensiveTME space currently test mobile applications,equal to the all-sample mean. Some 62% lack thenecessary tools to test mobile applications. This isonly slightly better than average (65%), and almostone-third (32%) lack in-house expertise, comparedto just 29% overall. Perhaps this is due to the TMEproviders focusing their QA efforts on the mobileinfrastructure and leaving the testing of mobiledevices to the handset manufacturers.Public Sector organizations (27%) are the slowestto adopt mobile testing. This may not be quite sosurprising in a sector not traditionally viewed asa leader in technological innovation. But it doespresent a major issue in an era when citizensincreasingly expect to interact while on the movewith public service providers, just as they do withcommercial brands.“It’s an area of our business that typifies how things are changing,in my opinion. The guys in marketing say ‘we need a mobile app’; salesor product execs agree and then send it across to development andtesting to push it through in half the time we might actually need.It’s not going away either. More and more of our business is doneonline [in general] so it’s something we need to get hold of andunderstand quickly.”A retail business, United KingdomPublic bodies are also the least focused onperformance, although this is still their greatest areaof priority, cited by more than half of organizations(53% compared to 64% average). The sector placesa higher emphasis than average on security and userexperience, in line with their public responsibilities.Despite its reputation for innovation, the FSindustry faces some of the most acute mobiletesting challenges. Almost three-quarters (72%)of FS firms lack the right tools, compared to 65%on average, and 38% lack mobile testing expertise,against 29% overall. Financial firms are the mostacutely concerned about application performancewith 72% citing this as a priority.User-friendliness is an understandable priority forthe CPRD space. Ease of use and the user interfaceare a focus for 50% of CPRD firms (compared to 36%on average).Mobile is certainly having a significant impact on theQA function and it seems that QA has been slow toreact to the expectations from within the business.We would expect swift reassessment of testingpriorities and increased investment over the comingyear to handle the rush of mobile applications eitherin-house or with specialist providers.The State of Quality 2012Mobile Testing28 WORLD QUALITY REPORT 2012-13
    • Base: 485 Respondentsgreatest challenge when testing mobileapplications or devices, by sector FIGURE 12CONSUMER PRODUCTS,RETAIL, AND DISTRIBUTION TRANSPORTATION ENERGY AND UTILITIES FINANCIAL SERVICESMANUFACTURING HIGH TECH HEALTH AND LIFE SCIENCESTELECOMS, MEDIA, ANDENTERTAINMENTPUBLIC SECTOR63% 50% 27% 23% 27% 20%63% 43% 40% 20% 26% 31% 62% 35% 50% 12% 23% 12% 66% 64% 22% 26% 22% 17% 62% 62% 30% 32% 12% 16%64% 51% 40% 30% 13% 19%59% 56% 41% 22% 7% 26% 68% 42% 32% 29% 29% 23% 72% 50% 36% 38% 21% 14%DO NOT HAVE THE DEVICESREADILY AVAILABLEDO NOT HAVE THE RIGHTTESTING PROCESS/METHOD Not enough time to testNo mobile testing expertsavailableDo not have in-house testingenvironmentDO NOT HAVE THE RIGHTTOOLS TO TESTBase: 1553 RespondentsPercentage of respondents who test mobileapplications or devices, by sector FIGURE 1127%PUBLIC SECTOR37%MANUFACTURING35%ENERGY ANDUTILITIES35%HEALTH ANDLIFE SCIENCES34%CONSUMERPRODUCTS, RETAIL,AND DISTRIBUTION31%TRANSPORTATION31%FINANCIAL SERVICES30%HIGH TECH30%TELECOMS, MEDIA,AND ENTERTAINMENTThe State of Quality 2012Mobile Testing WORLD QUALITY REPORT 2012-13 29
    • Cloud: Gathering pace“The cloud is something that has become a significantpriority for our organizational direction in theInformation Technology Department. With regardsto testing, it’s opening up a way of turning on andoff a stream of new skills and abilities in a way thatis both fast and cost-effective when we need them.We can reach out and get exactly what we needwithout delay.”A financial services business, The NetherlandsCloud computing continues to generate asignificant amount of attention as almostevery market forecast indicates exponentialgrowth in this area. Within this climate,cloud adoption in QA is also gatheringmomentum, because the business case isresoundingly clear. As more applicationsare being hosted in the cloud, the necessityfor testing in cloud environments isgrowing apace. And organizations areseizing the benefits that the cloud offersby buying services on demand and gettingaccess to physical resources more quicklyand cheaply to cut QA spend.Cloud coverage on the riseThe extent to which businesses and theirQA function are now embracing the cloudmarks a step up from the evolutionaryprogress revealed in last year’s report.Our research shows that almost a quarter(22%) of software applications are nowhosted in the cloud. This is expected to riseeven further to a third (32%) by 2015, anindication of the acceptance of cloud aspart of the mainstream of IT infrastructure.Testing in the cloud is also expanding at acorrespondingly healthy rate. Some 28% oftesting now occurs in a cloud environment,a proportion forecast to rise to 39% by 2015.Only two years ago, some 31% of companiesdeclared they would not test via the cloudover the next three years, and this hasnow dwindled to just 4%, a clear indicationthat testing in the cloud is becoming thenew norm.28% of testing currentlyoccurs in a cloudenvironment, andis set to rise to 39%by 2015The State of Quality 2012Cloud30 WORLD QUALITY REPORT 2012-13
    • PERCENTAGE OF APPLICATIONS HOSTED IN THE CLOUD,NOW AND 2015 FIGURE 13Base: 1,553 respondentsPERCENTAGE OF TESTING CONDUCTED IN THE CLOUD,NOW AND 2015 FIGURE 14Base: 1,553 respondentsNOW 201522% 32%NOW 201528% 39%The State of Quality 2012Cloud WORLD QUALITY REPORT 2012-13 31
    • “We experience peaks and troughs in our testing program, so TaaSprovides a certain degree of flexibility. While some applications maybe run of the mill, some may have more ad hoc requirements thatpush us to meet unique or temporary needs. TaaS allows us to dothis without the burden of great cost or adding to project time.”A retail business, DenmarkRising demand for on-demand servicesAs QA professionals become more comfortable with the cloud as atesting platform, organizations are increasingly capitalizing on theon-demand benefits it provides.Companies that are accustomed to using cloud-basedinfrastructures are now demanding similarly flexible offerings fortheir software licenses in the form of Software as a Service (SaaS),where costs incurred are matched against actual usage.Organizations areincreasingly capitalizingon the on-demandbenefits of the cloudAlmost a third (31%) of firms’ testing software is currently providedon a SaaS basis – compared to only a quarter (25%) last year. Thisis expected to reach close to half (43%) by 2015. Furthermore,some 30% of firms had no SaaS arrangements in place for testingsoftware in 2011; this has fallen to just 8% this year.Throwing caution to the cloud?Conventional wisdom dictates that lingering doubts over the cloudand, in particular, its perceived lack of security would discourageorganizations from migrating their most business-critical ITinfrastructure to the cloud. Contrary to this assumption, oursurvey reveals a growing tendency to test critical applications incloud environments.In fact, organizations are actually testing more critical applications(61%) – both internal and external – than non-critical software(39%) in the cloud. However, it is worth noting that internal softwareaccounts for two-thirds of these critical applications, and externalsoftware just a third.Testing as a ServiceAlthough starting from a low base, firms are also increasinglytreating the testing activity as an on-demand utility, giving rise tothe term Testing as a Service (TaaS) – the delivery of applicationtesting services on a pay-per-use model, using cloud-based testenvironments and software tools. While only 11% of businessescurrently use TaaS arrangements, almost four-fifths (78%) plan tointroduce the concept, and almost half (47%) will do so within thenext 12 months. By 2015, only 11% will have no TaaS arrangements, apointer to how ubiquitous this delivery model will have become.An opportunity to reduce costWhen asked about the benefits of TaaS, respondents were emphaticin their enthusiasm. The most highly-anticipated benefits from TaaSarrangements were reduced cost, identified by 58% of businesses,and more effective management of testing resources (cited by 49%).By contrast, reduced time-to-market (the main driver for TCOEconsolidation) only ranks third (42%), and improved quality is evenless of a focus, mentioned by just 24%.The cost driver is clearly important when assessing TaaS services,as the focus is on how to reduce, for example, licensing costs, to payonly for what is used, as opposed to improvement in quality that canbe achieved through other means.So why would organizations view cloud-based TaaS as a greatercost-saving opportunity than consolidation, as provided by TCOEs?TaaS offers the chance to bring down costs that have alreadybeen heavily optimized and the greater portion of testing spendgoes towards resource costs. In mature markets, this element hasalready been drastically reduced by successive waves of offshoringand restructuring. And in developing markets, low labor costs area given.So, adopting a cloud utility service model presents a rareopportunity to further squeeze costs by paying only for what isneeded when it is needed, as opposed to high investment in oftendormant technology. TaaS also provides increased flexibility inservicing and pricing and the ability to quickly scale up or down asrequired for ad hoc or ongoing projects.The State of Quality 2012Cloud32 WORLD QUALITY REPORT 2012-13
    • Base: 1553 RespondentsPLANS FOR MOVING TO TAAS MODEL FIGURE 15WITHIN THE NEXT 24 MONTHSWITHIN THE NEXT 12 MONTHSALREADY HAVE ITHAVE NO PLANS FOR TAASAT THE MOMENT11%31%47%11%Base: 1553 RespondentsEXPECTED BENEFITS FROM TAAS FIGURE 16BETTER RESOURCE MANAGEMENT(HW/SW AND PEOPLE)REDUCED TIME TO MARKETSTANDARDIZED TRACKING OF PRODUCTSTHROUGH METRICSBETTER QUALITYREDUCED COSTIMPROVED QA AGILITY TO SUPPORTBUSINESS OBJECTIVES58%49%42%32%24%19%The State of Quality 2012Cloud WORLD QUALITY REPORT 2012-13 33
    • Low hurdles to entryEncouragingly, as adoption has increased,the cloud in general is now seen as a fargreater opportunity than a risk. The potentialchallenges associated with cloud-basedoperations provoked far less responseamong respondents than the benefits, sothe gains clearly outweigh the pitfalls.The benefits ofcloud-basedQA activityclearly outweighthe pitfallsThis can be seen most sharply in the halvingof levels of concern over last year’s biggestworry – security. Predictably, back in 2011,more than half (55%) of organizations sawsecurity as the most acute risk associatedwith using a cloud environment. This year,taking TaaS as an example of a generalincreased ease with using cloud-baseddelivery, security came only third on thelist of the challenges anticipated whenadopting TaaS. Even the starkest currentchallenge – loss of control over total testingspend – is troubling the minds of less than athird (32%) of companies.Yet, despite the overall enthusiasm, theseconcerns about loss of control over spendsignal a continuing concern and confusionabout the cloud generally. In our experience,there is little reason to lose control oftesting spend, as this can be preventedthrough robust governance, spend metrics,and service-level agreements.Perhaps this anxiety is a legacy of pooroutsourcing experiences, where contractshave exceeded cost estimates. Or it mayreflect persistent economic instability:unable to reliably forecast market demand,businesses are struggling to calculate anaccurate assessment of the scale of theirtesting requirements. Whatever the reason,we would encourage organizations to lookto the benefits of this new model of testingservice provision, as the concerns canbe managed.Mature markets driveon-demand provisionLast year’s report found emergingeconomies had the most aggressiveplans to migrate QA activity to the cloud.In 2012, we see this leveling off, as cloudpenetration in developing regions begins tocatch up with that of mature markets.As indicated earlier, as an example of cloudadoption, companies are increasinglylooking to TaaS as a means of costreduction and process efficiency. How isthis reflected at a regional level? Plansshow little variation between the globalregions in the immediate term. For the mostpart, the more mature markets are drivingon-demand uptake, with a high percentageof firms already adopting or planning to useTaaS services. Only 3% of North Americanfirms have no plans to use TaaS, with asimilar figure of just 5% in Western Europe.In Asia, only 7% do not anticipate TaaS intheir future.Yet this jumps to almost half (46%) inEastern Europe and nearly a third (29%)in South America with no TaaS plans.However, looking ahead, South Americais predicting a significant change, withrespondents expecting almost half (46%)of testing to be cloud based by 2015,compared to the 39% global average.For all regions, reduced cost is the primarybenefit of TaaS, and taking into accountbetter resource management, efficiencyis the secondary driver for TaaS adoption.Emerging markets are particularly focusedon cost reduction – seven in 10 Asian andSouth American businesses identify this asthe primary benefit of TaaS, compared to58% across the whole survey.Sector leaders and laggardsTurning to the sectors, testing in thecloud is being led by the IT-intensive TMEsector, where 31% of testing activity iscurrently cloud-based, followed by FS at30%, compared to the 28% sector average.Both the TME and FS sectors are drivenby technology and have traditionally beenearly adopters. Late adopters, with onlya quarter of testing in the cloud, includeE&U (25%) and Manufacturing industries(24%). These two sectors tend to view ITas an enabler rather than a driver, andtraditionally have spent less on IT.The pattern is repeated when it comesto leveraging TaaS. Only 6% of TMErespondents and 8% of FS companies haveno plans to use TaaS, compared to 11%overall. Public Sector organizations are alsoembracing on-demand testing, with only 7%having no TaaS plans in place.By contrast, some 30% of manufacturersare disregarding the on-demand testingprovision altogether, as are 16% of E&Ufirms, and, as in last year’s survey, CPRDbusinesses are slow adopters – currently17% have no plans for TaaS.Small firms reluctant to adoptOrganizations of all sizes are integratingcloud technology into their QA operations.Businesses are investing in the cloud tomodernize and optimize IT infrastructures,while small businesses look to takeadvantage of its flexibility in order tocircumvent the need for costly IT set-up.Large, medium, and small firms are allrunning approximately 28% of testingactivity in cloud environments. Likewise, allthree groups forecast this to rise to around39% by 2015.However, smaller firms are much morereluctant to conduct testing on demand.Almost a quarter (23%) have no plans touse TaaS, compared to just 6% of large andmedium-sized organizations. Again, this maybe driven by the fear that testing costs willspiral out of control or by a lack of awarenessof the true benefits that TaaS can offer.More small businesses (35%) identify thisas a challenge than their corporate (30%) ormedium-sized (31%) counterparts.Our research shows that the move towardsa cloud-based infrastructure, supported byofferings such as TaaS, continues to growacross companies of all sizes. Organizationsare primarily looking to achieve a morecost-effective and efficient means for QAthrough a flexible, agile, on-demand model.We expect that this trend will accelerateas the cloud matures and competitivepressures force non-adopters to reconsidertheir position.The State of Quality 2012Cloud34 WORLD QUALITY REPORT 2012-13
    • “We are moving moreof our testing intothe cloud, but it’salways going to beproblematic for us.National legislationrestricts the natureof data that can bestored and accessedoutside the country;therefore we arecurrently limited asto what exactly canbe done via the cloud.”A manufacturingbusiness, GermanyThe State of Quality 2012Cloud WORLD QUALITY REPORT 2012-13 35
    • Quality AssuranceResources: A questionof valueIn an increasingly digitized era, the ability tomonitor and assess the quality of softwareapplications is critical to an organization’soperations – not to mention its reputation.So confidence in the quality of QA resourcesto deliver on this priority is essential. Ourdata shows that this, unfortunately, is notalways the case.Our study finds that organizations considertheir internal QA resources to be “average”overall, which we believe is of some concerngiven the increasing importance beingplaced on testing. On balance, companiesbelieve that they can secure slightly betterquality resources via external partners thanby hiring internally.One contributing factor could be that,historically, testing has not been viewed asa well-defined professional career acrossthe world, although it is gratifying to seemuch evidence of this now changing. Indeveloped countries, QA is becoming arecognized and dedicated profession,commanding increased salariescommensurate with the level of expertiseand experience. In the less-developedmarkets, there is often a limited number oftrained resources available, which is drivingthe need to use external expertise, often ata premium to internal cost structures.A fine balancing act betweeninternal and externalThe majority of organizations (59%)characterize their internal QA teams as“average” in their QA knowledge and notnecessarily up to speed with the latesttesting tools and technologies. Externalresources fare better with slightly fewercompanies (52%) describing externaltesters in the same way. More significantly,only 4-5% of firms are confident that theirresources (internal or external) are bestin class.However, when apportioning high praise,companies express more confidencein outsourced resources than theirown employees. A third of firms (33%)score their external teams’ knowledge andabilities a 4 or 5 (with 5 being the highest);less than a quarter (24%) score internalresource as highly.This may reflect the fact that many externaltesters work for service providers whosebusiness model is to offer large-scaleprofessional teams of resources designedto elevate a company’s existing capabilities.It may also reflect an increasing desireamong certain economies to increasethe level of QA skills they can offer to theglobal community.Our study found minimal regionaldifferences in confidence levels, whichmay reflect the fact that in the moremature markets, expectations are higherand the need to outsource greater, whilein the emerging markets, expectations arelower and the available talent is startingto come through. Whichever the case, it isclear that there is room for improvementin the skill levels of QA resources, if theyare to meet the exacting demands beingplaced on them.A question of valueInternal QA resources, while rated lower interms of knowledge, are marginally moreexpensive, which is to be expected, asorganizations seek external outsourcedskills predominantly to reduce costs. Firmsare paying a global average hourly rate ofUS$54.58 for internal resources, comparedto US$52.62 for external support.Although this weighted average is notdesigned to be exact, the comparisonis revealing, as it would indicate thatorganizations are getting more valuefrom outsourced support, because theyare paying less per hour for a resourcethat is more highly regarded, althoughthe difference is small. This is potentiallythe case in economies where there is agreater number of available testers, andoutsource providers compete on costagainst in-house teams.However, these figures need to be viewedin context. Hourly rates naturally varygreatly by geography, and price needs to beconsidered alongside the value that firmsperceive they are getting for their investment.Regional perceptionsIndeed, regional comparisons paint acontrasting picture, divided broadly alongmarket maturity lines. While there isgeneral agreement across the regions thatexternal resources are more effective, thereare wide disparities between how mucheach region is paying for both external andinternal support.In developed regions, there is a pricepremium on internal testing capacity. NorthAmerican firms, for example, are payingaround 20% per hour more for internal thanexternal resources. We believe this reflectsthe increased standing of QA resourcesin North American organizations and thesalaries they can command.The State of Quality 2012Quality Assurance Resources36 WORLD QUALITY REPORT 2012-13
    • Base: 1553 RespondentsINTERNALRanking of external and internal testing resources(scale of 1 to 5, where 5 is highest) FIGURE 17EXTERNAL5%0% 19%59%17%4%29%52%14%1%THEY ARE KNOWLEDGEABLE INTESTING METHODOLOGY. THEYARE BEST IN CLASS ACROSS THETESTING INDUSTRY AND THETESTING ACTIVITIES RUN SMOOTHLYTHEY ARE AVERAGE IN THEIRKNOWLEDGE. THEY KNOW ENOUGHTO PERFORM THE JOB, BUT THEYMAY NOT BE UP TO SPEED ON THELATEST TESTING TECHNOLOGIESTHEY DO NOT EXECUTE THETESTING ACTIVITIES EFFICIENTLYAND MANY DEFECTS ESCAPE THEIRDETECTION. THEY DO NOT HAVEDEFINED TESTING PROCESSESAND ARE NOT USING TESTINGTECHNOLOGY EFFECTIVELY54321“Outsourced testing will always be an importantoption, but how we source those testers doestend to depend on project or cost, rather thanany specific predefined rules. We used to useIndia-based suppliers a lot more than we dotoday, but the cost benefits were not balancedby our internal client satisfaction feedback.”A financial services business, AustraliaThe State of Quality 2012Quality Assurance Resources WORLD QUALITY REPORT 2012-13 37
    • $47Average hourly rate (US$) for external and internal testers FIGURE 18NORTH AMERICA UK AND IRELAND NORDICSWESTERN EUROPE SOUTHERN EUROPE EASTERN EUROPEASIA SOUTH AMERICA AUSTRALIA AND NEW ZEALANDGLOBAL AVERAGE$32Base: 1553 Respondents$41$30$23$48$41$68$68$40$51$48$58$57$85$44$36$55$53INTERNALEXTERNAL$66The State of Quality 2012Quality Assurance Resources38 WORLD QUALITY REPORT 2012-13
    • As we have seen, North America is the mosthighly outsourced region, with businesseshaving transferred large proportions of theirQA to lower-cost, offshore labor markets.As a result, they are paying well below theaverage for external support, and balancingthis by paying a relative premium for highquality in-house experts.Yet interestingly the quality rating doesnot differ by much between the twosets of resources. This may indicate apragmatic approach from North Americanorganizations in assessing both sets ofresources on a relative value-for-moneybasis, balanced by availability constraints.This indicates a mature and measuredapproach in North America.In emerging economies such as SouthAmerica and Eastern Europe, however, thereverse is true. In these areas, QA is a less-developed, less-professionalized field; insome organizations, it may barely exist asa recognized function at all. As a result,dedicated external resources are morehighly prized, and so come at a premium.South American companies pay around30% more for external than internal teams;in Eastern Europe, the disparity is around25%. The conclusion is clear – emergingmarkets may not immediately be savingcosts by using external expertise, butjudging by the recognition of the skills ofexternal resources, companies are buyingan enhanced quality of output.Sector perceptionsOne industry that does recognize theprice premium and value of experiencedQA resources is clearly FS. The industryis paying a hefty premium for internalsupport: $62.18 per hour compared to the$54.58 average. The FS sector is, of course,a unique, complex, and highly regulateddomain, requiring very specific functionalityexpertise in addition to generic QA skills.Not surprisingly, this rare combination ofskills pushes up the cost of QA capabilityfor the industry.Manufacturing companies are the mostsatisfied overall with the quality of theirQA resource. Just over a third (34%)of manufacturers score their internalresource a 4 or 5, compared to the overallaverage of 23%. Almost two in five (38%)rate their external support highly (average33%). It may be that the importance thesector places on ensuring the qualityof its production output extends to itssoftware applications, resulting in a greaterappreciation of technical ability.At the other end of the scale, only around aquarter of Public Sector organizations ratetheir internal (21%) or external (28%) QAresources highly. This might be a reflectionof constrained local pay scales, orcomplexity of large-scale legacy systems,meaning that quality may be more difficultto achieve.Two sectors – where a breakdown in qualitycould have dramatic consequences – rely onexternal expertise to compensate for a lackof confidence in their in-house resource.The Health and Life Sciences businessesand E&U have the lowest confidence in theirinternal teams: only a fifth of firms in theseindustries rate them highly, yet, by contrast,external resources in both are given arelative vote of confidence. Around twice asmany organizations in these industries rateexternal providers highly.Paying the price?Experienced QA experts – both in-houseand third party – possess a rarecombination of highly technical abilities:business domain expertise, businessprocess analysis skills and development,scripting, and testing capabilities, andup-to-date QA knowledge, to name a few.In our experience, when it comes to payingthe price for that expertise, organizationsoften have unrealistic expectations. Somemay be unwilling to pay the higher cost forhighly qualified, experienced professionals;others may be restrained from doing so bybudgetary considerations.Value for money is undoubtedly subjective.Getting the balance right between in-houseand external teams, factoring in technicaland business knowledge and relativecosts, is challenging and requires constantmonitoring and investment.“I’d love to be able to professionalize our testing function – give people acareer and the pay to match and they’ll stick with you. But the reality is quitethe opposite. Testing is seen as a ‘necessary evil’ in many organizations. Thismeans organizations never give [testers] the proper attention [they] deserveand we end up buying external experts who may have the skills, but notnecessarily the intimate knowledge of our business.”A financial services business, United KingdomThe State of Quality 2012Quality Assurance Resources WORLD QUALITY REPORT 2012-13 39
    • sectoranalysis42 Consumer Products,Retail, and Distribution:Mobile and internet commercedrives sector transformation44 Energy & Utilities: Commitmentto improve quality of testingthrough increased investment46 Financial Services:Evolving QA – Riding thetransformation wave48 High Tech: Increasing focuson data-driven customerservice delivery50 Public Sector: Renewedfocus on efficiency gainslinked to budget reductions52 Telecoms, Media, andEntertainment: Maximizingassets, unleashing growth,and transforming to succeed40 WORLD QUALITY REPORT 2012-13
    • WORLD QUALITY REPORT 2012-13 41sector analysis
    • The Consumer Products, Retail, and Distribution (CPRD) sectorrepresents a large and diverse group of companies, includingconsumer products manufacturers, retailers, distributors (such aslogistics companies, postal systems, trucking), and transportation(including airlines, airports, and rail operators). Organizationsin each of these sub-sectors have distinct technology needs.However, they are all part of the broader consumer value chain andtherefore need to be extremely responsive to the changes in demandcoming from the end consumer. And that consumer is increasinglyempowered by technology.Given the growing importance of technology enablement duringthe shopping journey – from awareness and choosing, through thetransaction, and to delivery and service – the CPRD sector todayis more focused on mobile and internet commerce. As a result, weexpect to see an increase in testing needs for transformationalprojects, cloud testing, and Testing as a Service (TaaS), as well asgrowth in adoption of Testing Centers of Excellence (TCOEs).CPRD companies report investing more in QA/testing than the overallmarket and other sectors, with 22% of total IT budget going towardthe testing function. Last year, 35% of respondents in the CPRDsector said their QA budgets had increased over the previous twoyears; this year, 41% say it has increased and 57% say they expect tosee this increase again by 2015.This indicates that companies are continuing to invest in QA/testing in order to stay competitive and meet the high expectationsof consumers. In the growing, interconnected world of consumersand customers, failure is quickly and easily communicated, socompanies cannot afford to provide less than perfect products/services to customers.A high 45% of CPRD respondents say their testing activities arewith new or transformational projects, which reflects the need forinnovation in this sector. Asked in which areas of QA the greatestincrease in spending has occurred, 51% of companies say it is oninternal professional testing resources, 45% on more investment inexisting tools, and 44% on external outsourced resources.ConsumerProducts,Retail, andDistributionMobile and internetcommerce drives sectortransformationBy Bernard HeldersGlobal LeaderConsumer Products & Retail SectorCapgemini Etienne CartignySenior Test ManagerCapgemini Rashmi SinghSenior ManagerCapgemini42 WORLD QUALITY REPORT 2012-13sector analysisConsumer Products, Retail, and Distribution
    • Interestingly, when asked to look ahead to 2015 and where theythought the greatest area of investment would be, the top answer(48%) was in using external outsourced resources – perhapsreflecting the need for more specialized and specific resourcesrequired to test leading-edge services and products, such asmobile commerce.Across all sectors and respondents this year, our survey saw thecontinuing growth of the adoption of TCOEs. This is also reflected inthe CPRD sector, but it is clear that organizations in other sectors aretaking a slightly different path. While only 6% of CPRD respondentssay their in-house TCOE is already fully functional (on par withthe overall figure study-wide), a further 25% say they are alreadydeveloping or planning to develop an internal TCOE over the nexttwo years. This compares to 40% of the overall survey.The difference is apparent in the percentage of organizationsplanning to use a third-party company with a TCOE capability;35% of all surveyed CPRD firms (compared to 21% overall) say theyare taking this TCOE route. The scope of their ideal TCOE is broad,but executing performance testing (50%) and providing ProgramManagement Office services (44%) are top capabilities. The keybenefit in adopting a TCOE for CPRD sector firms is reducing time-to-market and reducing costs.The growth in the impact of the cloud continues to change the wayorganizations approach multiple areas of their operations, andthis is no different for CPRD firms. Respondents say 21% of theirapplications are currently cloud based, with an expected figure of32% by 2015. Indeed, our survey suggests that the percentage oftesting in a cloud-based test environment will rise by almost 50%by 2015 – suggesting CPRD firms are keen to exploit this growingmethodology for their testing function.Asked about TaaS offered by a third party, just 9% of CPRDrespondents say they already use it, but 45% say they will within thenext year, and a further 28% within the next two years. With 59% ofrespondents seeing reduced costs and 48% seeing better resourcemanagement as key benefits, TaaS appears to deliver distinctadvantages for an organization’s testing. The survey also sees apredicted increase in the use of Software as a Service (SaaS) for QA/testing software licenses between now and 2015, no doubt part ofthe broader move towards the cloud.Some 34% of CPRD respondents say that their organizationscurrently test mobile applications and devices, slightly higher thanthe overall figure of 31%. Given the growing importance of mobileand internet commerce, consumer/customer mobile communicationand associated services, it is hardly surprising that this is an area ofexpected growth, particularly for the CPRD sector. However, it comeswith new and additional QA/testing requirements, which manyorganizations are finding challenging.So while organizations that test mobile applications focus onefficiency, performance, and functionality, there are shortcomingsin those testing processes. Some 63% of CPRD firms say they don’thave the right tools to test, and a further 50% say they don’t havethe devices readily available to test. Indeed, when asked about theirkey criteria for selecting an outsourced mobile testing resource,60% of respondents say the capacity to test their application onseveral networks would be most important. A further 40% saythe key criterion is enabling their development team to focuson development.With the economy likely to remain challenging over the nextyear or so, CPRD companies will continue to explore new waysto stay competitive, while remaining innovative and attractive tobudget-minded customers. New developments in mobile and webcommerce will continue to offer new routes for the sector to exploit,but it must do so from a position of assurance and knowledge, asconsumers are unlikely to be loyal should they be disappointed.The rising and positive investment in QA/testing is no guarantee ofsuccess, of course. Investing in the right skills, tools, and experiencewill be essential to ensure that the challenges of remaininginnovative and competitive are balanced with those of ensuring thelevels of quality demanded by customers. WORLD QUALITY REPORT 2012-13 43sector analysisConsumer Products, Retail, and Distribution
    • This year’s World Quality Reportsurveyed firms in the energy andnatural resources sectors: oiland gas; public and private utilitycompanies; and those involved inlarge-scale chemicals production.In most parts of the world, the Energy and Utilities (E&U) sectorcontinues to undergo major structural transformation with theliberalization of consumer electricity and gas retail markets.Utilities firms are constantly looking at how new technologies suchas smart metering and smart grids can help to improve operationalefficiencies as well as differentiate their service offering in alargely price-driven, commoditized market characterized by highcustomer churn. Consistently high oil and gas prices over the pastfew years have opened up new areas of exploration taking advantageof new drilling technologies.Most E&U firms also face increased regulatory and public scrutinyregarding their ongoing operations and investments made inrenewable technologies. At the point of consumption, smartmetering should lead to greater collaboration with other sectors,such as telecommunications carriers and service providers.However, IT functions within E&U companies struggle to keep pacewith market change and face increasingly limited budgets, partof which is spent trying to maintain antiquated legacy systems,leaving little to spend on innovative product lines that supportnew customer interaction models and leverage up-and-comingtechnologies, as is reflected in our findings.Energy and Utility organizations say that their testing budget hasremained largely static this year, with 51% indicating that theirbudget is the same as last year. However, some 37% of respondentsin this sector say their budget has increased to some extent, and just8% record a decline. Respondents are somewhat more upbeat whenconsidering how their budget will change between now and 2015,with 54% believing it will increase. This continuing commitment toimproving the quality of testing through investment in both internaland external resources is characteristic of the E&U sector.Energy &UtilitiesCommitment to improvequality of testing throughincreased investmentBy Perry StonemanUtilities Global Sector LeaderCapgemini Willem-Jan Van Der MeerSenior Test ConsultantSogeti44 WORLD QUALITY REPORT 2012-13sector analysisEnergy & Utilities
    • Organizations continue to recognize the importance of applicationstandardization and quality process improvement by creating aTesting Center of Excellence (TCOE). Last year, E&U organizationswere ahead of other sectors in their plans to implement a TCOE, withnearly three-quarters (74%) of respondents stating that they hadeither started rolling out a TCOE or had plans to do so within the nexttwo years. However, one year later, we see a much higher proportionof E&U respondents saying that they had no plans for a TCOE – some44% of companies in the E&U sector, compared to 34% overall.In fact, only 5% of energy and utility firms surveyed stated that theyhad started to build a TCOE in the last two years, compared withother industries at 15%. Nonetheless, 43% say that they will begindeveloping their own TCOE or start using a TCOE offered by a thirdparty in the next two years. This seems to indicate an apparentdisparity between stated intention and the reality of developinga TCOE. When respondents were asked what they thought thegreatest benefit of a TCOE would be, 38% said it would reduce time-to-market, 33% said it would reduce costs, and 30% said it wouldprovide better resource management.So, if firms are not developing TCOEs, where are they investing?Firstly, energy and utility firms are investing 18% of theirtotal IT budget on testing, in line with the average across allvertical markets. There is an encouraging trend of more budget(42%) being allocated for transformational work. However, whenrespondents were asked where they saw the greatest increase intesting spend, 65% said it would be in existing tools, whereas itmight have been thought that it would be directed more towardssupporting innovative services to enhance the customer experience.The adoption of the cloud in terms of application hosting continuesto grow in the E&U sector. This is also reflected in the move towardstesting in a cloud-based environment – which also shows significantuptake now and in the next few years. Respondents in this sectorsay that approximately one in five of their applications is now hostedor has been migrated to some form of cloud and, looking ahead to2015, E&U organizations see this increasing to over a third. There isalso an indication that the amount of testing by E&U organizationscarried out in a cloud-based environment will increase over the nextthree years – with respondents predicting an average of 37% of theirtesting done this way by 2015.Almost half (47%) of E&U respondents said they plan to use Testingas a Service (TaaS) provided by a third party within the next 12months, which may enable firms to create added value fromtheir QA/testing spend not allocated towards existing tools andtransformational work. The primary benefits from a TaaS servicecited were cost reduction (58%), followed by time-to-market (46%)and standardized testing (42%).Hindering E&U firms from greater adoption of TaaS is the perceived“loss of control” with 41% of respondents selecting this. Althoughshrewd firms can navigate this issue in a strong client/agentrelationship, there is definitely a perception of loss by E&U firmscompared to 31% of all vertical markets.Moving on to mobile testing, an overwhelming 65% of E&Urespondents state that they currently do not test mobile applicationsand devices, and that the major reason is a lack of tools (68%). Asconsumers and customers continue to expand their use of mobiledevices, and the bring-your-own-device culture gathers pace, therewill be increased pressure for the E&U sectors to test mobile devicesand applications as a matter of course.Just over half of energy and utility companies (54%) use an in-housemodel for testing. However, a concerning 67% said that the qualityof their internal onshore testing resources was only “average”, andalmost half (49%) said the same for external testing resources.Combine this finding with that regarding investment mainly focusedon existing tools and this would seem to indicate a preferenceto upgrade internal expertise, together with the development ofexternal skilled resources via the cloud.With the industry under much scrutiny regarding quality processes,we may see a rise in the pace of improvement of testing capabilitiesand standards among E&U firms in the future, and increasedadoption of TCOEs. WORLD QUALITY REPORT 2012-13 45sector analysisEnergy & Utilities
    • The Financial Services (FS) industry, encompassing banking, capitalmarkets, and insurance, continues to be volatile, susceptibleto external events and therefore unpredictable. IT spend hasfollowed this trend, though there is considerable focus onrationalization and legacy transformation. US-based organizationsand other transnational financial institutions have appliedconsiderable energy and effort to understanding and implementingregulatory compliance-related changes while driving growththrough consolidation. Much of the rest of the market is focused ontransforming their current state to yield long-term gains.As systems change and evolve, a hybrid approach is emerging fromthe “build versus buy” debate – heavily customized commercialoff-the-shelf systems. This is predominant in the banking sectorfor front office operations and in the insurance sector wheremiddle and back office systems are automated using customizedpre-built software. Capital market firms continue to focus oncreating systems that stand out as differentiators, closely aligned totheir products and related business processes.As technology catches the consumer’s attention, more and morebusinesses across the globe are adopting newer channels, suchas mobile and social media. In several cases, this has resultedin a re-evaluation of the firm’s data strategy and architecturalframeworks, eliciting more change. While net funding for bothdiscretionary and non-discretionary areas has not changeddrastically, there is a renewed awareness of the need to ensure andimprove quality of the end product.This continued commitment to quality by the FS sector is evident inour survey results. The average proportion of the IT budget allocatedto the testing function for FS is about 20% – more than any othersector, with the exception of CPRD. Within FS, banking and capitalmarket firms spend more on QA.In general, QA budget has increased over the past year, trendingupwards in the forthcoming years, despite economic turbulence andslow recovery of global financial markets. The need for, and visibilityof, excellence in service levels, and adoption of new or innovativeproducts (such as mobile banking and social media), means it islikely that this commitment by FS firms will continue.The trend towards building Testing Centers of Excellence (TCOEs)has continued to improve this year, with 6% of organizations nowsaying they have a fully operational TCOE. Interestingly, this wasseen more in the insurance sector than any other. These firms tendto be highly decentralized (suggesting a federated model) and relyFinancialServicesEvolving QA – Riding thetransformation waveBy Govindarajan MuthukrishnanVice–President, Testing Practice Lead,North America, UK, APAC and Financial ServicesCapgemini Nikhil JoshiNorth American Financial ServicesTesting Practice LeaderCapgemini46 WORLD QUALITY REPORT 2012-13sector analysisFinancial Services
    • heavily on their internal resources. Some 18% of all respondentsstated that they have started developing a TCOE and expect it to befully operational within the next two years, this trend led by capitalmarket firms globally. A further 24% say they plan to start developinga TCOE in the next two years, the majority being banking firms.The number of FS respondents indicating that they intend to use athird-party TCOE has crept up to 22%. The greatest benefits seenby FS respondents are reducing time-to-market (39%) and reducingcosts (25%). The importance of cost optimization, increasedcompliance/regulatory requirements, and business growth islargely seen as the new norm. There is increasing focus by bankingand capital markets firms to combine environment managementwith traditional testing services, whereas insurance companies arefocused on cost reduction through innovative structural models.Cloud computing has passed the so-called hype cycle, and now isconsidered to be a technology that brings real benefits to both ITand business aspects of organizations. FS organizations recognizethe practical elements of the cloud, cloud-based services, andapplications, as ways to avoid ongoing costs such as buildingand maintaining their own data centers. This has resulted inrevolutionizing the way test environments are designed andoperated, with QA organizations taking a lead in the definition oftheir strategy.While there are still concerns about security and scale, there is adefinite move toward using the cloud. Just 14% of FS respondentssay they are not going to migrate any applications to the cloud (downfrom 16% last year). This year, some 79% of FS respondents say that11-50% of their applications have been migrated or are hosted on thecloud, compared to 58% last year.Indeed, the amount of testing carried out via a cloud-basedenvironment is set to increase between now and 2015. There is asignificant improvement since last year: some 60% say that up to30% of their testing now occurs this way (and only 4% say none).Banking firms lead this trend in North America, whereas in Europe,capital market firms show the way.Insurance organizations in general are more focused on extendingexisting tools, whereas banking/capital markets firms are keen toidentify newer products that would yield higher benefits.More than 60% of organizations intend to use Testing as a Service(TaaS) offered by a third party within the next 12 months, and afurther 33% say they will do so over the next two years, indicatingthat TaaS obviously resonates with the FS sector for the broadadvantages it brings. FS respondents are looking to reduce costs(61%), get better resource management (50%), and cut their time-to-market (41%) by adopting TaaS. FS organizations in the Asia Pacificregion are currently lagging behind their counterparts in mainlandEurope, UK, and North America, but this picture might undergo arevision next year as awareness around TaaS increases. In general,insurance firms seem keen to adopt TaaS in the next 24 months, ascompared with their banking and capital markets counterparts.One area of increasing attention for all sectors, and particularlywithin the banking industry, is the growth of mobile banking andmobile commerce, and the need for appropriate testing. Integrationwith current systems and validating usability and performanceneeds to be addressed within the IT environment. Some 31% ofFS respondents say they currently test mobile applications anddevices internally. Western European firms seem to have a leadon this across all sectors. There are significant challenges: 72%of respondents say they don’t have the right tools to test mobileapplications, 50% don’t have the mobile devices readily available,and a further 38% have no mobile testing experts available.Clearly, then, there is a shortcoming in the in-house ability totest mobile applications, and we can expect that gap to be filledthrough outsourcing. Indeed, when asked what would be the keycriterion in choosing an outsourced solution to mobile testing, somethree-quarters of respondents say it would be the capacity to testtheir application on multiple networks. Defining a strategy androadmap for mobile testing is a critical first step for organizationsthat have just joined the bandwagon.As worldwide financial markets continue to recover, we expect to seeongoing commitment to quality, modernization, and innovation byFS companies. Quality assurance at insurance firms is increasinglyfocused on reducing cost and enhancing existing capabilities.Banking and capital markets firms are market driven; their QAis focused on acquiring new capabilities, while increasing itsoverall scope.QA transformation should therefore be led by an increased focus oncost optimization, improving testing tools and skills, and focus onmoving quality to the left of the development lifecycle. These shouldbe evidence of the essential importance FS companies continue togive to software quality WORLD QUALITY REPORT 2012-13 47sector analysisFinancial Services
    • The High Tech sector extends across a diverse range of companiesinvolved in the design, manufacture, and support of componentssuch as semiconductors and software used in telecommunicationsequipment, transportation, and defense systems, includingthe networking and systems architecture. This sector is afiercely competitive one, with the pace of change faced by HighTech organizations being perhaps more significant than in anyother sector. Partly driving this change is the rapidly evolvingnature of the relationship and interaction between consumers andorganizations – not least the growth of mobile technology that hasallowed consumers to interact with organizations using smartphoneand tablet devices to an ever-greater extent. This has meantincreased focus on, and investment in, systems that help improvethat customer interaction and make better use of the data collectedfor marketing and sales purposes.Given their nature, High Tech organizations have always traditionallymaintained a high level of quality in their internal applications,and it is largely ingrained in their operations. However, the growingpace of change in the sector, and in particular of mobile technology,means that consumer-facing applications are becoming increasinglythe focus for QA efforts. High Tech sector organizations are atthe leading edge of the change in how consumers, citizens, andorganizations interact – how they shape their QA approach in thecoming years will help determine how other sectors also focus onmobile and web testing.In 2012, companies surveyed in the High Tech sector are increasingtheir total investment in the testing function and expect to continuethat to 2015. Some 41% of High Tech respondents say that theirtesting budget has increased this year, compared to 38% in 2011.Looking ahead to 2015, some 57% say that their budget will continueto increase – with 15% saying it will be a significant increase,compared to just 8% of the overall study. Clearly, the pressures,driven partly by fierce competition and also a zero tolerance of failure,will mean a continued focus by High Tech companies on QA/testing.Indeed, 56% of High Tech companies say they currently intend toinvest more in 2012 in improving and developing existing testingtools to maximize their usefulness. However, by 2015, High Techrespondents predict the top two areas of investment will be testenvironments/infrastructure (50%) and new testing tools (45%).High TechIncreasing focus on data-driven customer servicedeliveryBy Jean-Pierre HervéNational Testing Manager, High TechSogeti48 WORLD QUALITY REPORT 2012-13sector analysisHigh Tech
    • These figures are far beyond other sectors, with just 22% of all theother sectors indicating they will be investing in new testing tools by2015; so it is clear that High Tech companies will be leading the wayto meet the fast pace of innovation and fierce competition they face.High Tech companies also continue to push hard in the developmentand adoption of Testing Centers of Excellence (TCOEs), with a quarterof respondents planning to develop an internally managed TCOEwithin the next two years, and a further 17% planning to use a third-party company (up from 8% last year). What are companies in theHigh Tech sector looking to gain from setting up a TCOE? Principalreasons given are reduced time-to-market (30%), improved quality(29%), and better resource management (28%).The level of industry maturity found in the High Tech sector helpsto explain the high levels of cloud adoption. Last year, 56% of HighTech respondents said that they planned to migrate 11-50% of theirapplications to the cloud. This year, some 62% say that 11-50% ofapplications are now hosted or migrated to the cloud and a further9% predict that by 2015 more than 50% of their applications will behosted or migrated to the cloud.This appears to indicate that this mature sector recognizes theadvantages of moving into a cloud environment and is keen to exploitthose opportunities. Technology vendors are very familiar with theconcept of service delivery through shared networks, and have beenamong the earliest adopters of cloud computing.Technology companies are also well positioned to judge therelevance of these technologies to their business models, and whichapplications are best suited to be used in the cloud. Indeed, lookingforward to 2015, two-thirds of respondents within the High Techsector believe that more than 30% of their testing activity will bedone in a cloud-based environment.Testing as a Service (TaaS) offered by a third party is an areathat High Tech sector companies are looking to leverage over thecoming years. Some 13% report that they already use the service,43% say they intend to use it in the next 12 months, and 31% in thenext 24 months. High Tech respondents report that the benefits tothem of using TaaS are reduced costs (61%) and reduced time-to-market (50%).A growing area of focus for all sectors is the testing of mobileapplications and devices. As more mobile devices enter boththe consumer and business arenas, there is a growing need toensure that the applications run on the devices smoothly and linkseamlessly with other existing systems.Some 30% of High Tech respondents say they currently test mobileapplications and devices, and those that do say they face significantchallenges in terms of not having the right tools (62%) and not havingthe right testing process/methods (50%). These challenges willinevitably become less significant over time, but at this juncturethere is clearly a significant gap that will need to be bridged.Indeed, when a High Tech company considers an outsourcedresource for mobile application testing, some 62% of respondentssay the key criterion is that it allows their development teams to stayfocused on development.As the economy recovers and new markets continue to expand incomplexity and demand, there is an expectation that High Techcompanies will continue to invest in applications and platformsthat help them to better respond to demands made of them, bothinternally and externally. These companies have always been amongthe earliest adopters of emerging technologies, and this positioncontinues to be reflected in their approach to QA and testing. WORLD QUALITY REPORT 2012-13 49sector analysisHigh Tech
    • The pace of technological innovation, strong demand for customer-centric processes, and the need for ever-greater efficiency areresulting in profound changes within the Public Sector. While manygovernments in developed countries are cutting public spending asthey attempt to reduce deficits or debt, those in rapidly developingcountries with fast-growing economies face the challenge ofimproving tax compliance and social welfare provision in order tosupport social development and reduce inequality.Technology continues to enable improved performance and greaterefficiency in all areas of government. Automating processes,applying business analytics to yield insights from large volumesof complex data, making services available to citizens via onlinechannels, and replacing traditional client-server infrastructureswith cloud-enabled models are just some of the strategies thatPublic Sector organizations are pursuing in order to modernizetheir operations. Freeing up resources and reducing running costsare common goals, but so, too, is the need to offer citizens simplerand more integrated services.While large-scale Public Sector IT projects have always been subjectto close scrutiny, the renewed focus on value for money, broughtabout by the debt crisis, means that the stakes are now higherthan before. Implementing new systems and processes smoothlyand successfully the first time is more vital than ever. This perhapsexplains why most respondents this year actually report an increasein the percentage of their total IT budget that is invested in their QA/testing functions, despite downward budgetary pressure.While Public Sector organizations across the world face reducedbudgets and increased demands to do more from less, almost half ofrespondents this year actually report an increase in the percentageof their total IT budget that is invested in their QA/testing functions.Over half of Public Sector respondents expect their QA/testingspend to increase over the next three years. This may represent arenewed focus on ensuring applications and services are deliveredon time and without error, since mistakes and delivery failures arenow a matter of intense public focus and scrutiny.Indeed, the greatest challenge Public Sector organizationsreport regarding their QA/testing activity is that of changingbusiness requirements. However, fully 30% of respondents do notbelieve their QA budget will increase between now and 2015.Public SectorRenewed focus on efficiencygains linked to budgetreductionsBy Nancy MeiersDirector of IV&VNA Public SectorCapgemini Chris BurnsNA Testing Public SectorCapgemini Ian PrettyVice PresidentGlobal Sectors, Tax & WelfareCapgemini Mervyn JacksonHead of Testing Public Sector UKSogeti50 WORLD QUALITY REPORT 2012-13sector analysisPublic Sector
    • Asked about the areas of QA where there is currently the greatestincrease in spending, Public Sector respondents cite investment inexisting testing tools (58%), external outsourced resources (45%),and internal professional testing resources (45%). However, whenasked to look ahead to 2015, we can see significant changes. Whilethere is still predicted to be a large focus on investment in existingtools (48%), there is a significant increase in the investment in testenvironments (44%) and new testing tools (44%); indeed, the level ofinvestment in new testing tools is double that of the global average.This changing focus of investment may reflect the increasingimportance of Testing Centers of Excellence (TCOEs) to thePublic Sector. Some 19% of Public Sector organizations say theyalready have a TCOE fully or partly operational. A further 52% plan tostart using a TCOE over the next two years, 30% say they will developan internal TCOE, and 22% say they will use a third party. Our PublicSector respondents believe that a TCOE will help reduce costs andimprove their speed/agility to support organizational objectives.This year, as with last year, our survey shows that the cloud is asignificant element of Public Sector thinking regarding applicationsand testing. In 2011, nearly half of our respondents said thatthey were planning to move 11-50% of their applications to acloud environment. This year, some 60% of respondents say that, by2015, 11-50% of their applications will have been migrated or hostedin the cloud.The nature of Public Sector organizations makes the adoption ofcloud (with private cloud infrastructure probably predominating) asensible approach to the interconnection and sharing of information.Linked to this move to the cloud for applications and that ofTCOE adoption is a growing use of testing in a cloud-basedenvironment, with the percentage of testing done by Public Sectororganizations via this method expected to be 38% by 2015. Thechallenge, of course, is to develop the QA necessary to ensure thatsuch a move to the cloud is carried out while removing the risk ofsignificant application failure, which tends to be very public andhighly damaging.Testing as a Service (TaaS) also continues to grow in importance toPublic Sector respondents. While 13% say they already use TaaSoffered by a third party (compared to 11% overall), a further sizeable53% say they will start to use TaaS in the next 12 months, and 27%within the next 24 months.The benefits that respondents believe they can achieve through TaaSare reduced costs (56%) and better resource management (49%). Aspart of the Public Sector’s wider attempts to significantly cut theircosts, whilst still improving their QA/testing function, this wouldtherefore seem an attractive option.Equally, and with regard to their QA/testing software licenses, PublicSector respondents see the use of Software as a Service (SaaS) asa way to reduce costs yet maintain access to specific tools. Theyalso forecast that, by 2015, up to 43% of their QA/testing softwarelicenses will use the SaaS model, in which licenses are paid for onlywhen being used.A growing area of activity that all organizations have to take intoaccount is the proliferation of mobile applications and devices, andthe challenges of ensuring that services work across all platformsand devices. Just 27% of respondents from the Public Sector saythat they currently test mobile applications and devices, comparedto 31% overall. With the growing importance such devices play inthe everyday lives of citizens, it is clear that mobile communicationis important, not just for consumer-oriented organizations, butgovernments and public bodies too.However, the internal existing testing and QA function within thePublic Sector may not have the right skills or environments to testthese devices and applications. Indeed, some 64% of our PublicSector respondents report that they do not have the right tools, anda further 51% say they do not have the devices readily available.Given the significance of mobile communication in citizen interactionwith Public Sector organizations and the rise of a bring-your-own-device culture, this is an area that will demand greater attentionand quite possibly the use of specialist testing skills outsidethe organization.The findings in the World Quality Report 2012-13 for Public Sectororganizations show significant commitment to improving theirinternal testers and testing tools – it is clear that they are turningto external resources for specialist requirements. The developmentof TCOEs and the increasing take-up of both TaaS and SaaS showthat third-party involvement is seen to be essential to both reducingcosts and improving levels of QA/testing. WORLD QUALITY REPORT 2012-13 51sector analysisPublic Sector
    • The Telecoms, Media, and Entertainment (TME) sector includes avariety of telecoms businesses such as internet, cable, and mobileservice providers, as well as network equipment suppliers andmedia and entertainment companies. It is fair to say that TME isamong the fastest-paced and most innovative industries, withcompanies continuously revising their business models, offeringnew packages, and bringing additional services to their clients. Injust a few years, telecoms companies have shifted from offeringseparate landline and mobile phone services to delivering multi-playbundles that include everything from mobile apps to broadbandinternet and even TV.In that context, for any telecoms service provider nowadays,the relationship with the customer is critical. A weak customerrelationship management (CRM) system could lead to rapid loss ofcustomers, breaches in data security, damage to the brand, andsoaring costs.Yet the support systems of many TME companies are being pushedto their limits more than ever before. In order to remain competitive,telcos need to provide multiple services over a range of devices,engaging with their customers across mobile and internet platformsin real time.Failure to put in place a robust and evolving testing environmentrisks not keeping pace with fast-moving service environments andalso damaging the customer experience. Conversely, sufficientinvestment in a testing and quality assurance strategy will enableorganizations to reduce costs, standardize processes, and improveservice quality and time-to-market.Testing of mobile applications and those in the cloud is on the rise,as is the process of consolidating outsourced partners and testingpractices into centralized and standardized Testing Centers ofExcellence (TCOEs).This year’s report confirms that companies in the TME sector arecontinuing to increase their QA/testing budgets to maintain theirdrive towards a better quality of service, although the outlook forthis growth is less optimistic than in the past. Last year, 54% ofTME respondents said that their QA budget had grown over the pasttwo years. In 2012, this has fallen. Some 48% respondents say thatTelecoms,Media, andEntertainmentMaximizing assets, unleashinggrowth, and transforming tosucceedBy Mark BuenenGlobal Testing Services, Telco LeadSogeti Pierre BlanchardVice PresidentStrategic Sales & AlliancesTME Global SectorCapgemini52 WORLD QUALITY REPORT 2012-13sector analysisTelecoms, Media, and Entertainment
    • their QA budget has increased over the past year, and 51% predictthat it is likely to grow again by 2015. This reflects the need forcompanies in the sector to react to continued fast-paced innovationand competition by ensuring their service levels and delivery arewithout fault.As noted last year, TME companies are beginning the process ofconsolidating their outsourced partners and testing practicesinto centralized and standardized TCOEs. While the number ofTME companies who say they have a fully functional TCOE hasonly slightly increased from just 4% last year to 6% this year, thevast majority of respondents (65%) report that they are currentlyworking on developing a TCOE or have plans to do so within the nexttwo years.The timing may be ambitious, but the intent and desire to gain thebenefits of an active TCOE are quite clear to respondents – andunsurprisingly, given the dynamics of the TME sector, a reduction intime-to-market is seen as the most significant benefit of all.Cloud continues to spread into various activities in an organization,and the QA/testing function is no different. Respondents amongTME companies say that the amount of testing done in the cloud willrise by 12% between now and 2015. Indeed, they predict that this willrise to a third of their applications that are migrated or hosted in thecloud by 2015.Testing as a Service (TaaS) offered by a third party is an increasinglyutilized option. With 12% of respondents saying they already haveTaaS in place, and a further 48% reporting that they plan to useTaaS within the next year, we can see a definite move towards TaaSin this sector. Indeed, only 6% of TME companies say they have noplans to use TaaS, well below the global average of 11%.Why has this sector adopted this model so wholeheartedly? Well, oursurvey suggests that TME respondents see TaaS as offering a way toreduce costs (58%) and develop better resource management (51%).The survey also shows a similar move towards using QA/testingsoftware licenses exploiting the Software-as-a-Service (SaaS)model, by which organizations only pay for licenses when they arebeing used. TME respondents report that 36% of their QA/testingsoftware licenses use this model currently, but that this will rise to48% by 2015.A growing area of focus for all sectors, but obviously of particularinterest for TME companies, is that of mobile application testing.Some 30% of TME respondents say they are currently testingmobile applications and devices. Since one of the most significantchallenges to testing mobile applications cited by our respondentsis that of not having the right tools to test (62%), we can expectan outreach for specialist expertise in developing mobileapplication testing. Indeed, 58% of TME companies surveyed saythat if they were to look at outsourcing their mobile testing, theirkey criterion for choosing to outsource would be to keep theirdevelopment team focused on development.The TME sector continues to grow at a faster pace than mostother sectors, driven by the decline of monopolies and growthof non-traditional actors, which has meant service providershave to compete for customers more vigorously. The rapid paceof innovation, new product development, and roll-out of 4G/LTEtechnologies will continue to fuel this competition.This means additional demands for IT and testing toensure that there is a seamless link between the varioustechnological components. To meet these demands and win andretain customer loyalty, companies will need to ensure their QA/testing is fit for the task and agile enough to cope with the growingand changing demands made of them. WORLD QUALITY REPORT 2012-13 53sector analysisTelecoms, Media, and Entertainment
    • CountryandRegionalAnalysis56 Australia and New Zealand58 Brazil60 China62 France64 Germany66 The Netherlands68 The Nordic Region70 North America72 United Kingdom54 WORLD QUALITY REPORT 2012-13
    • WORLD QUALITY REPORT 2012-13 55Country and Regional Analysis
    • Top TrendsÊÊ There has been a significant movetoward the development of TestingCenters of Excellence (TCOEs).ÊÊ 72% of Australian respondentssay they do not have the right toolsto test mobile applications anddevices adequately.ÊÊ Australian organizations usingTesting as a Service (TaaS)offered by a third party are set toincrease significantly.Australia is a truly dynamic market, with a competitive,industrialized economy underpinned by a strong service sector, andhas avoided much of the pain of global financial contraction. Keyorganizations still need to plan strategically, act with caution, andoptimize spending on processes and productivity in order to ensurethey stay ahead of global and local competition. Major players arefocusing on transformational projects, to increase efficiencies andachieve operational gains so that QA remains a key lever to generatemeasurable benefits, while also looking to control change in theIT ecosystem.Some 60% of respondents in Australia (compared with 51% globally)prefer to use an onsite testing resources engagement model, asopposed to nearshore or offshore resources. However, only 27%rate the quality of their testing resources as above average. Withthe market changing rapidly and the desire to use new applicationsfor enterprise gain, there has to be a drive to lift the quality ofonsite testing resources or look to trusted providers for additionalskilled resources.One challenge faced by many organizations is that of structuringthe testing function for optimized delivery. Only 8% of Australianrespondents (equal to that of all other countries) use a testingfunction centralized across the organization, and around half use amix of decentralized and centralized testing. This often leads to thequestion of who really owns the testing activity, which in turn canlead to underperformance and suboptimal development.Australiaand NewZealandBy Nick FinlaysonGlobal Testing ServicesAustralia/New ZealandCapgemini Melanie BrockSenior Test ConsultantAustralia/New ZealandCapgemini56 WORLD QUALITY REPORT 2012-13Country and Regional AnalysisAustralia and New Zealand
    • Getting quality development delivered on time provides furtherchallenges for Australian enterprises: lack of defined testingprocesses and metrics (32%); lack of overall coordination betweenbusiness and development teams (30%); poor development/codingstandards (27%); and lack of overall coordination between businessand testing teams (28%).However, we also see that change is in progress, with a significanttrend towards the use of Testing Centers of Excellence (TCOEs) todrive down testing/development time, reduce cost, and also improveoverall resource usage. Only 34% of Australian respondents haveno plans to implement a TCOE – down from a very high 63% in 2011– with 22% planning to have an internal TCOE operational within thenext two years.Australia reports one of the highest levels of current TCOE usage,with a focus on Program Management Office (PMO) services,together with training and knowledge management and control oftesting methodology and processes. Having seen that the qualityof internal testers is at best average, there is a clear drive to lift theoverall quality of the testing process. Achieving innovation throughthe sharing of people, processes, and tools in a TCOE can alsoincrease overall productivity.The major driver for introducing a TCOE is cited as being the desire toreduce time-to-market for core developments and the improvementof resource management. TCOE is not just about cost reduction, andindeed this is seen as only a minor benefit.Views on budgets indicate a strong commitment to testing. Theaverage proportion of IT budget invested in testing is the same asthe global average of 18%. While it could have been higher – takinginto account the strength of the Australian market – it still indicatesan underlying commitment, supported by analysis of where thebudget is invested. Some 45% of the budget, compared with 41%globally, is being invested in new transformational projects, asopposed to maintenance work, and some 58% on customer-facingactivities, as opposed to employee-based ones. The QA areas forgreatest investment in 2012 will be in existing testing tools, externaloutsourced resources, and test environment/infrastructure.Looking at actual testing activity in more detail, 60% of firmsagree that they consistently measure QA metrics across the entirecompany, but methods of measurement vary greatly. Only one in10 Australian firms uses industry-standard estimation methods;around a quarter of firms use in-house-developed estimationmethods, and 30% take a percentage of the development effort.This may reflect an increasing maturity in terms of testing, orsimply an added focus on metrics. Either way, there is still roomfor improvement – in our opinion, there is still too much manualcollection and reporting of data. The market also accepts that itneeds to mature, because, when questioned on the ideal percentageof automated test cases, the response is 40%, while companiescurrently only achieve 32%.Another key trend identified is the move to cloud-based activities.Currently 24% of testing is in a cloud-based environment, but thisis estimated to grow to 36% by 2015, by which time, it is estimated,those making the move to the cloud will have done so. Testing asa Service (TaaS) will also see significant acceptance in the nexttwo years. Some 19% of Australian respondents (higher than the11% global average) report they already have TaaS offered by athird party. Looking ahead, 38% expect to use it in the coming 12months, and a further 24% in the 12 months after that – a significantincrease over the next two years. Only one in five Australiancompanies reports they will not be using TaaS.The key drivers for the increased use of TaaS include cost reduction,better resource management, and reduced time-to-market,with only limited challenges being reported, which center on lossof control over the total testing spend (mentioned by a third ofAustralian firms) and security issues.Mobile applications and devices are also set to change the wayorganizations operate, driving greater functionality and productivity.But this demands additional testing, and, to date, only 39% ofrespondents report that they are undertaking testing on any mobileapplication or device – still higher than the global average of 31%.The key reasons for the lack of mobile testing are that organizationsdon’t believe they have the right tools, mentioned by 72%, while 39%don’t have the appropriate mobile devices available. Mobile testingwill be a critical area of development in the next few years as mobileapplication development grows rapidly. Robust testing is crucial to asmooth customer experience.For those respondents who do test mobile applications, testingeffort is focused on application efficiencies and performance(including network performance), portability (various mobiledevices), and functionality. If mobile testing is outsourced, thekey criterion identified for selecting an outsourced vendor is theability and capacity to test applications on several networks.Finally, in terms of ensuring that the applications are secure, ahigher percentage of Australian firms than global ones leave thefunction to information security, but one in 10 say they have noprimary owner.Looking forward, there is clear evidence in the Australian marketthat the focus is now on new and emerging technologies to generatethe organizational and productivity efficiencies required, and thatthese include increased mobile application and device usage, bring-your-own-device, and controlled use of the cloud. WORLD QUALITY REPORT 2012-13 57Country and Regional AnalysisAustralia and New Zealand
    • BrazilBy Flavio Leomil MariettoGlobal Testing ServicesBrazilCapgemini/CPM BraxisTop TrendsÊÊ The high level of investment inQA is set to continue.ÊÊ Testers are kept close to theorganization, with constantinteraction between parties.ÊÊ Movement toward testing in thecloud is growing.Brazil, now positioned as the sixth largest economy in the world,continues its rise as a dynamic and flourishing economy. While itsgrowth rate has been impacted by the global banking crisis andresulting economic turmoil, Brazil remains a robust market and islikely to become stronger as the global recovery picks up. Indeed,with strong inward investment, significant oil and gas discoveries,large-scale government initiatives, and investment in infrastructure(upcoming World Cup and Olympics), analysts widely predict that itcould soon become the fifth largest global economy.As the Brazilian market continues to grow in both size and reach,so its testing services continue to mature, with increased focuson resources and investment. The majority of organizations preferto keep their testers very close to the organization. Some 56% ofrespondents in Brazil say their testers are internal. Indeed there’sa major preference for testers to remain close to the business, withsome 62% saying that onshore is the preferred location (compared to52% of the global market). Whilst more organizations are likely to useoffshore/nearshore testers than last year (38% this year versus 30%in 2011), it is still below other geographies, and reflects the growingmaturity of the Brazilian market. This analysis indicates a view of amarket in which testing services are closely aligned with the end-users and require constant interaction between the parties.It is interesting to note that, in contrast to the overall average for thestudy, Brazil rates the quality of its internal testers far higher thanother countries do. Some 45% of companies say that their internaltesters’ quality is better than average – compared to just 23% of theoverall market. Brazilian organizations also rate the quality of theirexternal testers far higher than the overall market – suggesting acontinued move towards external testing, despite the confidence inin-house testers.58 WORLD QUALITY REPORT 2012-13Country and Regional AnalysisBrazil
    • Further evidence of this commitment to developing and improvingtesting can be seen in budget allocation – most clearly in thepercentage of IT budget allocated to the testing function. Brazilianrespondents cite a figure of 25% – far higher than any othergeography (global average is 18%). Despite the difficult globaleconomy, 37% of Brazilian respondents say their QA budget hasincreased over the last year.Indeed, 55% say they expect it to increase slightly or significantlyby 2015. It is encouraging to note that 48% of the budget is allocatedto new, innovative or transformational work (compared to 41%globally), further supporting the drive for quality, innovation, andgrowth by Brazilian organizations.While our observation of the Brazilian market shows that testing is,in many cases, still rather immature, our survey shows there is acontinuing trend to increase the level of testing expertise and skills.Where are Brazilian organizations looking to increase spending intheir testing budget? Right now, 55% say it is on their existing testingtools and 48% say it is on internal professional testing resources.Looking ahead to 2015, respondents still expect to spend onimproving their internal professional testing resources (52%), butalso foresee significant investment in external outsourced resources(46%) and new testing tools (37%).This investment in improving the sophistication of testing andQA metrics is necessary, as much is still done manually andwithout automation. Almost double the amount of respondents fromBrazil (40% versus 21% globally) say their QA metrics are collectedand shared manually using Excel.While only 12% of respondents from Brazil (compared to 21% of thetotal respondent base) say they have already started an in-houseTesting Center of Excellence (TCOE), 51% say they plan to developan internally managed TCOE or use a third-party company. Indeed,last year, 29% of Brazilian organizations said that they planned todevelop their internally managed TCOE within two years. This year,that figure has risen to 35%.While organizations may slip on the timing, this represents asignificant move towards developing and adopting a TCOE. What willbe involved in the scope of these TCOEs? Fully 60% of respondentssay that training and knowledge management of testing activitieswill be their top priority, followed by execution of functional testing(54%) and providing Program Management Office services (51%).Turning to the cloud, we reported last year that organizationswere keen, if cautious, in their move towards using the cloud as atesting environment. In 2012, we see this trend continuing, with just16% of respondents in Brazil saying none of their testing currentlyuses the cloud as a testing environment. This figure is expected tofall even further to 6% by 2015, showing a significant adoption of thecloud as a testing platform.The predicted increase in the percentage of testing that occurs ina cloud-based environment will be nearly half of all testing activityby 2015 – higher than the overall global figure of 39%. Despitetheir reservations, Brazilian organizations expect the percentageof applications to be hosted or migrated to the cloud by 2015 toincrease by 55% – significantly higher than that of the overall figurefor the survey.While 29% of Brazilian respondents say that they have no immediateplans for deploying Testing as a Service (TaaS) offered by a thirdparty (compared to 11% of the overall global figure), 59% say thatthey plan to start using it within the next 24 months. Althoughorganizations may not have adopted this model so far, there isclearly a significant proportion of the market looking to TaaS as away to reduce costs, better manage resources, and reduce time-to-market. Further, while there seems to be a growing realization thatdeploying a TaaS model can help reduce infrastructure and softwarelicense costs, and bolster the lack of mature testing skills within theorganization, the research found that the desire to have the testersclose to business operations may prove to be a barrier for manyorganizations surveyed.Just over a quarter (26%) of Brazilian organizations say they arecurrently testing mobile applications and devices – compared to 31%globally. What are the greatest challenges to organizations in testingmobile applications and devices? Nearly two-thirds (65%) say thatthey don’t have the right tools to test appropriately, a further 35%indicate they don’t have the right testing process/method, and 35%say they don’t have time to test.Clearly, given the growing sophistication of the Brazilian market,mobile testing will become a necessary requirement of organizationswith heavy consumer/citizen exposure. Finding the right skills, theright resources, and the time to test mobile applications and deviceswill become increasingly critical to business success.The Brazilian market is clearly maturing in its approach to QA andtesting, with an ongoing and expanding commitment to investmentin IT in general and in testing specifically. This is starting to showdividends, and, in a more competitive environment, the qualityof business products and services will become a key factor fororganizational success. Looking ahead, Brazilian organizations willcontinue to mature as they move along the path of testing in order torealize tangible benefits from their QA activities, while adopting newmodels and technologies along the way. WORLD QUALITY REPORT 2012-13 59Country and Regional AnalysisBrazil
    • Top TrendsÊÊ Increased investment in QA/testingfunctions is focused on improvinginternal resources and the use ofexternal expertise.ÊÊ Developing a Testing Center ofExcellence (TCOE) is seen as away to catch up with more maturetesting markets.ÊÊ There is continued and stronginterest in the use of cloud services.China is the world’s second largest economy after the United States.It is the world’s fastest-growing major economy, with growth ratesaveraging 10% over the past 30 years. China is also the largestexporter and second largest importer of goods in the world, and issecond to the US in the value of services it produces.The provinces in the coastal regions of China tend to be moreindustrialized, while regions in the hinterland are less developed.As China’s economic importance has grown, so has attention to thestructure and health of the economy. The government’s twelfth Five-Year Plan, adopted in March 2011, emphasizes continued economicreforms and the need to increase domestic consumption in order tomake the economy less dependent on exports in the future.As part of this rapid advance of China as a key and leading economicplayer, Chinese companies are very aware of, and seeking to address,real or perceived gaps in their QA/testing functions. Chinesecompanies continue to grow their investment in QA and testing, with20% of their total IT budget going to the testing function (comparedto 18% overall). Indeed, the investment level has increased slightly orsignificantly for some 50% of Chinese respondents since 2011.There is little sign that this commitment to QA/testing will slow, as afurther 60% say the same is likely through 2015. When asked whichareas of QA are seeing the greatest increase in spending, they saythat more investment in existing tools (47%), internal professionaltesting resources (45%), and external outsourced resources (41%)take the lead. Looking ahead to 2015, Chinese respondents saythe leading area of spend will be test environments/infrastructure(51%). Clearly, Chinese companies are seeking both to improve theirinternal expertise and to make use of external resources to drive uptheir levels of QA excellence.ChinaBy David ChenSenior ManagerGlobal Delivery Center, ChinaCapgemini Jerome XieVice PresidentGlobal Delivery Center, ChinaCapgemini60 WORLD QUALITY REPORT 2012-13Country and Regional AnalysisChina
    • While the level of fully operational Testing Centers of Excellence(TCOEs) remains below that of the global picture, Chinese companiesare well on the way to developing and adopting the TCOE concept.Last year, we reported that 84% of Chinese organizations planned toset up a TCOE within the next two years (either developed internallyor using a third party). This year, 36% report that they have begundeveloping an in-house TCOE (but yet to be fully functional), whichis more than double the 15% of companies globally. A further 24%of organizations aim to develop an internal TCOE within the nexttwo years, and 22% plan to use a third-party company. Chinesecompanies are clearly keen to invest, in order to catch up withthe more mature markets, and they see the adoption of TCOE asa means to reduce time-to-market, reduce costs, and improveresource management.Chinese respondents are among those most interested in exploitingthe cloud, with some of the highest rates of adoption seen inour study. Last year, 69% of Chinese companies expected 11-50% oftheir applications to be hosted or migrated to the cloud. This year’sstudy shows some 74% of Chinese respondents expect that 11-50%of their applications will be migrated to the cloud over the nextthree years.Indeed, Chinese companies report that an average 25% of theirapplications are already hosted in the cloud, with an average of32% predicted by 2015. This latter figure is very much in line withthe global average predicted for 2015, by which time it is likelythose applications that can be moved to the cloud will have alreadybeen moved. This move to the cloud is also reflected in specifictesting functions, with Chinese respondents saying that a thirdof testing now occurs in a cloud-based testing environment. Thisis predicted to increase slightly between now and 2015, but againappears to have reached a “capped” level (39% of testing) – in linewith expectations globally.Associated with this strong adoption of cloud and cloud-basedservices and the drive for improved QA/testing resources, Testing asa Service (TaaS) and Software as a Service (SaaS) continue to showsolid take-up among Chinese companies. While just 11% of Chineserespondents (in line with the global average) use TaaS currently, ourreport shows this will change significantly over the next two years.Nearly half (47%) of all Chinese respondents say they plan to useTaaS over the next 12 months, with a further 37% saying they plan touse it over the next 24 months.What do Chinese companies hope to achieve through their use ofTaaS? A large majority of them (70%) are seeking to reduce theircosts, with a further 57% seeking better resource management, and39% looking to standardize tracking of projects through metrics.Among Chinese companies, SaaS is also increasingly important– with respondents saying that, by 2015, 44% will adopt themodel of only paying for QA/testing software licenses whenthey are being used. All of this demonstrates a rapidly maturingQA environment.Of growing importance to both consumers and organizations ismobile technology and its associated applications and services. Thisexplosion of mobile devices and applications has led to a revolutionin communications, interaction, and commerce, but also offers freshchallenges to QA and testing. Some 30% of Chinese respondents(in line with the overall figure) indicate they currently test mobileapplications and devices – but even among those who do, there aresignificant areas of concern. Some 79% of Chinese companies whocurrently test mobile applications say that they don’t have the righttools to test, and a further 34% say they don’t have mobile testingexperts available. It appears that organizations may need to find newresources and skills to ensure the proper level of attention is given tothis important aspect of testing.In summary, our report shows that, to add excellence of quality tothe fast-paced Chinese economy, Chinese companies are rapidlymoving along the path of QA and testing maturity. The continuedinvestment in QA budgets in general, and adoption of specific,specialized skills and resources that further lift and develop QAlevels, will bring world-class, quality practices and leading-edgetechnology to Chinese organizations.74% of respondents expect that up to 50% oftheir applications will be migrated to thecloud over the next three years WORLD QUALITY REPORT 2012-13 61Country and Regional AnalysisChina
    • Top TrendsÊÊ 24% of French respondents plan todevelop an internal TCOE within thenext two years.ÊÊ A third of testing is done insome form of cloud-basedtesting environment.ÊÊ There is dramatic growth of mobileapplication and device testing.In last year’s World Quality Report, we remarked that the Frencheconomy has proven to be more resilient than many other countriesduring the recent economic crisis. France, the second largesteconomy in the euro zone after Germany, is now facing a publicfinance squeeze. In the private sector, there continues to be muchevidence that firms are expecting to increase their IT budgets, with acontinued focus on new projects and infrastructure transformation.In this year’s report, French respondents indicate that their testingbudget represents 21% of the total IT budget, higher than thatin the rest of Western Europe and globally. Some 68% of Frenchcompanies also report an increase in their QA budget over thepast year. A further 56% predict an increase between now and 2015.However, 29% say that their QA budget is likely to decrease slightlyor significantly between now and 2015. This perhaps representsa general ongoing cautiousness regarding the wider Europeaneconomic situation over the next few years.Where do French respondents see the greatest level of QAinvestment now and over the next three years? Some 47% of Frenchorganizations say that their current increase in spending is inexisting testing tools, 45% in internal professional testing resources,and a further 40% say it is in external outsourced resources.By 2015, this changes, with the greatest investment in externaloutsourced resources, then internal professional testing resourcesand test environment/infrastructure. This may represent thecontinued twin drive of improving internal resources while alsomaking use of external resources for specialized or specific testingrequirements driven by technological developments.FranceBy Natacha PickaertNational Practice Manager TestingFranceSogeti Olivier NoraVice President, PracticesStrategic Alliances & DeliveryFranceSogeti62 WORLD QUALITY REPORT 2012-13Country and Regional AnalysisFrance
    • French respondents seem to take a slightly different approach tothe development of Testing Centers of Excellence (TCOEs) than theoverall study. Some 40% of French companies indicate they have noplans in place to develop a TCOE (compared to 34% overall and 21% inGermany). However, a significant majority either plan or have begunto develop/use a TCOE. Some 24% plan to develop an internal TCOEwithin the next two years, 14% plan to use a third-party company, anda further 14% have begun development of an internal TCOE.The main areas included in the scope of French TCOEs are trainingand knowledge management of testing activities (49%) and controlover testing methodology and processes (45%). The key benefits forFrench respondents are improved QA agility/time-to-market (66%),better quality (38%), and reduced costs (32%).The growing role of cloud computing in organizations throughoutthe world is echoed here in France. While local laws and regulationsregarding data hosting and security undoubtedly shape the patternof national cloud computing uptake, French organizations alreadysee the potential and are exploiting it. Last year, nearly half of allFrench companies surveyed said that 11-50% of their applicationswould be hosted or migrated to the cloud (including both public andprivate clouds). This year, that figure has risen to 65%, and Frenchrespondents believe this will continue to increase.When asked to look to 2015, 18% of respondents expect that morethan half of their applications will be hosted in the cloud (comparedto just 2% currently). And as applications move to the cloud, sodoes an element of QA/testing. French companies indicate thatapproximately a third of testing is done in some form of cloud-basedtesting environment, although they do not expect this to increasebetween now and 2015, suggesting that any testing that could bemoved to the cloud has already been moved.Connected to cloud adoption is the growing move towards bothTesting as a Service (TaaS) and Software as a Service (SaaS). Frenchrespondents show some of the strongest movement towardsadopting TaaS, with just 3% of companies saying that they have noplans (compared to 11% overall). Some 45% of French respondentsplan to start using TaaS within the next year, and a further 43% planto start over the next two years.The main reasons companies in France are looking to use TaaSare to reduce costs (61%), have better resource management(45%), and reduce time-to-market (40%). While there are fewperceived challenges to adopting TaaS, almost a quarter of Frenchrespondents say that loss of internal know-how is an issue.On average, French companies say that a quarter of their QA/testingsoftware licenses use the SaaS model (“pay as you go”), and this isexpected to rise to 35% by 2015 – somewhat lower than the overallfigure of 43%.A growing area of focus for organizations is the dramatic growth ofmobile application and device testing, with the related business andconsumer interactions providing a significant QA challenge. Newmobile applications and services have to be tested and deliveredquickly to match demand. Some 37% of French respondents saythey already test mobile applications and devices, but also reportsignificant challenges in doing so.Top of these challenges, mentioned by 73% of French respondents,is the fact that they don’t have the right tools to test mobileapplications and devices. Other challenges are non-availabilityof devices to test (43%) and lack of in-house mobile testingenvironments (41%).In summary, French companies continue to look to improve QA levelsand to find new ways to reduce costs, enhance their time-to-market,and increase levels of excellence within their testing function.Looking ahead, we expect to see further investment in internaltools/resources and an increase in external outsourced resources –particularly regarding the adoption of cloud-based services and newtechnology-led challenges such as mobile testing.45% of French respondents planto start using TaaS within thenext year WORLD QUALITY REPORT 2012-13 63Country and Regional AnalysisFrance
    • Top TrendsÊÊ There is a significant increase inmove towards development of aTesting Center of Excellence (TCOE).ÊÊ A cautious migration ofapplications towards the cloud hasbecome evident.ÊÊ More German organizationsare testing mobile applicationsand devices than in any othercountry surveyed.Germany is known as one of the leading nations inindustrialized production. One of the best examples of this is theautomotive industry, where a high degree of automation goes hand inhand with high flexibility in respect of individual client requirements.Industrialization has also reached the German IT industry. Thecontinued success of German companies increasingly depends ontheir ability to quickly introduce IT-based innovations at competitivecost levels. This, in turn, raises the pressure on IT departments tostreamline their processes and increase the use of specialized ITservice providers. The general careful pace of introducing cloudcomputing innovations reduces the opportunities for Germancompanies to make further cost reductions in IT. This cautionis mostly driven by the uncertainty of data security issues. Theexpectation is that, as soon as the data security issue is solved,Germany will invest heavily in a quick and high-quality transferto cloud computing. Cloud testing will then become an importanttheme, due to the strict German data security laws.German companies are among the highest investors in testingfunctions and show a continued desire for excellence in QA/testingin this extremely mature and competitive market. Asked howthe testing budget has changed since 2011, some 46% report anincrease, and 44% predict further increases by 2015. While thisindicates a commitment to ensuring German QA/testing functionsare appropriately funded, there is also a note of caution. Some39% of German respondents predict a decline or severe decline intheir testing budget over the next two to three years. Perhaps thisGermanyBy Stefan GerstnerVice PresidentGlobal Testing ServicesGermanyCapgemini Group Vincent GroenerVice PresidentGlobal Testing ServicesGermanySogeti64 WORLD QUALITY REPORT 2012-13Country and Regional AnalysisGermany
    • represents a general pessimism about the wider European economicsituation and the need to control costs in anticipation of rougheconomic times.We asked German respondents which areas of QA will see thegreatest increase in spending. Some 56% say existing testingtools, 44% internal professional testing resources, and 40% testenvironments/infrastructure. When asked to look ahead to 2015,52% predict that test environments/infrastructure will receive thegreatest increase in spending. Interestingly, German companiesdo not seem to view increased spending on external outsourcedresources as being particularly significant.This year’s survey shows that the concept of Testing Centers ofExcellence (TCOEs) has risen in take-up across the world, andGerman respondents have made a clear move in that direction. Some53% of German companies in 2011 said that they had no plans touse a TCOE. This year that figure has dropped to only 21%. Moreover,some 42% of German respondents indicate that they have started orplan to start developing a functional TCOE internally within the nexttwo years – in synch with the global average of 40%.A further 31% say they plan to use a third-party company, some 10%more than the global average. German companies expect a TCOE toprovide Program Management Office services (51%) and training/knowledge management of testing activities (48%). What do Germancompanies expect to gain from a TCOE? Reduced time-to-market iscited by 48% of respondents, and a further 26% say it will allow forstandardized tracking of projects through metrics.Last year, we reported that German companies seemed morecautious towards cloud computing than other countries. The vastmajority of German respondents (67%) in 2011 expected that onlya quarter of their applications would be hosted or migrated to thecloud over the next year. Indeed, this appears to have been realistic:one year later, we see that 66% indicate that fewer than 30% of theirapplications are currently hosted on or migrated to the cloud.In fact, the reality of cloud adoption is proving a far more cautiousmove globally than estimated in previous years, and Germanrespondents forecast only a slight increase in the number ofapplications hosted on or migrated to the cloud between nowand 2015. It may be that those applications that can be moved to thecloud have already been moved and that only small increases willoccur, or that traditional cloud concerns remain.That said, the percentage of testing in a cloud-based environmentcited by German respondents is among the highest found inour survey. Respondents say that 35% of their testing currentlytakes place in some form of cloud-based environment (compared to28% globally) rising to 38% by 2015. As we saw with cloud-hostedapplications, it is debatable whether further increases beyond thispoint are likely.The use of Software as a Service (SaaS) continues to grow, withGerman respondents saying that currently 32% of their QA/testingsoftware licenses use the SaaS model, where they only pay forlicenses when the software is being used. When asked to look aheadto 2015, respondents believe that some 42% of their QA/testingsoftware will be used in this way.Similarly, Testing as a Service (TaaS), offered by a third party, isanother area where we see an increasing adoption of the serviceover the next two years. While just 8% of German companiescurrently use TaaS (compared to 11% overall), some 39% forecastthat they will use it over the next year, and a further 47% plan touse TaaS within the next two years. A small minority (5%) of Germanrespondents (compared to 11% overall) say they have no plans to useTaaS at all.A relatively new and growing area of importance to manyorganizations is that of mobile application and device testing. MoreGerman companies (43%) report that they are currently testingmobile application and devices than in any other country surveyed(31%). Although more German companies are testing mobileapplications, they report significant challenges in doing so. Topamong those challenges are not having the right tools (67%), devicesnot readily available (60%), and non-availability of mobile testingexperts (40%).Clearly then, German companies will need to look externally tofind the skills and tools required to ensure mobile developmentis not compromised. Indeed, one of the key factors in choosing anoutsourced resource to support their internal mobile testing is thatof keeping the development team focused on development.In summary, our survey of German companies continues to showthe maturity and sophistication of the German QA/testing world.Increased investment in the testing budget combined with targetedinvestment in improving internal skills, testing environment, andinfrastructure are only likely to increase this level of sophistication.However, it is also clear that German companies are reaching out tospecialized testing resources to provide expertise that cannot bemet internally, or where this is more cost-effective or specialist. Thegrowth of mobility-related services means the pace of innovationin application development will become ever more important. Beingable to develop products and services that meet this pace, that havebeen tested and are fit for purpose, will require greater agility andspecialized skill sets. WORLD QUALITY REPORT 2012-13 65Country and Regional AnalysisGermany
    • Top TrendsÊÊ Dutch organizations show amajor move to Testing Centersof Excellence (TCOEs) over thelast 12 months as they strive toupgrade their QA activities and getapplications developed faster.ÊÊ Despite pressure to reduce costs,increased investment in QA budgetsis expected.ÊÊ Dutch organizations remain some ofthe most mature and developed intheir focus on QA/testing.This year’s report confirms that Dutch organizations remain someof the most mature and developed in their focus on QA/testing;the high levels of sophistication, automation, and outsourcing areall indicators of maturity. While there is a continued preferencefor in-house testing and internal TCOEs, the future for QA/testingin the Netherlands seems to lie in the adoption of new servicesand technologies that will accelerate the testing processes andmeasures and prove their value to the business.A key issue for Dutch respondents is the organization of thetesting function. Some 41% have a combination of centralizedand decentralized teams, which can lead to inefficiency andcommunication issues. Only one in 10 Dutch organizations has acentralized testing stream across the organization, and 19% haveno separate testing stream at all. A majority of Dutch respondents(52% – slightly higher than the global percentage) have adopted anin-house testing engagement model. Also, more Dutch respondentsthan the global average approach this by taking on extra staff or atemporary contractor.It is therefore surprising that only 12% of Dutch organizations ratethe quality of their internal testers as being above average, and ashaving above-average knowledge. This is very low compared to theglobal average of 23%, itself a low rating. Even the quality of externalTheNetherlandsBy Marco van den BrinkManaging Director Software ControlNetherlandsSogeti Jeffrey WannéeCluster Manager TestingNetherlandsCapgemini66 WORLD QUALITY REPORT 2012-13Country and Regional AnalysisThe Netherlands
    • testers is under question in the Netherlands, as only 27% (33%globally) rate their quality as above average. This must put strainon the QA function in many firms, as they strive to reduce time-to-market for applications.In 2011, we reported that 30% of Dutch enterprises had plans tocreate a standardized and industrialized TCOE, either on theirown or through a third-party provider. In 2012, while 35% of Dutchrespondents still have no plans to create a TCOE, in line with ourglobal findings, there has been a major move to TCOEs over the last12 months as organizations strive to upgrade their QA activitiesand get applications developed faster. Some 5% already havea fully operational internal unit, 13% have implemented a unitinternally that is yet to be fully functional, and a further 28% planto use a third-party company. Finally, one in five respondentsplans to develop their own facility within the next two years. Asplans developed in previous years become operational, we can seeDutch organizations moving towards a more centralized testingcompetence – a move we can only see gathering pace over the nextfew years. It is clear that transforming plans into reality may besignificantly more of a challenge than organizations imagine – third-party expertise may be the key to easing that challenge.Dutch respondents expect the scope of a TCOE to include ProgramManagement Office services, training/knowledge management, andimproved control of the testing methodology/processes. The keybenefits of adopting this approach include reduced time-to-market,better resource management, and reduced costs.It is perhaps surprising that only 16% of the total IT budget in Dutchorganizations is currently allocated to testing. This is marginallylower than the 18% global figure and potentially explained bythe sophistication of testing functions in this market, where newinvestment in testing is not deemed as necessary as in other markets.Respondents report that 62% of testing budgets is focused onmaintenance work (again higher than the global average) and the reston new transformational work. The fact that some 58% of the testingbudget is currently invested in customer-facing applications showsthat many organizations in the Netherlands see the growing importanceof web, mobile, and cloud developments, and are keen to invest. Thedirect exposure of the organization to its clients in these kinds ofapplications means that quality is deemed of highest importance.Dutch respondents report that the greatest increase in spending is nowin existing testing tools and internal professional testing resources.In 2015, they expect the focus to remain on existing testing tools, butpredict a shift towards investment in test environment/infrastructureand outsourced resources as organizations strive to speed up time-to-market and reduce costs.In 2011, the majority of Dutch respondents suggested that thepercentage of budgets allocated to testing had mostly remainedthe same over the previous two years. Now, in 2012, we are seeingsome major changes. While only 5% report a significant budgetincrease, 36% indicate a slight increase, and only 7% show any formof decrease. By 2015, things will have changed again. Some 57% ofDutch respondents expect budgets to have increased, while only 8%forecast a decrease – true evidence of a commitment to testing.Many Dutch enterprises still show a high level of sophistication inestimating the QA effort for a project. In 2011, we saw that 73% usedformal methods; in 2012, this has increased to 84% reporting use ofQA metrics across the entire organization, even though teams are notalways centralized. Fully 64% of respondents (higher than the 56%global average) automatically collect and share QA metrics usingMicrosoft Excel or a similar spreadsheet program, and 49% also use adefault QA management tool. This, again, is evidence of a keen focuson professionalizing the discipline and realizing a demonstrable ROI.There is strong evidence for a significant uptake in cloud use in 2012,with 21% of applications hosted in the cloud, according to Dutchrespondents – very much in line with the global average. This isexpected to rise to 33% by 2015. Dutch organizations also report that,to date, 26% of testing is undertaken in a cloud-based environment(compared with 28% globally). But by 2015, this will have climbed to 41%(58% growth over the period, compared with the global average of 39%).Testing as a Service (TaaS) offered by a third party is also a key focusfor Dutch organizations. One in 10 already uses it, 59% will makethe move to TaaS in the next 12 months alone, and only 3% indicatethey have no plans to use TaaS. The primary driver is cost reduction,mentioned by 65%. Additional benefits are expected to be betterresource management and, to a lesser degree, improved time-to-market. In 2011, our research found that Dutch enterprises were inline with other countries in their approach to deploying Softwareas a Service (SaaS). In 2012, 34% said they will use the SaaS model(compared to a global average of 31%), growing to 46% by 2015(compared to a global average of 43%).In line with the overall average, 31% of respondents currently testmobile applications and devices. The key reported challenges area lack of the right tools (more of an issue in the Netherlands thanelsewhere) and lack of available devices and mobile testing experts.Currently, mobility testing focuses on efficiency, performance,portability, and functionality – the basic core elements of testing. Wepredict that this situation will change significantly over the comingfew years, with many applications from the outset being designedaround mobile devices. WORLD QUALITY REPORT 2012-13 67Country and Regional AnalysisThe Netherlands
    • Top TrendsÊÊ Nordic respondents predictsignificant investment in new testingtools and internal professionaltesting resources.ÊÊ A quarter of Nordic respondents, morethan in any other region, say theyintend to use a third party with TestingCenter of Excellence (TCOE) capability.ÊÊ A high number of Nordicorganizations already test mobileapplications and devices, but facechallenges in doing so.Although there are signs that the worst of the recession is overin much of the Nordic region (defined in this survey as Denmark,Finland, Norway, and Sweden ), the global recovery has an impact onregional economies, and makes a return to economic growth unlikelyto be fast or robust. Companies are still mainly driven by costreduction as the number one item on the agenda.Today, in the Nordic region, QA is well established in many majororganizations, with a formalized testing procedure and supportingbudgets, and is furthermore recognized as a profession withdedicated resources. For smaller organizations, however, the testingfocus is much more erratic.Looking at this year’s Nordic survey data, organizations reportthat 18% of their overall IT budget (in line with the global average)is invested in testing. This ranges from 21% in Sweden to only14% in Norway. Some 42% of the overall regional testing budget isinvested in new transformational work, as opposed to maintenancework, so innovation is key to organizational testing behavior.The NordicRegionBy Mattias BergströmnerLeader Global Testing ServicesSweden & NordicsCapgemini Group68 WORLD QUALITY REPORT 2012-13Country and Regional AnalysisThe Nordic Region
    • Nordic organizations report that the greatest increase in testingspend is in the areas of existing testing tools and externaloutsourced resources. However, by 2015, the focus will shift, withsignificant increased spending, relative to 2012, directed at newtesting tools and internal professional testing resources.In 2011, 33% of Nordic respondents indicated that budgets for testingand QA activities had stayed the same over the previous two years, and31% indicated that it increased slightly over the same period. In 2012,49% of the respondents indicate that budgets have stayed the same,while 28% say they have increased slightly.Looking at the testing function, it appears that its overallmanagement may need to change. In the Nordics, 44% oforganizations (48% in Sweden) use a combination of centralized/decentralized teams, which can lead to difficulties in overall controland management. Only 11% of organizations in the region deploycentralized teams as a single stream across the business, but this ishigher than the global average.A key issue faced in the industry is whether testers – both internaland external to the business – are meeting expectations. Some 63%of Nordic respondents (80% in Norway) say that the quality of theirtesters is at best average. This should be an area for change over thecoming few years. With the need for testing to be faster and moresophisticated, it is our opinion that “average” will not do.Only 28% of all organizations across the region report that theirinternal testing resource quality is better than average. While 44%of respondents report that the quality of external testing resourcesare better than average (well above the global percentage), this stillleaves room for improvement.One approach to lifting the overall quality of the testing function isto look at developing a Testing Center of Excellence (TCOE). In theNordics, 40% of all organizations (higher at 65% in Denmark), saythey have no plans to implement a TCOE in the foreseeable future.Around a quarter of organizations plan to develop a TCOE within thenext two years, and a further third plan to use a third party. Almostone in 10 organizations (lower than the global average) has embarkedin the last two years on a TCOE that has yet to be completed.The core elements that Nordic respondents would want to beincluded in the scope of a TCOE include training/knowledgemanagement of the testing activities, Program Management Officeservices, and functional testing. The key benefits that Nordicorganizations seek in a TCOE include, in order of importance,reduced time-to-market, reduced cost, and better quality.Nordic respondents also recorded a significant move to Testing asa Service (TaaS). Around one in 10 organizations in the region hasimplemented TaaS to date (in line with the global average), with Norway17%, Denmark 16%, and Sweden 12% leading the way, and Finlandsomewhat lower at 3%. However, looking ahead, half the Nordicorganizations surveyed (60% in Sweden) are looking to implementTaaS in the next 12 months, and a further 19% in the next two years.For Nordic respondents, the move to TaaS is being driven by aneed to reduce costs, reduce time-to-market, and also improveresource management. Whilst some organizations may not be willingto embrace this approach, this still represents a strong signal thatorganizations see the benefit of moving to TaaS.In 2012, respondents say that 22% of testing (lower than theinternational average of 28%) is cloud based but that this will riseto 37% – in line with the international average – by 2015. To date,Nordic organizations report that they have migrated 18% of theirapplications to the cloud, less than the 22% overall seen in thesurvey this year.The technology world is changing, and mobility is one of the biggestchanges we are likely to see in the next few years. In the Nordicregion – itself a leader in mobile technology – 38% of organizations(higher than the 31% global average) currently test mobileapplications and devices, with those in Finland and Sweden aheadin this activity. The key reported challenges for mobile testing are alack of the right tools for testing, and availability of the devices.In the Nordic region, in light of the economic crisis, there is a clearmove to continue to reduce costs, as well as reduce time-to-marketfor applications and developments. The increased use of cloud-based technologies aids cost-cutting and also time-to-market, butthe issues around mobility testing will come to the forefront in theshort term, because robust mobile applications are essential to asuccessful organization. This adds cost and complexity to an alreadycomplex world.The evidence is that the testing discipline in the Nordic regionwill continue to develop, but in order to meet future marketdemands, IT and the business must broaden their approach andrealize that testing and QA work effectively at both strategic andoperational levels. This will be important to achieving core goals,such as lower cost, delivery precision, and quality. A greater focus ontesting can be an important route to improved efficiency and lower-cost productivity for many Nordic organizations. WORLD QUALITY REPORT 2012-13 69Country and Regional AnalysisThe Nordic Region
    • Top TrendsÊÊ 58% of North American respondentsplan to use Testing-as-a-Service(TaaS) in the next 12 months.ÊÊ The cloud is seen as a way tooptimize cost management, improveflexibility, and help align IT with theneeds of the business.ÊÊ 44% of North American organizationssay their QA/testing budget hasincreased this year, with 50% sayingit will increase further before 2015.North America combines the large, technologically powerful market-oriented economies of the US and Canada. With the private sectortasked with underpinning market growth, organizations are targetingincreased revenue by introducing products faster and engagingcustomers through new content-rich channels, as well as launchingnew programs to increase customer trust – highly important in a verycompetitive marketplace.As the North American economy returns to growth, organizationsare focusing their IT investments more than ever on innovation.Business-as-usual IT investments are declining as legacy‑basedsystems are overhauled and transformational projects areimplemented to drive faster time-to-market and exploitnew channels. Other factors, such as regulatory changes in theFinancial Services and Healthcare sectors, are also bringing aboutan enhanced IT focus. With all these changes, the role of QA withinorganizations is more important than ever, in order to maintain brandimage and customer loyalty, drive faster time-to-market, and ensurea strong return on investment (ROI).This year’s analysis highlights that only 20% of North Americanorganizations rate their internal testing resources as above averagein terms of quality. Even external testers are said to need additionalskills – only 26% of organizations rate their external testers’ qualityas above average. “Adequacy” will not drive change, nor will it drivebetter quality.A key question is how to structure the testing function. In NorthAmerica, 35% of organizations report that the testing functionis highly decentralized, and a further 41% operate combinedcentralized/decentralized teams. Moreover, 64% of companiesreport that internal testing functions are performed by businessNorthAmericaBy Dan HanniganSenior Vice PresidentManaged Testing Services, NASogeti Kevin QuickVice PresidentApplication Services Testing Lead, NACapgemini70 WORLD QUALITY REPORT 2012-13Country and Regional AnalysisNorth America
    • analysts or development teams. None of this engenders a commontesting approach or good communication, and can lead tosuboptimized output.In 2011, we reported an increase in standardized Testing Centersof Excellence (TCOEs) to optimize QA/testing capabilities, withalmost 50% of organizations using contractors to outsource theirtesting function. However, in the last 12 months, there has been aTCOE sea change. While only 35% of North American organizations(in line with the global average) have no plans to implement a TCOEin the near future, 30% plan to implement an internally managedTCOE in the next two years, and a further 19% plan to use third-partyTCOE capability.One in 10 organizations is still completing the implementation oftheir TCOE started in 2011, but only a minority have actually madethe full transition. Some 50% of respondents require a TCOE toinclude Program Management Office services, 42% functionaltesting, and 41% training/knowledge management. The key benefitsorganizations expect from a TCOE include reduction in time-to-market, better resource management, improved quality, and, to alesser degree, cost reductions. It is no surprise that cost reductionis no longer one of the top three reasons for moving to a TCOE, asNorth America has led the rest of the world in leveraging offshoretesting resources.The commitment to testing is demonstrated by the allocation ofNorth American IT budgets to testing. At 18%, it is on a par with theglobal average, but more encouraging is that 40% of this budget isallocated to innovation/new transformational projects, and 63%(higher than the 59% global average) to customer-facing projects.The drive for improved customer experience is clear.Some 44% of organizations also report that the proportion ofthe IT budget allocated to testing has increased at least slightly,and half say that by 2015 the proportion allocated to testing willhave increased. Only 6% of organizations report that the testingbudget is static or declining.North American respondents report the greatest increase ininvestment in existing testing tools (60%) and outsourced resources,confirming the 2011 trend in which more outsourced activitywas planned. These focus areas for investment are consistentthrough 2015.It is highly encouraging that 89% of respondents agree that they areconsistently measuring QA metrics across the entire organization,but they also list metrics as one of their greatest challenges.Companies have standardized their QA metrics reporting;however, they still struggle with how to clearly communicate to thebusiness the status of testing efforts. Only 20% of North Americanorganizations use industry-standard estimation methods, and 40%use an in-house developed estimation. Not surprisingly, 64% oforganizations automatically collect and share QA metrics via Excel,while 43% also use default QA management tools.Test automation continues to be a key area to improve time-to-marketand meet deadlines. Across the board, NA respondents claim to have36% of their test cases automated, and believe that 42% automationis a realistic goal. Measuring automation percentages can bemisleading and does not tell the true value of automation. In our view,determining ROI is the only way to really understand its value.Much has been written about the cloud, and, certainly, testing isleading the way in exploiting its value. North American respondentsreport that 28% of testing is currently cloud based. The trend ofexploiting the cloud to use or establish testing environments, andhost or use pay-per-use testing tools, fits with the call for on-demandtesting resources. Organizations expect to see this trend continue,and estimate that 40% of testing will be cloud based by 2015.North American respondents also report that 23% of theirapplications have been hosted in the cloud, with a further riseto 34% expected by 2015. This commitment to the cloud is alsoreflected in the increased use of the Software as a Service (SaaS)model. In 2012, 33% of software licenses are via SaaS and this isexpected to rise to 46% by 2015, higher than the global average. Itis clear that organizations will look to use the cloud for better costmanagement, increased flexibility, and IT alignment with the needsof the business.Testing as a Service (TaaS) from a third-party provider is also set tomake a major impact on the market in the coming 24 months, with58% of respondents planning to sign up for a bundled QA service inthe next 12 months. The desire to reduce costs, improve resourcemanagement, and reduce time-to-market is the key driver.The drive to rapidly develop high quality mobile applications isimpacting all organizations, yet we found that only 25% of NorthAmerican respondents test any mobile applications or devices.The key reasons cited are lack of the right tools, readily availabledevices, and appropriate testing methods/processes. An interestingobservation is that users expect robust functionality, but that ourrespondents’ major concern is stability and performance of theapplication, regardless of location.In our opinion, outsourcing mobile testing should be considered byNorth American organizations, as they need to focus on increasingthe quality, scale, and professionalism of mobile application anddevice testing, or consumer satisfaction will suffer.Much has certainly changed in the QA world in North America sinceour last survey, in particular optimizing the advantages of TaaS,SaaS, cloud and mobile applications, and the trend for TCOEs.The need to bring quality applications faster to market was nevermore evident in this highly competitive and demanding economy.Flexibility, responsiveness, reduced cost, and time-to-market arecore issues, and those organizations that can harness the balancebetween internal management and external sourced resources aresure to be in the strongest positions when the economy moves backinto real growth. WORLD QUALITY REPORT 2012-13 71Country and Regional AnalysisNorth America
    • Top TrendsÊÊ 19% of UK organizations currentlytest mobile applications or devices.ÊÊ More than 83% of organizationsintend to use Testing as a Service(TaaS) offered by a third party.ÊÊ Mobile application and device testingis a growing challenge.In 2011, we reported that fundamental changes were takingplace in the UK, but we could not have predicted the double-diprecession that hit the UK over the last 12 months. Inflationarypressures, austerity measures, unemployment, and the Eurocrisis that threatens to engulf the UK are all contributing to anunsettled market.But even in this economic uncertainty, UK corporations areconserving cash, looking at emerging markets for growth, and drivingout costs. So investment by the private sector in IT transformationprojects, and in technology in general, continues as firms seekto improve productivity, do more with less, and embrace themobile revolution.IT services companies, including those in the Quality Assurance (QA)sector, find that customers are demanding more of their providers:experienced project delivery but lower day rates, acceptance of morerisk, and in-built transformation – factors that are supported by ourUK findings.The UK QA/testing market is, by all definitions, mature – asevidenced by the fact that only 45% of organizations use an in-houseengagement model for testers, compared with 51% globally. Some18% of UK organizations (compared with 13% globally) use anexternal managed service where the external vendor owns thetesting delivery.It is surprising that only one in five UK organizations rates the qualityof their internal testing resources as above average, indicating,perhaps, that the outsourced route could enable firms to uplifttheir capabilities. Indeed, the quality of external testers is seento be superior, with 35% saying that the quality of the testers isabove average.Testing function structures vary greatly in the UK. Some 45%of respondents use a combination of centralized/decentralizedteams, thereby running the risk of some loss of overall integrationUnitedKingdomBy Brian SheaCEOSogeti UK Sunil MunsifVice President – TestingCapgemini UK72 WORLD QUALITY REPORT 2012-13Country and Regional AnalysisUnited Kingdom
    • and management. A quarter of organizations have no separatetesting streams, and 21% report highly decentralized teams acrossdifferent business units.In 2011, we saw the concept of the Testing Center of Excellence(TCOE) come of age, with 56% of respondents indicating that aTCOE would be developed in the next two years, and 37% reportingthey were already building and managing this internally. As afurther sign of UK market maturity in 2012, almost one in 10 TCOEsis fully operational, 13% of organizations have started to moveto a TCOE in the last two years, 25% plan to develop internallymanaged TCOEs in the next two years, while 38% of organizationsstill have no plans for a TCOE. By 2015, 62% of organizations willhave moved to a TCOE, including Program Management Officeservices, training/knowledge management, performance, andfunctional testing. The greatest benefits that UK respondents seekfrom this approach include reduced time-to-market and betterresource management.UK respondents report that 18% of the total IT budget is allocatedto testing, in line with the rest of the countries. Some may thinkthis is low, but IT budgets have been increasing year on year, evenin economically troubled times, so the real investment in testingis also growing. It is encouraging that 42% of the budget is for newtransformational work rather than maintenance, and equally that58% is invested in testing customer-facing applications.UK organizations also report that the two main QA areas of greatestincrease in spending are internal professional testing resources andexisting testing tools. In 2015, the focus is set to move to externaloutsourced resources and the test environment/infrastructure.The commitment to the testing function is also set to increase, as46% of organizations report that the percentage of the IT budgetallocated to the testing function has increased in the last year andfully 57% report that by 2015 the budget will have increased.UK market maturity is further demonstrated in the quantificationof quality. As in 2011, three-quarters of 2012 respondentsindicate that they consistently measure QA metrics across theentire company. But there is room for improvement. Some 16% oftest data is spreadsheet based and manually generated, and only8% of respondents use purchased automation tools.In 2011, we saw the move towards Software as a Service (SaaS)models and, more generally, a focus on cloud-based activities.This year sees a greater commitment to use the cloud – but onlyto a certain level. In the UK, on average, respondents say 25% oftesting is currently in a cloud-based environment – below the overallstudy average of 28%. But by 2015, the level will have risen to 40%,perhaps a natural cap, because it seems that organizations thathave not made the move to cloud-based testing by 2015 will beunlikely to move at all.Testing as a Service (TaaS) offered by a third party looks set formajor growth in the next two years. Currently, only one in 10 UKorganizations uses TaaS, but 45% will move down that road in thenext 12 months and a further 38% in 24 months. Only 5% of UKorganizations have no TaaS plans at all. The drivers for this arereported to be to reduce cost and time-to-market, and improvedresource management. Challenges with TaaS, mentioned by onlya few organizations, center on loss of control of the total testingspend, scalability, and security, but none was mentioned by morethan 25% of respondents.UK organizations estimate that 22% of applications are hosted inthe cloud now (in line with the global percentage) and that by 2015this is expected to rise to 32%. This year highlights a greater use ofSaaS compared to last year, when SaaS models were basically seenas pay-as-you-use software and there was no discernible differencebetween the number of organizations wanting to move further intoSaaS and into the cloud. This year, fully 30% of QA/testing softwarelicenses use the SaaS model, and respondents say this will grow to44% by 2015.It is surprising that only 19% of UK organizations (compared with31% globally) say they currently test any mobile applicationsor devices. The biggest issues faced are said to be lack of the righttools and availability of devices, and current testing is focusedon efficiency/performance, portability, and, to a lesser degree,functionality.If UK organizations were to outsource mobile testing, their reasonswould be suppliers’ capacity to test applications on a large rangeof devices, and consequent ability to refocus their internal teamson development.Looking ahead, there will be increased focus on testing qualityespecially because mobility will change the way IT functions withinmany organizations, and, while security is always a concern, mobilityis driving consumer experience, functionality, and performance.As in 2011, viewing the findings together, we can infer that UKorganizations clearly understand the benefit of investing inquality and using their resources to the full, but, at the sametime, recognize the need to reduce time-to-market and focuson customer-facing activities. The market has certainly movedbeyond a simple cost-reduction focus as part of its ongoingmaturity, and the increasing levels of investment and the linkingof testing excellence to success in the business serve to underlinethis conclusion. WORLD QUALITY REPORT 2012-13 73Country and Regional AnalysisUnited Kingdom
    • About the StudyThe 2012-2013 World Quality Report is based on a total of 1,553 detailedtelephone interviews undertaken during April and May 2012, with seniorexecutives from a range of IT and business-related functions in medium andlarge private companies, government and public sector organizations, across25 countries. This data was augmented by in-depth client interviews, and thenanalysis and commentary carried out by our own specialists.74 WORLD QUALITY REPORT 2012-13About the Study
    • COUNTRY NUMBER OFRESPONDENTSSIZE OF COMPANY10,000+ 5,000-9,999 1,000-4,999USA 296 119 99 78CANADA 25 10 9 6BRAZIL 87 36 29 22FRANCE 150 60 53 37GERMANY 135 50 48 37NETHERLANDS 126 50 44 32BELGIUM AND LUXEMBOURG 25 10 8 7SWITZERLAND 27 10 10 7UK 125 49 44 32IRELAND 25 10 9 6SWEDEN 85 33 31 21NORWAY 30 13 9 8DENMARK 31 12 9 10FINLAND 30 12 11 7CZECH REPUBLIC 25 7 6 12POLAND 22 7 9 6HUNGARY 20 8 7 5ITALY 20 8 7 5SPAIN 20 8 7 5PORTUGAL 20 8 7 5AUSTRALIA 90 32 31 27NEW ZEALAND 11 7 4 0CHINA 103 40 35 28HONG KONG 12 5 4 3SINGAPORE 13 6 4 3TOTAL 1553 610 534 409Interviews by country and size of company FIGURE 19Base: 1553 RespondentsSurvey sampleThis year we focused on companies and public sectororganizations with more than 1,000 employees, andthe detailed breakdown of the number of interviewsby size is shown in the table above. It should be notedthat a conscious effort was made to ensure that arepresentative distribution of interviews for each sizeof company was covered in each country. WORLD QUALITY REPORT 2012-13 75About the Study
    • 6046315017632187101128EASTERN EUROPESOUTHERN EUROPE SOUTH AMERICANORDICSWESTERN EUROPEAUSTRALIA AND NEW ZEALANDNORTH AMERICA ASIAUK AND IRELANDBase: 1553 RespondentsINTERVIEWS BY REGION FIGURE 2067A total of 25 countries participated in thesurvey. The total number of interviewsaggregated by region is shown below.76 WORLD QUALITY REPORT 2012-13About the Study
    • INTERVIEWS BY SECTOR FIGURE 21TRANSPORTATIONENERGY AND UTILITIESFINANCIAL SERVICESMANUFACTURINGCONSUMER PRODUCTS,RETAIL, AND DISTRIBUTIONHIGH TECHHEALTH AND LIFESERVICESTELECOMS, MEDIA, ANDENTERTAINMENTPUBLIC SECTORBase: 1553 Respondents1553TOTAL RESPONDENTS88888836694881652663101553TOTAL RESPONDENTSBase: 1553 RespondentsINTERVIEWS BY JOB TITLE FIGURE 22IT DIRECTORSVP APPLICATIONSQA/TESTING MANAGERSCIO’S533391297332In addition, responses were required fromcore vertical sectors with the number ofinterviews conducted shown in the graphon the right, which provides great insightinto the test and QA issues within eachindustry vertical. This also allows us tocompare and contrast the views sectorby sector.Finally, additional controls were usedto ensure that a representative range ofopinions and views was obtained duringthe survey.Questionnaire and methodologyThe survey questionnaire was devised byour QA and testing experts in the threesponsoring organizations involved incommissioning the research: Capgemini,Sogeti, and HP, in consultation withColeman-Parkes Research. All researchwas executed using a telephone‑basedapproach, rather than an onlinequestionnaire as used last year. Themove to telephone-based research waslargely driven by the change in the size ofcompanies being covered.The 40-question survey covered a rangeof QA and testing subjects enriched by thecollection of qualitative data in additionalin-depth interviews, which are included forthe first time as quotations in the report.During the telephone interviews, surveyquestions were posed based on their relevancyto the respondent’s job title. For this reason, thebase number of respondents for each surveyquestion shown in the graphs is not always thefull 1,553 sample size. WORLD QUALITY REPORT 2012-13 77About the Study
    • about Capgeminiand SogetiWith more than 120,000 people in 40 countries,Capgemini is one of the world’s foremost providersof consulting, technology, and outsourcing services.The Group reported 2011 global revenues ofEUR 9.7 billion. Together with its clients, Capgeminicreates and delivers business and technologysolutions that fit their needs and drive the resultsthey want. A deeply multicultural organization,Capgemini has developed its own way of working, theCollaborative Business Experience™, and draws onRightshore ®, its worldwide delivery model.Sogeti is a wholly-owned subsidiary of Cap Gemini S.A.,providing local professional services. Sogeti bringstogether more than 20 000 professionals in 15 countriesand is present in over 200 locations in Europe, the USand India.Together Capgemini and Sogeti have developedinnovative, business-driven quality assurance (QA)and testing services, combining best-in-breed testingmethodologies (TMap®and TPI®) and the globaldelivery model, Rightshore®, to help organizationsachieve their testing and QA goals. The CapgeminiGroup has created one of the largest dedicatedtesting practices in the world, with over 9,500 testprofessionals and a further 14,500 applicationspecialists, notably through a common center ofexcellence with testing specialists developed in India.Learn more about us atwww.capgemini.com/testing orwww.sogeti.com/testingAbout HPAt HP, we don’t just believe in the power of technology.We believe in the power of people when technology worksfor them. To help you create. To make the digital tangible.To harness the power of human information. At HP, wework to make what you do matter even more.Our Business Technology Optimization (BTO)products, along with our new and complete approachto Application Lifecycle Management (ALM), help ourcustomers to achieve better business outcomes.More information about HP is available atwww.hp.com78 WORLD QUALITY REPORT 2012-13about the sponsors
    • THANK YOUWe would like to thank all the respondents who provided in-market experience,expert opinion, and input into this year’s survey. In accordance with the UK’sMarket Research Society Code of Conduct (under which this survey was carriedout), all respondents are anonymous, but we appreciate the time that eachinterviewee provided on the call with our team.Our thanks also go to the many subject matter experts within the Capgemini,Sogeti, and HP teams, who provide invaluable local insight and input into thedevelopment of the sector and region sections of the report.We would like to thank the team of collaborators who helped create thisyear’s report.Contributors: The global business leaders and subject matter experts whoprovided valuable insight into their areas of expertise and market experienceMain Section Authors: Murat Aksu and Charlie LiProject Management: Grace ChanCreative Design: Liz TuttlePartner Management: Erwin Anderson-Smith, Alain Mey, Thomas VeltmanCore Project Team: Clare Argent, Stephen Davis, Kristin O’HerlihyMarket Research: Matt Gray and Ian Parkes (Coleman Parkes Research)*infographics: The SurgeryProofreading: Eleanor PatrickPrint and Distribution: David Cole (Crucial Colour)*Ian Parkes, CEO and a founder of Coleman Parkes Research, is a full memberof the Market Research Society, and all work conducted by Coleman ParkesResearch is conducted in compliance with the strict code of conduct andguidelines set out by the MRS in the UK, as well as the legal obligations underthe Data Protection Act 1998.©2012 Capgemini, Sogeti, and HP. All rights reserved.Rightshore®is a trademark belonging to Capgemini.TMap®, TMap NExT®, TPI®and TPI NExT®are registeredtrademarks of Sogeti, part of the Capgemini Group.
    • CAPGEMINIMurat AksuVice-President, Global AllianceExecutive for HPmurat.aksu@capgemini.comCharlie LiVice-PresidentGlobal Testing Servicescharlie.li@capgemini.comGovindarajan MuthukrishnanVice–President, Testing Practice Lead,North America, UK, APAC and Financial Servicesgovindarajan.muthukrishnan@capgemini.comSOGETIStefan GerstnerVice-President, Global Testing Servicesstefan.gerstner@sogeti.comHPAlain MeyGlobal Alliance Directoralain.mey@hp.comYou can also reach us by sending an email to:worldqualityreport.s.uk@capgemini.comor worldqualityreport.s.uk@sogeti.com