SwissQ Testing Trends &Benchmarks Switzerland 2012Software Testing Trends Switzerland 2012. Where are we now – where are w...
TABLE OF CONTENTS             SwissQ Testing Trends & Benchmarks 2012 2        3   EDITORIAL        4   TRENDWAVE 2012    ...
EDITORIAL                                                                                                                 ...
TRENDWAVE 2012                                                                                                            ...
TRENDWAVE 2012                                                                                                            ...
KEY MESSAGES                                                                                      SwissQ Testing Trends & ...
PROJECTS                                                                                                                  ...
MATURITY                                                                                                                  ...
ORGANISATION                                                                                                              ...
ORGANISATION                                                                                                              ...
PROCEDURES                                                                                                          SwissQ...
TESTING TOOLS                                                                                                             ...
FRAME OF SURVEY                                                                                                           ...
SWISSQ REQUIREMENTS ENGINEERING                                                                                           ...
SWISSQ AGILE TRENDS  BENCHMARKS REPORT 2012                                                                               ...
ABOUT USSwissQ supports its clients in the development and implementation of IT-solutions andassures that the end users ge...
Upcoming SlideShare
Loading in …5

SwissQ Testing Trends & Benchmarks 2012 (Englisch)


Published on

Zum vierten Mal in Folge hat SwissQ zum Jahresende bei der Schweizer IT Community die aktuellen Trends und Benchmarks erfragt. Daraus entstanden ein repräsentativer Überblick zum Stand des Software-Testings in der Schweiz im Jahr 2012 und ein Ausblick auf die wichtigsten Trends der kommenden Zeit.

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SwissQ Testing Trends & Benchmarks 2012 (Englisch)

  1. 1. SwissQ Testing Trends &Benchmarks Switzerland 2012Software Testing Trends Switzerland 2012. Where are we now – where are we going to?
  3. 3. EDITORIAL SwissQ Testing Trends & Benchmarks 2012 3For the fourth time in a row, SwissQ questioned the Swiss IT community about current trends and benchmarks. Not only did we receiveoverwhelming feedback from over 300 participants, we also personally interviewed many IT decision makers from various companies,segments, and regions about the current testing trends. The outcome is a representative overview of the most important trends. For the first time this year we are not only presenting a trends and benchmarking analysis of Software Testing, but are also taking a lookat Requirements Engineering and Agile. The corresponding reports will be published separately to highlight their importance.The group of participants mainly included test managers and line managers, but The same as last year, SwissQ offers companies the opportunity to present thealso test engineers and project leaders working in the area of IT, finance, insu- SwissQ Testing Trends & Benchmarks to their employees. The target audiencerance, manufacturing, or in public companies. For the first time this year we put consists of members of the testing community, software engineers, project leaders,a special focus on Agile and Requirements Engineering and have thus enhanced requirements engineers, and the management. The presentation allows you tothe group of participants accordingly. A quarter of the participants of this survey, take an outside look at the current changes, as well as to boost the motivation fortherefore, work in Requirements Engineering. In the end, these 300+ question- and significance of testing, and profit from ideas for further developments.naires served as the basis for the analysis and interpretation of the current statein the year 2012 and the future development. Having said this, we hope that the present Trends & Benchmarks will inspire you to actively pursue new challenges and take take the actions that best suit yourThe benchmarks depicted in a multitude of informative charts and diagrams form business.the backbone of this report and allow you to position your company in comparisonto others.As before, we use the SwissQ Trend Wave® to show the evolution of the maintopics. It shows in four phases how single trends will most likely develop over timeand in turn allows you to appraise the influence of the trends on your business.
  4. 4. TRENDWAVE 2012 SwissQ Testing Trends & Benchmarks 2012 4 INTRODUCTION GROWTH MATURITY DECLINEPRIORITY Automated Test Process Improvement Regression Tests ALM Tools SCRUM Agile Testing Regression Tests Load and Performance Tests Central Test Organisation Explorative Testing Risk Based Testing ISTQB FL & AL Level TDD Security & Penetration Testing Outsourcing ISTQB Expert Level Session Based Testing Mobile Testing Cloud Testing ATDD TIME INTRODUCTION – This topic has been GROWTH – This topic is more and MATURITY – Most companies are DECLINE – The topic has already identified and some companies are more accepted and many companies working on the implementation been implemented by most of the deploying initial implementations. are considering it. The first tools are or have already completed it. companies, with the exception of However, it cannot be foreseen being developed and consultancy The knowledge of this topic is individual latecomers. Often, there whether this trend will positively firms offer services for the same. often widespread, resulting in is no more added value in acquiring advance and whether testing will Often risks are associated due to sub-topics being raised. further knowledge in these areas, be considerably influenced. limited implementation experience. since it will become obsolete shortly.
  5. 5. TRENDWAVE 2012 SwissQ Testing Trends & Benchmarks 2012 5The aim of the trendwave is to show what topics will influence the testing discipline as well as the people and the organisationsdealing with it. It thus helps readers to specifically prepare for, and actively participate in these changes.The technologies that changed the consumer market in the last couple of years – The current focus remains on two areas that have continued to gain significancesmart phones and tablets, social media, cloud, and highly topical, voice control over the past few years, one of them being Agile Testing as most of the partici-– are increasingly being used in the business world as well. The new information pants of this survey are at least partly working with agile methods. The secondand communication channels accompanying this development lead to higher area is Requirements Engineering which is also directly involved in one of theconsumer expectations as the following example illustrates: biggest challenges in testing: the generally poor quality of requirements. Be it coincidence or fate, the parallel development of these two subjects can only be of advantage to us.During the après-ski break, we started a heated argument about the need ofa separate ski theft insurance. I got my smart phone and opened the app of my We can therefore assume that Mobile Testing and especially Cloud Testing for manyinsurance company to check my current coverage. I was startled to realize that businesses and software testers will only appear on the radar once the biggestI wasn‘t insured against ski theft! I immediately asked Siri to find a provider and obstacles in the transition to Agile Testing and the improvement of the quality oftook out a policy online. I was lucky I did that because I just saw a guy in a yellow requirements are overcome. Furthermore, test automation – still not on aski suit taking off with my skis! satisfying level – also remains in the focus. Still, it is conceivable that the “mobilization“ of IT will increase rapidly in theSuch seemingly utopian scenarios are on the verge of their break through today. near future, so the question to pose is: shouldn‘t we jump on the bandwagonThey are not only challenging business analysts and software engineers, but tes- right now before catching up will be too cost-intensive and time-consuming?ting as well will become a more complex and costly endeavour.Many companies have recognized these trends and are already including them intheir current projects and products, others are gaining experience and building upmore knowledge. Most businesses though still think that it is too early to deal withthe subject but should be careful not to miss the opportunity to catch up on it.
  6. 6. KEY MESSAGES SwissQ Testing Trends & Benchmarks 2012 6 1 More than half of the companies apply agile development practices. Thereby 84.5 % of the respondents use Scrum as their preferred agile method. 2 New technology and process driven processes like mobile testing, agile testing or session based test management are (still) getting little attention. 3 The focus of education and training has shifted towards requirements engineering and agile, closely followed by the ISTQB advanced level certifications. 4 While 40% of the respondents rate the maturity level of the testing process as good or excellent, 75 % of the respondents rate the maturity level of the requirements engineering 5 Not surprisingly, poor requirements are mentioned as the biggest challenges in software testing, Followed by late involvement of the testers. 6 The interdisciplinary collabora- tion between business analysts, software engineers, and testers gained further importance. But the significant mind shift hasn’t as medium or less. happened yet. 7 The most investments are made in test automation or the creation and extension of regression tests. In contrast there are just marginal investments in hiring of new 8 Even though 80% of all participants automate their functional tests, almost half of them automated less than 10 % of all functional tests. This is a step forward though, as 9 The trend towards integrated tool solutions has increased, parti- cularly thanks to HP ALM and partly MS TFS. Tools covering only one discipline find themselves testing employees. last year almost 2/3 of the in a difficult position. participants had automated less than 10 %.
  7. 7. PROJECTS SwissQ Testing Trends & Benchmarks 2012 7Project Type70 % of all projects are new developments or updates of existing software. >50% of the respondents describe the starting situation of their 12 % projects as satisfying or insufficient related to 8% New development Estimation 39 % Update of existing software Planning 10 % Migration Definition of requirements Implementation of standard Realistic expectations software Operation, support, maintainance, 31 % re-design, ... Project Outcome Just over a third of all projects are finished with the expected functionality, within the expected timeframe and within budget.Project Size (in Swiss Francs) 40 % 51 % 35.1 % 30 % 40 % 39.2 % 39.2 % 25.1 % 20 % 18.1 % 17.5 % 20 % 10 % 4.1 % 10.8 % 0 % Project Proj. finished Proj. finished Project Project finished in time, with budget with major extended / stopped 0 % budget and and / or time functional rescheduled up to 1 Mio up to 20 Mio more than 20 Mio functionality overruns changes
  8. 8. MATURITY SwissQ Testing Trends & Benchmarks 2012 8Maturity of Test Process Reputation of TestingIn average the respondents rate the maturity of their companies testingprocess as medium or good. This rating has remained constant in the lastyears. It’s a contrast to the maturity in requirements engineering (RE) Testing is strategic for 18 %which is rated substantially worse. company success50 % Testing is important for 49 % reliable software40 % Testing is a necessary evil 19 % Testing has low priority 13 %30 % Testing RE Testing is unnecessary, we can 2 % save the costs for testing20 % 0 % 10 % 20 % 30 % 40 % 50 %10 % 0 % The Biggest Challenges ea k ea k um od nt w W i Go lle ry ed ce Poor Availability Ve M Ex Requirements of test environment Testers are involved 63 % 30 % Auto- too late mation 38 % 40% 75% 33 % Not enough of the respondents rate the of the respondents rate budget / resources maturity of the testing process the maturity of the as good or excellent. requirements engineering 30 % as medium or less.
  9. 9. ORGANISATION SwissQ Testing Trends & Benchmarks 2012 9Who Performs Testing Activities Education The focus of the employees training is moving from ISTQB certificationTest Case Design – which most of them already have – to the topics of requirements engineering and agile. I already have it It is planned In the not too distant future IT Tester ISTQB Foundation Level 36 % Req. Engineer / ISTQB Advanced Level Business Test Manager Developer Analyst IT Tester End User ISTQB Advanced Level 21 % 16 % Test Analyst 19 % Developer ISTQB Advanced Level 7.4 % Technical Test Analyst IREB® CPRE Foundation Level Project Management (IPMA, PMI, ...)Test Execution Certified Scrum Master Agile Testing Certified Product Owner Certified IT Process and Quality Manager IT Tester Agile Requirements Engineering 34 % End User 26% ISTQB Expert Level Developer Req. IREB CPRE Advanced Level IT Tester Engineer/BA Elicitation & Consolidation 17% 14% Developer IIBA CBAP (Certified Business 9 % Analysis Professional) IREB CPRE Advanced Level Requirements Modeling 0 % 20 % 40 % 60 % 80 % 100 %
  10. 10. ORGANISATION SwissQ Testing Trends & Benchmarks 2012 10Use of External Resources Testing EffortThe use of external resources in testing slightly increased compared The variance of the testing effort is still wide. By average the testing effort into 2011. The external resources are mostly used in-house. proportion to the total effort is about 15 % to 20%. In proportion to the deve-Outsourced testing stagnates compared to Managed Service lopment effort this value is more than 50 % (not represented in the graph).(e.g. automated regression tests) which have expanded. 30 % 5.9 % 30 % 10,4 % No external resources 33.8 % Acquisition of external resources 20 % 21 % (Insourcing) 19 % Acquisition as a service 17 % (Managed Service) 50 % Outsourcing to external resources 10 % 8 % 4 % 1 % 0 % < 5 % 5 – 10 % 10 – 15 % 15 – 20 % 20 – 30 % 30 – 50 % above Testing effort in proportion to total effortInvestments Investments are Investments remain Investments are increasing constant decreasing Test automation Test environments Outsourcing testing activities Test management tools Hiring new testing employees Creation and extension of Test data management regression tests Training and education Structured testing process / methods
  11. 11. PROCEDURES SwissQ Testing Trends Benchmarks 2012 11 Software Development Process Implementation of Agile Methods More than half of the companies use agile processes. Many of them are using a combination of different development processes e.g. agile Drivers of Agile Methods in combination with waterfall. Agile processes are mostly promoted by the directly involved people. 60 % 39.2 % 52.1 % Agile 40 % 39.4 % 38.0 % 36.6 % 51 % Waterfall 10 % 19.8 % 40 % 0 % Developer Departement Head Project CEO Iterative mgr. / Team development Manager Hermes lead 22 % RUP 12 % 16 % Reasons for Agile Methods 84.5% Better handling Accelerate Time-to-Market of the respondents use Scrum as their favorite agile method. of changing priorities ProductivitySwissQ Agile Trends Benchmarks 2012 Collaboration betweenThere are many additional charts and details in the IT and businessSwissQ Agile Trends Benchmarks Report 2012.
  12. 12. TESTING TOOLS SwissQ Testing Trends Benchmarks 2012 12 Test Management Tools Test Automation More than 60 % of the respondents work with HP QC (since version 11 the name is changed to ALM). Many of them are working with more than one Tools used 80% tool, mostly in combination with MS Excel or Word. MS Test Manager has a The market leader by far is HP Quick Test Professional, followed growing share, due to its tight integration with MS Team Foundation Server. by proprietary tools. All in all the market is highly fragmented. Almost 30 tools were mentioned. of the respondents automate their functional tests. HP QC / ALM 62 % Selenium MS Office 44 % Tosca Canoo Webtest Proprietary Tools Ranorex Lisa 48% of the respondents have Open Source 20 % Quick Test Professional automated less than 10 % of their functional tests. Rational Functional Tester In 2010, the value was 60 %. QF Test OthersProprietary developement 8 %MS TFS 2010 Test Manager 5 % Cost Savings by Test Automation Most of the respondents expect low savings by test automation. Inflectra Spira 2 % In one third of the cases giving a statement to this question was not possible.IBM Rational TestManager 2 % 33.3 % No tool 1 % 22.6 % 23.7 % 0 % 20 % 40 % 60 % 10.2 % 2.8 % 7.3 % Costs up to 10 % up to 20 % up to 50 % up to 80 % No increased statement possible
  13. 13. FRAME OF SURVEY SwissQ Testing Trends Benchmarks 2012 13Industrial Sector ResponsibilitiesMore than 60 % of the respondents work either in the IT or in the More than 50% of the respondents describe their job with more thanfinancial sector. Compared to the last years their proportion has decreased, one role. Especially test managers don’t work 100% as test managers,demonstrating that the subject has arrived in other industries too. but also take responsibility for other roles. IT 36.1 % 30 % Finance, Insurance 28.4 % Manufacturing 7.4 %Public and semi-public companies 7.4 % 20 % Traffic and Transportation 5.6 % Telecommunication 4.0 % MedTech 3.7 % 10 % Others 7.4 % 0 % 10 % 20 % 30 % 40 % 0 %IT Employees ge r na ana er r A ge / B nee age r r Te st er na ge r ne er a e n gi a n a ngiA bit more than half of the respondents work in companies with more t M n M gin st E ct M s M re E es io n e e nt athan 500 IT employees. T is ts E T oj e iv Pr m ftw / D men u ire So t q en uire Re rtm eq pa R 2001– ... 33.0 % De 501 – 2000 17.6 % 60% 33% 251 – 500 13.6 % 51 – 250 15.4 % 11 – 50 14.2 % of the respondents mainly of the respondents are 1 – 10 6.2 % work in projects. line managers. 0 % 5 % 10 % 15 % 20 % 25 % 30 % 35 %
  14. 14. SWISSQ REQUIREMENTS ENGINEERING SwissQ Testing Trends Benchmarks 2012 14 TRENDS BENCHMARKS REPORT 2012 Do you want to know more? You can download the detailed report with further analyses about requirements engineering from SwissQ Requirements Engineering Trends Benchmarks 2012 There are many additional charts and details in the SwissQ Requirements Engineering Trends Benchmarks Report 2012. Future Investments Investments Investments remain Investments increase constant decrease Education and Training for Employees Better Cooperation 33 % 54 % 13 % 80% 36% of the projects check their do not check requirements of Business and IT 33 % 53 % 14 % requirements for operational for their need. Standardisation of the accuracy, implementability, internal RE-Processes 25 % 61 % 14 % and completeness. Elaboration / Definition of the RE-Role 24 % 60 % 16 % Development of Templates and Guidelines 22 % 61 % 17 % 2/3 Hiring new RE-Employees 22 % 55 % 23 % Establishing specific RE Tools 21 % 64 % 15 % of the respondents Establishing internal allocate less than 1 dayRE-Divisions/-Departments 17 % 63 % 20 % for stakeholder analysis. Outsourcing RE-Activities 11 % 48 % 41 % 0 % 20 % 40 % 60 % 80 % 100 %
  15. 15. SWISSQ AGILE TRENDS BENCHMARKS REPORT 2012 SwissQ Testing Trends Benchmarks 2012 15Do you want to know more? You can download the detailed report with further analyses about agile trends from SwissQ Agile Trends Benchmarks 2012 There are many additional charts and details in the SwissQ Agile Trends Benchmarks Report 2012. Main Reasons for the Failure of Agile Projects Only 40% Lacking Experience with agile Methods 52 % Corporate Philosophy is not connectable with agile Values 45 % are content with the implementation of their External Pressure to follow a agile approach. traditional Approach 41 % Lacking Support of the Management 38 % Lacking / Insufficient Training / Coaching 36 % 1/3 Lacking Interconnections between organisational units 35 % of the respondents have Lacking Team Motivation 22 % 2 or more years experience with agile methods. Others 12 % 0 % 10 % 20 % 30 % 40 % 50 % 60 %
  16. 16. ABOUT USSwissQ supports its clients in the development and implementation of IT-solutions andassures that the end users get the functionality they really need. This is achieved byunambiguously determining requirements and risk-based testing the implementation.Our vision is to improve the added value of IT through requirements management andsoftware testing. Along with providing high-quality services, we pursue this visionby establishing independent platforms, like the Swiss Testing Day and the SwissRequirements Day, which facilitate the exchange of know-how and experiences.In addition to that we help bright minds to expand their knowledge in our trainings. © by SwissQ Consulting AG | Stadthaus-Quai 15 | Switzerland-8001 Zürich | | Phone +41 43 288 88 40 | Fax +41 43 288 88 39 Twitter: @SwissQ | Facebook: swissqconsulting