The State of Testing 2017
•Download as PPTX, PDF•
0 likes•297 views
Curious to know what 3,400 software developers and QA experts think about the current and future state of the software testing industry? Whether you’re a developer, QA engineer, manager, or any other role working in the software industry, SmartBear’s 2017 State of Testing Survey Report will provide you an exciting deep-dive look into industry trends in software testing, team dynamics, development models, and outlooks on the future of software testing.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The State of Testing 2017
Similar to The State of Testing 2017 (20)
More from SmartBear
More from SmartBear (20)
Recently uploaded
Recently uploaded (20)
The State of Testing 2017
- 2. Webinar Hosts • Harsh Upreti , Product Marketing Manager, API • Gregory Hanson, Manager of Sales Engineering • Jeffrey Martin, Product Owner State of Testing 2017
- 3. Agenda • Survey Methodology • GeneralTestingTrends • Test Automation • Test Processes • Test Environments • Future of TestAutomation State of Testing 2017
- 4. Page Survey Methodology SmartBear conducted a global online survey from June to July 2017 and collected responses through the following channels: • Emails to an internal database of QA professionals, Developers, Managers, Business Analysts and more • Online advertisements & social media • SmartBear customers • In collaboration with SmartBear partners and influential industry groups 4 State of Testing 2017
- 5. General Trends State of Testing 2017
- 6. Page Proprietary & Confidential 6 State of Testing 2017 What Testing Professionals are Saying
- 7. Page Proprietary & Confidential 7 State of Testing 2017 What Testing Professionals are Saying “Testing will be considered more as development work.” Anonymous QA Engineer “More automation, less manual testing.” Anonymous QA Engineer
- 8. Page Proprietary & Confidential 8 State of Testing 2017 Testing is Beyond Frontend 63% respondents do both UI and Backend- API testing 14% 67% 19% QA Engineers API Both UI 37% 53% 10% Software Developers API Both UI 10% 62% 28% Manual Testers API Both UI
- 9. Page Proprietary & Confidential 9 State of Testing 2017 Testing Methodologies PercentageTeams Using aMethodology 72% Teams do some form of unit testing More than half of teams do performance testing Very few teams use service virtualization to reduce dependencies
- 10. Page Proprietary & Confidential • Majorityspendsmosttimetestingfunctionality • Veryfewrespondentsspendtimeoncreatingmocks andloadtesting 10 State of Testing 2017 API Testing Trends 82% 18%Less than 50 More than 50 • Majorityofrespondentshaveless than50endpointstotest. What API testing activity do you spend your most time on? How many unique APIs do you use in your daily testing/development?
- 11. Test Automation State of Testing 2017
- 12. Page Proprietary & Confidential 12 State of Testing 2017 Automation Overview Bigger companies have more than half of tests automated Company Size 44% Tests are Automated on Average
- 13. Page Proprietary & Confidential 13 State of Testing 2017 Scripting Proficiency is an Indicator of Automation • Beginner Scripting Skills – 11% tests automated • IntermediateScripting Skills– 47% tests automated • Expert Scripting Skills – 64% tests automated • Scripting proficiency correlateswith higher automation
- 14. Page Proprietary & Confidential 14 State of Testing 2017 Automation Potential by Industry Non Profit, Pharmaceuticals, Government/defense are sectors with good potential for automation Assumption: scripting skill is an indicator of automation Sectors where more than 40% of testers are proficient in scripting
- 15. Page Proprietary & Confidential 15 State of Testing 2017 Coding Languages Across entire survey, 61% responded theyuse Java. Higher number of API testers use Java, Python, Groovy and PHP Higher number of UI testers use C#, Visual Basic
- 16. Page Proprietary & Confidential 16 State of Testing 2017 Manual Testing 56%ofTestsarestillmanual ManualTestingmostprevalentinB2C,user-centricindustriesorwheretheapplicationundertestchanges frequently ExploratoryTesting,amanualtestingpracticeusedtochecksystemsontheflywithoutpreviouslydesigned testcases,isusedby45% Mostlikelytobeusedinmid-sizedcompanies(101-500employees)
- 17. Test Processes State of Testing 2017
- 18. Page Proprietary & Confidential 18 Processes Overview ‘It Depends’ for 23% of the teams 7% of the teams still do pure waterfall 63% of the teams claim to be Agile State of Testing 2017
- 19. Page Proprietary & Confidential 19 State of Testing 2017 Agile 63%of the respondents said that they wereagile.
- 20. Page Proprietary & Confidential 20 State of Testing 2017 Test Management Only practicedby 44% of respondent teams – and no realdifferencesacross teamsize
- 21. Page Proprietary & Confidential 21 State of Testing 2017 Release Cycles Bulk of the releases aredone weeklyand monthly, less releases everyquarter and veryfeweveryyear. More APIs are released than UI multiple times a day and weekly More UI projects are released than API quarterly
- 22. Test Environments State of Testing 2017
- 23. Page Proprietary & Confidential 23 State of Testing 2017 Platforms & Browsers for Testing Majority testers test on iPhone and AndroidMore IE users than Firefox
- 24. Page Proprietary & Confidential 24 State of Testing 2017 APIs are Mostly Internal • 72% of the respondents test against internal APIs • 16% respondents do nothing if the APIs are unavailable • 26% use scripts to create mocks, 22% use UI tools to create mocks • 17% had no idea something could be done for unavailable APIs
- 25. Future of Testing State of Testing 2017
- 26. Page Proprietary & Confidential 26 State of Testing 2017 Future of Test Automation 62% of allsurvey respondents think that the percent of automated testing will increase 0% 10% 20% 30% 40% 50% API UI Full Automated Testing Now 2 Years From Now 0% 10% 20% 30% 40% 50% 60% QA Engineers QA Managers Full Automated Testing Now 2 Years From Now
- 27. Page Proprietary & Confidential 27 State of Testing 2017 Conclusion Automationisgoingtoincreasedrasticallywithintwoyears Scriptingproficiencyandautomationhasadirectcorrelation AdoptionofAgilevarieswiththeteamsize Releasecycleshavegottenshorter ServiceVirtualizationiskeyandneedshigheradoptionintesting
- 28. Thank you State of Testing 2017 Download the full report here.
Editor's Notes
- Here are your hosts…. We will be recording the webinar, and for those of you in attendance, as well as for those who took the time to fill out the survey, we’ll be sending you the first copies of the full report at the end of the week.
- Its going to be an interactive presentation with lot of opportunities for you to participate through polling and questions. 1 . Look at the survey methodology: add credibility to the data and know that we are presenting insights from real testers who work in the same space as majority of th audience 2. General testing trends: Some very important high level insights that we got, how many API endpoints a tester is using? 3. Automation insights ruled the data, insights around automation in industries and what is the relationship between automation and scripting abilities? 4. Processes like Agile and release cycles ,how often is software being released and what seems to be the right team size for successful Agile? 5. Test environments : we will look at what testers are using to test their frontend, browsers, mobile platofrms etc and API environments. 6. Finally we will take a look at the future of test automation, what our participants believe is the future of automation.
- This survey was designed to establish benchmarks for the software testing industry regarding the ways in which teams are testing the APIs and Uis of their applications in 2017. Collected a total of 5,200 responses with 3,400 completed via email to an internal database of developers, testers, managers and more. Joined forces with SmartBear Partner Cigniti, and influential industry group Ministry of Testing in Boston Promoted the survey in-product to grab customers actively using our test tools And to the larger testing community via google adwords and social media including facebook, twitter, and linkedin
- Harsh: Participants in the survey got a chance to express themselves and provide open ended comments We tried to do some sentiment analysis and here are the results We asked: How do you think the software testing field will look in two years? Majority of the comments were around Automation tools -> shows the top of the mind thing for testers automation is the key term and is driving interest in processes and devops and impacting environments Development and coding is next on the top of mind of testers, more and more testing is being done in code, We found a link between coding proficiency and the extent of automation of tests
- Harsh: Testing activities are not just focused on either frontend and backend but as per the results majority of the professionals are doing both at the same time. 63% of QA professionals do both, UI testing : A larger proportion of manual testers are doing UI testing than QA engineers and Software Developers Software developers are doing very less UI testing and a very large proportion of them are doing API testing Manual testers are doing the lowest amount of API testing, seems like API testing is more dependent on coding, scripting and increasingly being done a lot by developers Interesting to note that Only 10% of Manual testers are doing API testing and only 10% of Software Developers are doing UI testing, QA engineers have a large proportion of both UI and API testing Why do you think are more developers testing APIs than QA engineers and Manual testers? Why are manual testers less involved in API testing?
- Harsh: Most of the teams are using unit testing! And some form of automation as well. Automated UI testing is used more than automated API testing. Service Virtualization seems to be a nascent concept and teams are not leveraging the value provide by service virtualization Interesting: Unit testing is prevalent amongst majority of the teams, Automation prevails in UI testing and slightly less so for APIs Testers are saying that unit testing is the most used methodology, what do you think are reasons for the popularity of this? Is there a learning here for teams not doing any unit testing? What about load testing 56% is it a high number? Service Virtualization is just 12% did you expect thart?
- Harsh: API testing is getting more and more important and people understand the importance of testing APIs behind the frontend. We wanted to know how much time testers spend on APIs and what are they spending it on Most of the time is being spent be testers on functional testing their APIs, surprisingly very few people said that they are spending most of their time on Load testing which is less than the number of people who said they spend most time on creating mocks Interesting: that many people are spending maximum time on creating mocks for services, indicates that mocking is hard to do and how service virtualization solutions like ServiceV can help speed up the process Load testing seems to be behind mocking in terms of number of people who said they spend most time on? Endpoints in testing: what is the number of API endpoints people are using to test and it seems that most of them, i.e. 80% are using less than 50 endpoints for testing
- Harsh: We asked people what %age of their tests are automated, also we asked them their company size. We found that 44% of the tests on an average are automated, larger companies automate more of their testing than smaller companies. Percentage of tests automated seems to be dependent on the size of the company, Interesting insights were around bigger companies having more tests automated and smaller companies will have lesser automation. What are the reasons that smaller companies have less exposure to automation? Bigger companies have
- Harsh: A very interesting insight we found was around the scripting proficiency and automation we had a hypothesis that both have a relationship and we did find a strong correlation here. Higher the scripting proficiency, higher is the proportion of tests automated. Experts have 64% of their tests automated which is way higher than the average number of test automated i.e. 44% Beginner scripters said that only 11% of tests are automated More proficiency in scripting indicates more automation? How do we explain that? Is it scalable? How easy is it for teams to scale scripting based automation system?
- We asked testers what industries they worked in and derived the percentage of testers in each industries who said that they have high proficiency in scripting As scripting skills correlates with the extent of automation in a team. So we decided to explore the industries where testers had the most scripting skills to assess their automation potential NGOs have the greatest proportion of testers who have proficient scripting skills, followed by pharma and government Finance and Healthcare are expected to have more automation, but what explains more scripting experts in NGO/Non Profit? What do you think about the potential of automation there?
- Lets talk about specific languages, what languages are testers using while writing scripts and probably automating We decided to segregate the and analyze language skills by UI and API. Higher proportion of API testers use java, Python, Groovy PHP Higher proportion of UI testers use VB and C# What are the reasons that UI testers and API testers have different languages they are interested in? Poll: Do you script or do you use a tool for UI testing? Script Tool Higher number of API testers use Java, Python, Groovy and PHP Higher number of UI testers use C#, Visual Basic
- We also got some insights about manual testing, seems like its not entirely dead yet, 56% of tests are still written manually Exploratory Testing was used by 45% of the testers, its mostly used in midsized companies, not very popular in small and very large size companies
- We asked what’s the process that your team follows? Trends: https://trends.google.com/trends/explore?date=today%205-y&q=%2Fm%2F0867l,%2Fm%2F02zhbn Why are teams still doing waterfall? What does ‘It depend’ on the project means? Doo teams have to be flexible in choosing a process depending on the circumstance?
- Here we segmented the results on the team size and this graph represents the %age of respondents in each segments that said that their teams follow Agile. 64% of the teams with members between 6-15 said that they were Agile. This number dips down to 56% for teams with 50 or more teams. Although 56% is a high proportion but 64% is higher, are teams finding immense success with the size of 6-15 in staying agile? Is this the sweet spot for the teams size?
- Here again we looked at segments by team size and analyzed the proportion of teams that did Test Management Seems like bigger the team size the lesser the proportion of teams doing test management Poll: Are you happy with your test management solution? Yes/No
- Harsh: We asked how often the participants release their software? And tried to see if there are any differences in release cycles between API projects and UI projects. UI exclusive testers more likely to release quarterly/yearly. More API projects get released in shorter release cycles multiple times a day or weekly APIs seems to have shorter release cycles and more UI is released over longer periods quarterly and yearly.?
- This slide indicates the usage of different browsers by testers and the Mobile phone platforms used by testers Browser consumption, firefox more than IE? Why firefox not the biggest because of selenium? Blackberry? Fragmentation of platforms on android? Selenium angle? Poll: Do you write selenium scripts for multiple browsers? Ans: Yes/No
- Liana: Which type of API do you test against most frequently? Pie chart with rounded numbers 6% are using primarily 3rd party API For unavailable APIs, 16% do nothing and 17% have no idea they have solutions available to emulate services. Poll: What do you do when you have to test against third party APIs? I doMocking API Virtualization I do Nothing
- Explain slowly: This slide indicates the percentage of API testers and UI testers who think that their tests are fully automated now and will be automated 2 years in the future. So for example around 24% of API testers said that they have full automation now and 40% said that they expect to achieve full automation in 2 years from now. Similarly on the right side the percentage of QA engineers and QA managers amongst overall respondents who believed that their tests are fully automated now and will be automated 2 years in the future. API testers seem to be more confident about 100% automation in the future than UI testers, is it because they are seeing higher automation now and that makes them more optimistic? QA Engineers are more confident about 100% automation than are QA managers.
- Definitely share with us your interpretation of the data with us. We are eager to hear from you.
- Thanks for attending - we’ll be sending everyone the first copies of the full report at the end of the week. If you’re local to the greater Boston area, join us in September for our user conference that will consist of live, interactive product training classes and insights from industry experts and SmartBear customers. We’ll be dropping the link in the chat box for those of you interested in buying tickets.