Mobile testing and its part in testing tools
Upcoming SlideShare
Loading in...5
×
 

Mobile testing and its part in testing tools

on

  • 380 views

Mobile testing and mobile test automation tools case (iOS case with Android and Windows Phone roadmaps), testing tool survey results analyzed, consideration about tool and cloud trends in general, but ...

Mobile testing and mobile test automation tools case (iOS case with Android and Windows Phone roadmaps), testing tool survey results analyzed, consideration about tool and cloud trends in general, but reflected with mobile testing tools.

Statistics

Views

Total Views
380
Views on SlideShare
380
Embed Views
0

Actions

Likes
1
Downloads
22
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Mobile testing and its part in testing tools Mobile testing and its part in testing tools Presentation Transcript

    • Mobile testing and its part in testing tools Kari Kakkonen, March 25, 2014, ASTQB Software Testing Conference, San Francisco © Copyright Knowit Oy 2013 | Confidential | Version 1.0
    • Kari Kakkonen, Knowit • Roles • Knowit, Director, Quality and Competences & Lead Consultant • Finnish Software Testing Board FiSTB, chairman • ISTQB international exam and expert working group member • Finnish Software Measurement Association FiSMA ry board member 2004-2007 • Chairman of research project STX, Lappeenranta University of Technology • Achievements • Testing promoter since 1996 • Tietoviikko-magazine 100 most influential IT-persons listing 2010 • Presentations in national and international conferences • Candidate for Tester of the Year 2010 (TestausOSY – FAST) • TestausOSY/FAST founding member • Education • ISTQB Advanced certified • SPICE provisionary assessor certified • M.SC, Helsinki University of Technology / Aalto-university • Marketing studies, University of Wisconsin-Madison • Domain background • Embedded • Public sector • Pension insurance • Training provider • Telecom • Finance • Manufacturing • Services • Training ISTQB Advanced and Foundation + Knowit Testing Professional • Service management • Test management • Test assessments (TPI Next etc.) and test development • Agile testing services © Knowit Oy / SECRET (Internal slide) Twitter: @kkakkonen LinkedIn: fi.linkedin.com/in/karikakkonen/
    • 110 145 154 189 215 215 220 2007 2008 2009 2010 2011 2012 2013 Trusted ICT Services Partner in Nordics Experts in 1800 Sweden, Finland, Norway, Denmark, Russia and Estonia Knowit AB is listed on the Nordic Exchange in Stockholm Grouprevenue(MEUR)
    • Case description © Copyright Knowit Oy 2013 | Confidential | Version 1.0 4
    • Client • Worldwide company with many products • Top of the line in their business • Consumer products (B2C) • Established software testing, including automation • Now wanted their first mobile app 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 5
    • Three perspectives of mobile solutions 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 6 Technology • Responsive web • Mobile optimized web • Native application • Hybrid application Target group • Consumers (B2C, B2B2C) • Retailers and suppliers (B2B) • Stakeholders • Own personnel Benefits • Growth in sales • Customer loyalty • Savings Client choices in bold
    • Mobile app scope • To enable users to access the service also from mobile devices • not only through the established web service • Utilize mobile device’s location services heavily • First platform - iOS • For anonymity the actual functionality is here called just “activity” • Supporting functionalities in a bit more detail, meaningful things for testing here 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 7
    • Expected challenges • Connectivity to web service, integral part of the service • Usability – will the vast user base like it? • Can we automate? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 8
    • 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 9 Functionalities of the application Location • Activity • Data • Map • Graphs • Camera Profile • Latest info • Following • Fans • Shouts • Edit Profile Personal • Latest info • Calendar Other • Following • Shouts • Settings
    • Our team • Compact scrum-team • 2 developers in Sweden • Project manager, 2 testers and 1 designer in Finland • All participating in daily scrum meetings • Customer committed to be in close contact with the team on daily basis • Customer geographically dispersed • Remote connection within team and to customer (Lync, Skype) 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 10
    • 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 11 We piloted automation
    • Knowit test automation process used in the mobile project 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 12 Preparation & piloting Implementation Testing Testing Recording and scripting Test execution New tests Script maintenance Reporting Feasibility study Piloting POC Implementation Training 1.version of software under test
    • © Copyright Knowit Oy 2013 | Confidential | Version 1.0 Feasibility study Phase Description • Benefits of automation in this project? • Discussions about test automation and tools with customer Proof Of Concept (PoC) • Verifying that the test automation tool is compatible with application under test • Identifying parts that are impossible/no benefit to automate • Installation of test tool to machines • Placing the hooks to software • Automating 1-2 test cases • Investment calculations and workload estimations • Scripting test cases • Running test cases in development environment • New versions of sofware under test in regular intervals Piloting • Execution and maintenance of test cases • Scripting of new test cases • Reporting Implementation Testing
    • Tool options 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 14 Availabletools MonkeyTalk Sikuli Testing with Frank Xcode Selenium NativeDriver Robot Framework Junit eggPlant for mobile Silk4mobile Piloted Robot Framework Silk4mobile Xcode Chosen Silk4mobile Xcode
    • Criteria for selection • Target operating system iOS • Several iOS options (iPad, iPad mini, iPod, iPhone) • Other operating systems in the future (Android, Windows) • Cost-effectiveness • Ease of installation and learning • Existing experiences of use 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 15
    • Silk4mobile • Supports all mentioned mobile platforms • Knowit Silk4Mobile know-how • Straightforward implementation • Requires hooks in the software or a jailbreak • Borland is a Knowit partner 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 16
    • Xcode • Default Apple development environment • Includes test automation environment • Automation tests run with emulator • Operational testing in emulator • Used in Knowit as development and unit test environment 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 17
    • Project implementation © Copyright Knowit Oy 2013 | Confidential | Version 1.0 18
    • Testing process 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 19 Development Developers’ tests Test team • Smoke test • UI testing • Field testing • Automation Beta testing Feedback • Internal testing • Beta testing
    • Early challenges • Challenges of automation • Tried out Silk4Mobile (Borland) • Requires hooks within the app • Part of iOS test automation requires ‘jailbreaking’ • iOS7 • Apple updated its operating system at the same time with this project -> retesting • Other challenges • Application needed a lot of field testing in true environment • A lot of outdoor use • The web service also updated during the project • Web service interfaces were modified with changes to app behavior • Downtimes in web service • Testing consisted of interaction between app, location information from phone, and web service21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 20
    • Test planning • Division of testing tasks between testers and beta testers • Testers continuously validated the correct functionality of the app features • Beta testers were required to validate real life scenarios • Levels of use • Basic use: activities and following the web service • Heavy use: more activities, following the web services, shares own information to social media, likes and comments of friends • Benchmarking: does app + web service work with same ease and content as web service only? • Use of exploratory testing © Copyright Knowit Oy 2013 | Confidential | Version 1.0
    • 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 22 Test management - ChiliProject… • User story - > test scenarios (different variety) - > field tests • Incidents, bugs and other issues reported back to Chili • Roadmap • Sprints • Features • Bugs • Status • Priority • Target version • Files • Reports
    • Implementation of tests Basicuser Indoor Outdoor City location Forest On the water Sailing 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 23 Specialcases Network is lost At sea In a forest In the cityConnection is lost Web service is down Data is transferred after connection is restored
    • Beta testing – field testing • After smoke tests app moved to beta testing stage • Purpose of beta testing • To get the actual end-user experience • Real life use cases • Achievements / what was good? • Wide coverage • Use cases : active and basic users, like-minded group of active users discussing with others online and at different events, continuously growing number activities, usage in different countries, various programs, all iPhone models plus iPad and iPod… • Validation testing at the end of the project • A group of test users, about 10-20 persons who were not involved in the project in any way, but had previously used the web service, from heavy to basic users in different countries • Using mobile app normally and giving feedback 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 24
    • Summary • Challenging start • Challenging test environment in mobile testing • Mobile test automation can be tricky, settled to unit test automation only • Demanding usage environment in this project – field testing was a success • Despite the challenges, comprehensive and versatile testing • A total of 140 functionalities and user stories tested • 445+ issues were reported to Chili, fixed and verified • Especially happy customer and successful stable application • Based on this project the customer ordered a new mobile application development and testing from Knowit 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 25
    • Test tool survey reflections © Copyright Knowit Oy 2012 | Luottamuksellinen
    • Mobiles roll… Number of smartphones worldwide: Q3 / 2012 1 000 000 000 2015 2 000 000 000 Source: Strategy Analytics
    • Test automation options 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 28 Mobile testing Emulator Remote device testing • Devices as a service Device specific automation
    • Mobile testing software 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 29 Open source • Robot framework • Robotium • JUnit / OCUnit • Appium • Calabash • MonkeyTalk • Sikuli • Testing with Frankk Commercial • Borland Silk4Mobile • Squish – Frologic • Keynote DeviceAnywhere • Testdroid • Appurify • EggPlant for mobile
    • The survey © Copyright Knowit Oy 2013 | Confidential | Version 1.0 30
    • Background • Research target: Software testing tools usage in Finland • Research is a part of Master Degree of Business Informatics thesis for Metropolia in co- operation with Knowit Oy • Used research method was quantitative method with web-form survey • Survey was sent to IT- and testing professionals in Finland • Knowit customer contacts • TestausOSY society • Soliditet register of Bisnode Finland • Survey was done in February 2013 • You could choose many options in most questions © Copyright Knowit Oy 2013 | Confidential | Version 1.0
    • Respondent’s background • Response rate was 39 % i.e. total 107 responses • 64 % was men and 36 % women • Age distribution; 84 % age between 31 to 50 years • Education; highly educated, 86 % have university level education • 87 % of respondents from South of Finland and 10 % from West of Finland 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 32
    • Profession/role in the company 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 33 38% 4% 6% 5% 5% 13% 3% 1% 3% 2% 3% 1% 1% 1% 12% 4% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Most of the respondents were test managers/ test coordinators and system testers
    • Operating area of the company 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 34 The largest operating areas were Information and telecommunication, Finance and insurance business and Industry Other service 12 % Science or technical area 7 % Finance and insurance business 21 % Information and telecommunication 35 % Retail business 2 % Industry 16 % International organization and commission 1 % Business area unknown 2 % Mining 1 % Administration and support 1 % Medical 2 %
    • Size of the company 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 35 2% 7% 5% 24% 14% 45% 1% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 - 10 51 – 100 11 - 50 101 – 500 501 - 1000 I don't know Over 1001 employees The respondents were mostly from large companies
    • Current life cycle model in the project 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 36 5% 67% 13% 7% 21% 29% 50% 24% 11% 5% 8% 0% 10% 20% 30% 40% 50% 60% 70% 80% Current projects life cycle models were Scrum, V-model and Iterative model
    • Future life cycle model 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 37 18% 25% 4% 2% 5% 8% 7% 16% 40% 9% 30% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Scrum seems to be future trend, Kanban and Lean are following, V-model use is going down
    • Which employees of your company are using software testing tools? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 38 72% 74% 89% 19% 24% 28% 25% 3% 9% 64% 6% 80% 49% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Testing tools are used by testing professionals and developers
    • What kind of training do you have for software testing? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 39 5% 1% 25% 2% 1% 0% 0% 0% 7% 12% 15% 1% 2% 3% 6% 49% 4% 1% 4% 0% 0% 27% 24% 18% 3% 14% 0% 10% 20% 30% 40% 50% 60% 70% 80% Men Women 74 % of respondents have ISTQB/ISEB Foundation course, next most common are tool specific courses and scrum master course.
    • What kind of testing tools are in use? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 40 88% 60% 48% 4% 0% 20% 40% 60% 80% 100% Commercial software testing tools Open Source testing tools Proprietary tools Other Tools from many sources are used side by side
    • Which way the testing tools are acquired? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 41 40% 6% 3% 4% 25% 22% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Bought Cloud service Other way Rented Open Source Coded by company itself Normally testing tools are acquired by buying. Next most common is open source usage and self coding According to this research cloud services are not yet a popular acquisition method..
    • Which way the testing tools are acquired by company size? (number of persons in the company) 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 42 1 1 2 1 - 10 5 5 2 2 1 11 - 50 5 42 2 2 1 Open Source Bought 12 22 3 13 2 3 101 - 500 6 142 7 1 1501 - 1000 27 466 24 1 5 over 1000 51 – 100
    • What test management tools do you have in the company? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 43 1% 14% 9% 2% 54% 3% 2% 5% 2% 47% 1% 1% 1% 5% 1% 9% 0% 10% 20% 30% 40% 50% 60% Most popular Test management tools were HP: QC, Atlassian: Jira and TestLink
    • Test management- vs office-tools 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 44 1% 54% 9% 47% 0% 1% 1% 1% 1% 0% 0% 0% 5% 9% 1% 2% 0% 0% 5% 3% 2% 2% 1% 13% 8% 7% 2% 1% 1% 1% 1% 0% 10% 20% 30% 40% 50% 60% 70% No Yes Also QC and Jira users are utilizing often office tools (spread sheets, word processors)
    • What incident management tools do you have in the company? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 45 41% 11% 3% 5% 3% 12% 25% 5% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Atlassian: JIRA IBM Rational ClearQuest Polarion Proprietary tools HP: Quality Center Bugzilla (Open source) Other Mantis Atlassian: Jira is clearly the largest in this category and second open source tool Bugzilla – QC is surprisingly very little used for incident management.
    • What test execution tools do you have in the company? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 46 33% 7% 3% 4% 2% 1% 2% 2% 2% 2% 1% 3% 4% 5% 21% 2% 4% 29% 4% 4% 45% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Open source tool Selenium was most popular, next HP:QTP and open source tool Robot Framework. There are a large number of different tools in use in this category.
    • What test design tools do you have in the company? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 47 1% 1% 1% 2% 2% 46% 2% 3% 8% 1% 4% 4% 2% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Major test design tool was HP: QC, note also MBT tools and test technique tools (CTE, allpairs)
    • What test harness/ unit test framework tools do you have in the company? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 48 27% 5% 17% 2% 3% 1% 12% 7% 7% 2% 7% 2% 0% 5% 10% 15% 20% 25% 30% These tools are mostly used by developers. Most used tools in this category were CppUnit, Junit and .TEST
    • What specialized unit test frameworks tools do you have in the company? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 49 2% 2% 4% 1% 6% 1% 2% 5% 1% 0% 1% 2% 3% 4% 5% 6% Specialized unit test frame work tools are less used but most popular is JBehave
    • What performance/ load/ stress testing tools do you have in the company? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 50 1% 3% 7% 1% 4% 4% 7% 22% 7% 12% 4% 24% 0% 5% 10% 15% 20% 25% 30% Most popular in this category were open source tool Jmeter and HP Performance Center.
    • Overall usage of all software testing tools 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 51 88 4 4 12 4 4 98 32 21 7 3 4 6 24 15 27 5 26 5 5 2 29 45 15 3 5 9 3 31 12 0 20 40 60 80 100 120 %ofrespondents 98 % have QC in use or have used and 88 % have JIRA in different testing phases.
    • Purpose of software testing tools? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 52 79% 76% 80% 29% 48% 50% 77% 30% 4% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Reporting Test execution Planning test cases Generating test cases Requirement management Defect tracking Defect control and management Reviews Other Typical testing activities have the lead in this category, i.e. planning test cases, reporting and test execution.
    • Do you use ‘office tools’ in testing (Spreadsheet, word processor etc.)? 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 53 35% 1% 17% 12% 1% 1% 10% 19% 1% 3% 0% 5% 10% 15% 20% 25% 30% 35% 40% Reporting Requirement management Test data Test design Defect control and management Test management Test execution Documentation Design Calculation • Yes, office tools in use 80 % • Which functions are you using them for? Office tools are used to same functions as the software testing tools
    • Testing tool markets directions • Open source is gaining momentum, in midsize companies • Big (commercial) test tool suites work for big companies • Cloud (software testing tool as a service) is just emerging, but a growing trend • Specialized tools, like mobile test automation, emerge all the time • Are not first considered testing tools (or testers are not the users) • Are soon swallowed by big test suite providers • Integrated tool sets for multiple platforms are seen as most useful • Support e.g. all mobile platforms and all browsers 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 54
    • Recommendations • Pay attention to test tool life cycle – does it have future? • Open source – the user and developer base • Commercial – company size, market shares • Consider tool integrations • Mix and match open source and commercial • Integrate tools to work together – e.g. Jira – QC – Robot Framework • Look for good tools • Superb usability • Excellent technical flexibility • Even small vendors grow soon with superb tools • Word of mouth is big player (Finnish Robot Framework phenomena) 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 55
    • ”There are plenty of software testing tools. Expertise is needed to select effective testing tools to suitable areas of testing processes” © Copyright Knowit Oy 2013 | Confidential | Version 1.0 56
    • Questions? © Copyright Knowit Oy 2013 | Confidential | Version 1.0 @kkakkonen or kari.kakkonen@knowit.fi