TMF2014 Mobile Testing Workshop Michael Palotas
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

TMF2014 Mobile Testing Workshop Michael Palotas

  • 377 views
Uploaded on

A very big thank you to Michael Palotas from Grid Fusion & eBay International for taking the time and effort to travel across the globe to present at the Australian Test Managers Forum 2014. If you......

A very big thank you to Michael Palotas from Grid Fusion & eBay International for taking the time and effort to travel across the globe to present at the Australian Test Managers Forum 2014. If you would like any information on TMF please email tmf@kjross.com.au

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
377
On Slideshare
363
From Embeds
14
Number of Embeds
1

Actions

Shares
Downloads
6
Comments
0
Likes
1

Embeds 14

https://twitter.com 14

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. TEST MANAGERS FORUM 2014 SYDNEY, AUSTRALIA MOBILE TESTING WORKSHOP 1
  • 2. 2 WHO AM I? Gridfusion Software Solutions Contact: Michael Palotas Gerbiweg 2 8853 Lachen SWITZERLAND Tel.: +41 79 6690708 Email: michael.palotas@gridfusion.net Head of Productivity & Test Engineering, eBay Founder / Principal Consultant Gridfusion Software Solutions
  • 3. SETTING THE STAGE Tell me about yourself J What are your expectations for today? 3
  • 4. MOBILE TESTING SPECIFICALITIES 4
  • 5. MOBILE IS A VAST LAND 5
  • 6. YOU ONLY HAVE ONE SHOT 6
  • 7. MOBILE TESTING CHALLENGES DEVICE DIVERSITY 7
  • 8. MOBILE TESTING CHALLENGES NEW OS VERSIONS New versions usually fix issues but May also introduce new ones 8
  • 9. MOBILE TESTING CHALLENGES NEW OS VERSIONS 9
  • 10. MOBILE TESTING CHALLENGES NEW OS VERSIONS 10
  • 11. NATIVE APPS AND MOBILE WEB 11
  • 12. MOBILE TESTING CHALLENGES LOCALIZATION 12
  • 13. MOBILE TESTING CHALLENGES NETWORK 13
  • 14. MOBILE TESTING CHALLENGES OPERATORS 14
  • 15. MOBILE TESTING CHALLENGES MOBILE ENVIRONMENT 15
  • 16. 16
  • 17. 17
  • 18. 18
  • 19. 19
  • 20. MOBILE TESTING CHALLENGES DEVICE MANAGEMENT 20
  • 21. DEVICE OR EMULATOR? 21
  • 22. RISK ASSESSMENT AND STRATEGIES 22
  • 23. RISK & STRATEGIES Hotfixes are difficult and expensive Agile? ...yes...but...for mobile the goal should be to get it right the first time 23
  • 24. RISK & STRATEGIES May shift back to waterfall (waterfall is not that bad) Difficult to get (mobile) experienced people Need to grow capabilities in-house 24
  • 25. RISK & STRATEGIES No automation in most places today No continous integration / continous delivery 25
  • 26. RISK & STRATEGIES DEVICE SELECTION 26 Be pragmatic Focus on where your money is coming from Forget the “Long Tail”
  • 27. RISK & STRATEGIES Multi code base Many devices Device / OS management Simulator vs. real devices 27
  • 28. CLOUD TESTING 28
  • 29. CLOUD VENDORS - DEVICE ANYWHERE 29
  • 30. CLOUD VENDORS - SAUCELABS 30
  • 31. TESTDROID 31
  • 32. GENERAL EXPERIENCES WITH THE CLOUD No dedicated devices Or no devices at all Device management Your company firewall... Trust issues Slow(er) No in-house knowledge Not onsite It is a bit like outsourcing $ in the beginning $$$$$$ when you are scaling 32
  • 33. CROWD TESTING 33
  • 34. REALLY? 34
  • 35. CROWD TESTING – WHAT USUALLY WORKS? Fresh set of eyes Access to many regions, devices, OS combinations (at least theoretically) Lots of issues reported Relatively low cost 35
  • 36. CROWD TESTING – CHALLENGES Significant ramp up required for testers Low bug quality You don’t know who is on the other end... People in mature markets won’t work for 5$ / bug Only access to production environment QA system access very difficult Legal implications, NDAs etc. No development resources to fix the bugs 36
  • 37. MOBILE DEVELOPMENT APPROACHES 37
  • 38. MOBILE DEVELOPMENT APPROACHES Mobile First Responsive Design 38
  • 39. TEAM ORGANIZATION 39
  • 40. AGILE AND MOBILE CI – OK CD – OK for QA versions CD – NOK for prod versions 40
  • 41. SKILLSET - THE PERFECT TESTER 41
  • 42. GENERAL TEST AUTOMATION 42
  • 43. FLIP THE AUTOMATION TRIANGLE Unit Tests E2E Automation Tests Manual Tests Unit Tests E2E Tests Manual Tests Integration Tests
  • 44. TEST AUTOMATION WHY? 44
  • 45. TEST AUTOMATION WHO? 45
  • 46. TEST AUTOMATION WHAT? 46
  • 47. TEST AUTOMATION WHEN? 47
  • 48. INSTANT FEEDBACK D T D T D T D T D T D T D T D T Development Test
  • 49. AUTOMATION IS GOOD…BUT Tools help, but tools don’t test Manual testing is equally important Manual (exploratory) and automated testing complement each other 49 Automation Manual
  • 50. WHY IS E2E TESTING SO IMPORTANT? Because it is the most difficult way to test Because our users use our products End2End Because you can verify that the whole system works together 50
  • 51. PERSONAL THOUGHTS ON AUTOMATION Manual testing is still very important E2E tests are expensive, so keep them to the minimum Automation is software development Be careful with record & playback tools Automate the repetitive tasks (regression) Build testability into your product ROI... More at http://gridfusion.net/thoughtsonautomation.html 51
  • 52. MOBILE TEST AUTOMATION 52
  • 53. MOBILE TEST AUTOMATION Not as mature as web automation Immature tools market Tools are usually platform specific Multi code base Need to understand the underlying architecture much more than for the web Need people who can code well 53
  • 54. EMULATORS VS. DEVICES What to test where? 54
  • 55. WHAT TO TEST WHERE - EMULATOR functional test 55
  • 56. WHAT TO TEST WHERE - DEVICE performance look and feel memory usage carrier specific tests 56
  • 57. IOS SIMULATOR SPECIFICS 57 Comes from Apple Works well & reliable
  • 58. ANDROID EMULATOR SPECIFIC 58 SLOW
  • 59. ANDROID VS. IOS Android: lots of variations, versions, vendors Android: open source (sort of) iOS: easy to update iOS: everything comes from one vendor iOS: not exactly open source J 59
  • 60. SAMPLE MOBILE AUTOMATION REQUIREMENTS Use the same approach and infrastructure as web Business is borderless à test automation needs to be as well Aim for REAL E2E test automation Use the Load Balancer / Resource Management Features of Selenium Grid Manage multiple applications / versions / languages No modification of mobile app No modification of mobile device Mobile App Runtime Inspection Support Hybrid app support Support for emulator + real device Integration into CI system 60
  • 61. SAME AUTOMATION CODE BETWEEN PLATFORMS? Sounds good first but Most apps are different between platforms Different element locator strategy Do reuse the helper functions 61
  • 62. MOBILE AUTOMATION IOS / ANDROID DIFFERENCES iOS has no IDs à harder to automate iOS relies on xcode Android is open source Android: ID support 62
  • 63. TEST INFRASTRUCTURE 63 Driver Browsers Mobiles AUT DB API
  • 64. TOOLSETS - OVERVIEW Ranorex QTP TOSCA SeeTest Selenium/ ios-driver/Selendroid Selenium / APPIUM 64
  • 65. SeeTest 65
  • 66. WHAT ABOUT BDD / ATDD? I have seen it working Required to close the loop Removed complexity (while adding it at the same time) Have you ever seen a product owner test? 66
  • 67. SAMPLE TOOLS JBehave Fitnesse Cucumber 67
  • 68. SELENIUM 68
  • 69. SELENIUM – WHAT IS IT? Web testing tool Highly successful open source project Stable Protocol describing the user interactions Supports most browser implementations Support for most common programming languages 69
  • 70. SELENIUM – WHAT IT IS NOT a drag & drop tool a network testing / monitoring tool a performance testing tool a reporting tool a testcase management tool 70
  • 71. SELENIUM i.e. Firefox driver Server JSON WIRE PROTOCOL Client Java C# Ruby Python Server i.e. Selendroid, iOS-Driver
  • 72. SCALING - SELENIUM GRID 72
  • 73. SELENIUM GRID 73
  • 74. SCALING USING SELENIUM GRID 74 Client serverGRID HubClient CI server server server server server server
  • 75. 75 Implements the JSON Wire Protocol (so it is fully compatible with Selenium) Runs on emulators and devices Can be run as a node in Selenium Grid Support for localized native apps (1 test for all locales) No need to change the AUT No need to jailbreak the device
  • 76. 76 Hybrid support Different locator types are supported to find elements Gestures are supported: Advanced User Interactions API Hot plugging It can be integrated as a node into the Selenium Grid for scaling and parallel testing Android target API support from 2.3.3 to 4.3 (latest) Built in Inspector to simplify test case development.
  • 77. REPORTING Invest into proper reporting Easy analysis of the reports is crucial Selenium does no reporting TestNG or JUnit are responsible for reporting 100% customizable reports (i.e. different reporting levels, PDF, HTML, Email, SMS etc...) Hooks into test case management tools 77
  • 78. REPORT WITH SCREENSHOT 78
  • 79. SELENIUM SELENDROID IOS-DRIVER 79