Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mobile test automation perfecto star east


Published on

Eran Kinsbruner presentation at StarEast 2015 around mobile & Continuous Quality best practices.

Published in: Mobile
  • Login to see the comments

Mobile test automation perfecto star east

  1. 1. 101 – How To Deliver Winning Mobile Apps Perfecto Mobile & Toronto Association of System and Software
  2. 2. Eran Kinsbruner is a director & mobile technical evangelist at PerfectoMobile, one of the leading mobile cloud and automation companies. Formerly CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. The co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile testing world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and his professional mobile testing blog at
  3. 3. What To Do? Great News – Your first Mobile Project has arrived! Now What?
  4. 4. Presentation Objectives • Mobile Market State • Mobile QA/Dev Objectives • Sustaining a Mobile QA Plan Based on CQL • Functional Testing • Non Functional Testing • Regression Testing/CI • Pillars Of A Continuous Quality Lab • Mobile QA Best Practices
  5. 5. Mobile Market State Digital experience drives innovation • IOT • Complex use cases Fragmentation becomes more painful • Devices • OS’s Tool mix as a key for agile adoption • Commercial, Open Source, Proprietary
  6. 6. Mobile QA & Dev Objectives Source: OpenXcell Quality Velocity Increase Quality Increase Velocity Mobile QA & Dev Objectives
  7. 7. QA & Dev Testers Vision (aspiration) Increase mobile app velocity without compromising on quality Keep up with mobile market changes and meet end user requirements Develop & Test against the real end user profiles Leverage unattended automation as part of any nightly build through CI workflow Receive actionable mobile app quality insights per build
  8. 8. RBC – QA Objectives v Improved Quality Optimized Cost Faster Time to Market Operational Effectiveness Operational Efficiency Business Value • Increase Test Coverage • Test Effectiveness – reduce test duplication between QA & business • Reduce defect leakage to production • Reuse test automation during UAT test cycles • Decrease QA test cycles – via test automation • Near zero downtime/ production failures • Applications perform as expected • Test automation cost savings • Manual and automation test management savings 10% using BDD Approach • Onsite to offshore ratio 40:60 • QA Technical Solution Lead across all projects • Leverage Perfecto Mobile cloud solution to save on devices and data plans • Reduction in Cost of Testing due to productivity improvements- Maintenance and Projects • Reduce regression test cycle by 5-10 days per project • Perfecto Mobile UFT • Continuous improvement in test design & execution • Maximize Test Automation coverage • Increase re-use - BDD • Enable business to respond to market change • Proactive feedback on market changes to existing production mobile applications
  10. 10. Sufficient Coverage • Choose the right devices and OS platforms for your project • Get Perfecto Mobile’s Q1 2015 Mobile Test Coverage Index
  11. 11. Q1-2015 Mobile Test Coverage Index (Preview)
  12. 12. What’s Coming? Why It’s Important?
  13. 13. Manual & Interoperability • Starting point for Pre-Automation phase (foundation) • Candidates to keep manual: – Incoming events • Calls • Text Messages • Notification pop ups – Complex use cases to automate
  14. 14. Functional Automation – That Works • Reusable, Repeatable, Predictable Tests – Unattended – Continuous – Easy to maintain • Object Support On Any Mobile App Style – Dom, Visual, Native – Instrumentation (Yes/No) – System Level Control • Flexible Frameworks 100% accurate native object rec. Much faster than OCR Support all languages Does not often detect GUI defects Imperative for end user’s experience Imperative for GUI glitches Implement the WYSIWYG Slower then object level analysis Dependent on 3rd party Does not support all languages Object level analysis Visual analysis (OCR) May require Instrumentation
  15. 15. Levels of Test Automation Level 1 Level 2 Level 3 Level 4 Level 5 Navigation flows Content validation Error Validation Data validation (if feasible) UI validation (if feasible) Check automation feasibility Identify scenarios for basic navigation & successful function flow Navigation flows are automated Check automation feasibility of screen content Level 2 Scenarios are ready Check automation feasibility of error validation Level 3 scenarios are ready Check automation feasibility of data validation Level 4 scenarios are ready Check automation feasibility of UI validation and other complex features Automated scenarios of basic Navigation % happy path are ready Content validation scripts are ready and plugged in with exiting automated navigation BPT component Error validation components are ready and plugged in with automated components of Level 2 Data validation scripts are written and plugged in with existing automated components of Level 3 Automated scripts for UI screen & other complex scenarios are ready and plugged with existing scripts DataDrivenAutomationFramework
  16. 16. Non Functional Testing • Cover key business transactions response time & general mobile app availability • Early actionable performance feedback & insights • Assures high end user experience in various conditions (aka: Wind Tunnel) – Various network profiles – Various locations – Various load conditions – Incoming events – Per app insights
  17. 17. Mobile App Monitoring • The only Efficient Way to Monitor Mobile Apps – Browsers and Emulators - irrelevant for mobile monitoring • Capture Your True Mobile End User Experience – Monitor mobile engagement scenarios, including interoperability on real devices on real networks • Early Warnings and Fast Correction – Real-time alerting triggered when transaction availability or response times are compromised • Extend your existing quality assets for monitoring – Automation test code – Performance test code
  18. 18. What Makes A True CQL Quality Lab
  19. 19. What Makes A Complete Continuous Quality Lab Real Device Lab (Device as a Service) Functional Manual Testing Functional Automation Performance Testing Real environments & Profiles Production Monitoring Hybrid Cloud (Local, Public, Private) Operations, Security & Support 3rd Party API’s & Integrations
  20. 20. Mobile QA Best Practices
  21. 21. Mobile Continuous Integration Recommended Work Flow Dev Team Source Control Code Commit CI Server ReportingUnit Tests Launch Perfecto Job (Release Package) Support today: PM Native & Selenium Automated App Deployment On Real Devices Functional Testing Integration Testing Performance Testing Continuous Feedback Test Planning & Test Execution Bug Reporting
  22. 22. Build Your Mobile App Test Plan Properly Functional • Validation of Functionality • Smoke / Regressions Testing • Offline access testing • Negative Testing Non Functional • Network Strength / Outage / Recovery • Different NW Types • Peripheral Testing • Services Interoperability (IOP) • Voice / SMS interrupts • Notifications • Battery /Cable Removal Memory Leak • Memory Usage • Memory Leaks • Garbage Collection Installation Testing • New App Install • Uninstall and Reinstall • Upgrade testing Language Testing • Validation for Locales (French & English) • Images and Text • Currencies, time zones etc. • Context Performance Testing • CPU Usage testing • Network Usage • Page Render time or activity Render time Security Testing • OWASP Vulnerabilities • Sensitive data (App/Device) • Dynamic Testing • Static Code Analysis • Data Encryption Usability Testing • User Experience • Competitive Analysis • Expert Review
  23. 23. RBC Mobile Quality Lab Team Size – 45 (Onshore/Offshore Mix) >100 devices - Blackberry, Android & iOS platforms Using Selenium & UFT
  24. 24. Continuous Automation Regression Model • Executing automated test scripts 24*7 (consider offshore team) – Private Perfecto Cloud – used for project development • Execute automated testing suite to reduce project timelines – Internal Perfecto Cloud –used for project and maintenance testing on supported devices – Perfecto Public Cloud – test non supported devices (new market devices/ mimic customer behaviour). Leverage Perfecto public metrics i.e. what is currently in use by public (device and configuration) Hybrid Cloud
  25. 25. Q&A