Advanced Topics In
Continuous Testing
How to Avoid
Crashing a Very
Large Boat
Who Am I?
 Principal Software Engineer at Liberty
Mutual
 15+ Years of Experience in Open Source
Platforms
 Organize the Liberty Mutual Testing Guild
 Never been in a boating accident
Waterfall - A Scenic River Cruise
 Encourages detailed records
 You have well-defined expectations
 Easy to on-board new team members
 In a word: Safe
 In a second word: Slow
Agile – Cruising on the Open Sea
 Get changes to market faster
 Pivot quickly based on learnings
 Iterate, learn and improve ideas
frequently
 In a word: Fast
 In a second word: Dangerous
Testing – Your Lifeboat, Life Vest,
Radio, Flare Gun and First Aid Kit
 Different types of tests give you layers of
protection
 Valuable tests are worth the effort
 No such thing as too many types of testing
 Don’t ignore manual testing
Closing - Smooth Sailing From Here
 A good testing setup will prevent most
errors from getting to your users
 Public mistakes hurt your reputation far
more than any missed deadline
 Strong testing will raise internal confidence
in your product and team
 Again… never crashed a boat
Questions?
About Me
• Lead Technical Evangelist at Perfecto
• Blogger and Speaker
• http://continuoustesting.blog
• https://www.infoworld.com/author/Eran-Kinsbruner/
• 18+ Years in Development & Testing
• Author of The Digital Quality Handbook
Weekly Podcast - Testiumpod
Twitter: @ek121268
Email: Erank@perfectomobile.com
Agenda
 A word about the digital transformation
 Test Automation Key Challenges & Tools Selection Criteria
 Mobile Landscape Overview
 Cross-Browser Testing Landscape
 PWA
 RWD
 Headless
 Q&A
5/15/2018 10© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Digital is breaking new grounds
Drop of 32% in branch visits in last 4 years
Chatbots as the new Digital
Interface - everywhere
$380B industry by 2024 (26% CAGR)
Face ID Authentication
AR As a Digital Engagement
Enabler
Cross-Browser Testing != Desktop Web Testing
There is no Web Testing vs.
Mobile Testing
• 4 out of 10 transactions today take place on multiple
devices
• 48% of users today complain that the websites they use are
not optimized for their smartphones and tablets
DevOps == Continuous Everything
5/15/2018 13© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Credit: TestingWhiz
Top 5 Automation Challenges
Tighter release schedule
leaves less time to
automate and test
(lack of) Test
automation stability
and reliability block
continuous testing
Test execution
management isn’t
intelligent and
optimized enough to
guide teams through
the DevOps pipeline
activities
Evolving and
maintaining test sets
and maximizing
productivity isn’t an
ongoing practice
Overall test automation
tool stack isn’t in sync
with organizational
capabilities
People – Process - Technology
The Digital Continuous Testing Landscape
© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Mobile Testing Landscape
Criteria Appium Espresso XCUITests
Language any Java swift/objective-C
By Open source google Apple
App supported APK and IPA APK IPA
Code required No yes yes
Testtype Black box White box White box
Speed 8t t 2t
Setup Hard Easy Medium
CI medium easy Hard
Flakiness of test very low Low
Object Locators Xpath (external) Id (from R file) id
Used by QA Android dev* iOS dev*
Progressive Web App - Definition
Progressive Web Apps (PWAs) are web applications that are
regular web pages or websites, but can appear to the user like
traditional applications or native mobile applications. The
application type attempts to combine features offered by most
modern browsers with the benefits of a mobile experience
(Wikipedia)
5/15/2018 19© 2015, Perfecto Mobile Ltd. All Rights Reserved.
The Rise of PWAs – Challenging Native Mobile Apps
A Progressive Web App is
• Progressive - Works for every user
• Responsive - Fits any form factor
• Connectivity independent
• App-like - Feels like an app
• Safe - Served via HTTPS
• Search Friendly -allowing search engines to find it.
• Re-engageable -Support push notifications.
• Linkable - does not require complex installation
(store access)
5/15/2018 20© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Source :medium.com
Progressive Web App – The Offline Network Benefit
5/15/2018 21© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Source: VentureBeat
Progressive Web App - Architecture
5/15/2018 22© 2015, Perfecto Mobile Ltd. All Rights Reserved.
• A PWA App consists of few key components
• Manifest.Json - That's the file within the PWA that describes the app, provides metadata specific
to the app like icons, splash screens, and more
• Service Workers – A service worker is a script that your browser runs in the background, separate
from a web page, opening the door to features that don't need a web page or user interaction.
Today, service workers already include features like push notifications and background sync
Service worker push notification
PWA on iOS is Different than on Android
5/15/2018 23© 2015, Perfecto Mobile Ltd. All Rights Reserved.
iOS Android
App type shortcut apk
Offline data Deleted after idle time No limit
Offline storage (browser based) Safari < 50Mb Chrome 6% available storage
Sensors support Limited (BT) All
access to private information No Yes
Payment system No supported Supported
Voice support No Yes
Progressive Web App – Test Plan Strategy: 6 Steps
5/15/2018 24© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Validate PWA
Manifest
Validate PWA
Service Workers
Validate PWA
Specific
Capabilities
Properly Test
Across Platforms
(RWD)
Object
Identification
Strategy
Google’s PWA
Checklist
Compliance
Progressive Web App – Test Plan Strategy: Step 4
5/15/2018 26© 2015, Perfecto Mobile Ltd. All Rights Reserved.
• UI and visual/layout testing across multiple form factors
• Performance and rendering of the content across platforms
• Network related testing – in addition to the offline mode that is covered through service workers,
make sure to cover the app behavior throughout various network conditions (packet loss, flight
mode, latency %, 3G, 4G, 5G etc.)
• Functionality of the entire page user flows across platforms and screen sizes and resolutions
• The differences between browsers, platforms must be covered as well (Chrome, Safari, Android,
Samsung Browser vs. built-in browsers like Facebook - number of instances or copies a PWA app
can have on a device.
Properly Test
Across Platforms
(RWD)
Progressive Web App – Test Plan Strategy: Step 5
5/15/2018 27© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Object
Identification
Strategy
&
Progressive Web App – Test Plan Strategy: Step 6
5/15/2018 28© 2015, Perfecto Mobile Ltd. All Rights Reserved.
• https://developers.google.com/web/progressive-web-apps/checklist
Google’s PWA Checklist
Compliance
Headless Browsers Role
Motivation
• Receive fast feedback (Unit and basic functional tests) – Mostly Dev
• Easy environment setup, no IT dependencies
• Uses standard JS language
• Supported and promoted by browser vendors (Google, Mozilla)
• Additional use cases to cover:
• HAR File
• Performance testing
• Basic UI
• Security
5/15/2018 29© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Responsive Web Design (RWD) – Tools are already here
Responsive Web App – Test Plan Strategy: 6 Steps
5/15/2018 31© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Choose The Right
Platforms
Assure Visual Web
Pages Across
Platforms
Functionality of
Navigation Flows
and Menus
Client Side
Performance
Testing
Test Accessibility
for Mobile and
Web
Test Environment
Conditions
(Networks,
Sensors)
• Identify your object in a robust fashion that fits all digital Platforms
• Build object repository and use smart locators
Responsive Web Design (RWD) - Objects
The average website includes nearly 400
different objects.
Now try locating them on each and every
DIGITAL platform in your lab…
• Take screenshot and use Visual Checkpoint/assertion to
validate responsive aspects
Responsive Web Design (RWD) – visual validation w/ Screenshots
Web Testing Tools (Open-Source) - Download Trends
5/15/2018 34© 2016, Perfecto Mobile Ltd. All Rights Reserved.
JSDom & Mocha
Selenium
Protractor
Web Market Testing – Selection Criteria
5/15/2018 35© 2016, Perfecto Mobile Ltd. All Rights Reserved.
Read More Here
Free Giveaway to Meetup Attendees
10 Test Automation Frameworks for Cross Browser Testing
5/15/2018 36© 2015, Perfecto Mobile Ltd. All Rights Reserved.
https://info.perfectomobile.com/10-test-
automation-frameworks-for-cross-browser-
testing_mot.html
Q & A
© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Thank You

Mapping mobileandweblandscape motb

  • 1.
  • 2.
    How to Avoid Crashinga Very Large Boat
  • 3.
    Who Am I? Principal Software Engineer at Liberty Mutual  15+ Years of Experience in Open Source Platforms  Organize the Liberty Mutual Testing Guild  Never been in a boating accident
  • 4.
    Waterfall - AScenic River Cruise  Encourages detailed records  You have well-defined expectations  Easy to on-board new team members  In a word: Safe  In a second word: Slow
  • 5.
    Agile – Cruisingon the Open Sea  Get changes to market faster  Pivot quickly based on learnings  Iterate, learn and improve ideas frequently  In a word: Fast  In a second word: Dangerous
  • 6.
    Testing – YourLifeboat, Life Vest, Radio, Flare Gun and First Aid Kit  Different types of tests give you layers of protection  Valuable tests are worth the effort  No such thing as too many types of testing  Don’t ignore manual testing
  • 7.
    Closing - SmoothSailing From Here  A good testing setup will prevent most errors from getting to your users  Public mistakes hurt your reputation far more than any missed deadline  Strong testing will raise internal confidence in your product and team  Again… never crashed a boat
  • 8.
  • 9.
    About Me • LeadTechnical Evangelist at Perfecto • Blogger and Speaker • http://continuoustesting.blog • https://www.infoworld.com/author/Eran-Kinsbruner/ • 18+ Years in Development & Testing • Author of The Digital Quality Handbook Weekly Podcast - Testiumpod Twitter: @ek121268 Email: Erank@perfectomobile.com
  • 10.
    Agenda  A wordabout the digital transformation  Test Automation Key Challenges & Tools Selection Criteria  Mobile Landscape Overview  Cross-Browser Testing Landscape  PWA  RWD  Headless  Q&A 5/15/2018 10© 2015, Perfecto Mobile Ltd. All Rights Reserved.
  • 11.
    Digital is breakingnew grounds Drop of 32% in branch visits in last 4 years Chatbots as the new Digital Interface - everywhere $380B industry by 2024 (26% CAGR) Face ID Authentication AR As a Digital Engagement Enabler
  • 12.
    Cross-Browser Testing !=Desktop Web Testing There is no Web Testing vs. Mobile Testing • 4 out of 10 transactions today take place on multiple devices • 48% of users today complain that the websites they use are not optimized for their smartphones and tablets
  • 13.
    DevOps == ContinuousEverything 5/15/2018 13© 2015, Perfecto Mobile Ltd. All Rights Reserved. Credit: TestingWhiz
  • 14.
    Top 5 AutomationChallenges Tighter release schedule leaves less time to automate and test (lack of) Test automation stability and reliability block continuous testing Test execution management isn’t intelligent and optimized enough to guide teams through the DevOps pipeline activities Evolving and maintaining test sets and maximizing productivity isn’t an ongoing practice Overall test automation tool stack isn’t in sync with organizational capabilities People – Process - Technology
  • 15.
    The Digital ContinuousTesting Landscape © 2015, Perfecto Mobile Ltd. All Rights Reserved.
  • 16.
    Mobile Testing Landscape CriteriaAppium Espresso XCUITests Language any Java swift/objective-C By Open source google Apple App supported APK and IPA APK IPA Code required No yes yes Testtype Black box White box White box Speed 8t t 2t Setup Hard Easy Medium CI medium easy Hard Flakiness of test very low Low Object Locators Xpath (external) Id (from R file) id Used by QA Android dev* iOS dev*
  • 17.
    Progressive Web App- Definition Progressive Web Apps (PWAs) are web applications that are regular web pages or websites, but can appear to the user like traditional applications or native mobile applications. The application type attempts to combine features offered by most modern browsers with the benefits of a mobile experience (Wikipedia) 5/15/2018 19© 2015, Perfecto Mobile Ltd. All Rights Reserved.
  • 18.
    The Rise ofPWAs – Challenging Native Mobile Apps A Progressive Web App is • Progressive - Works for every user • Responsive - Fits any form factor • Connectivity independent • App-like - Feels like an app • Safe - Served via HTTPS • Search Friendly -allowing search engines to find it. • Re-engageable -Support push notifications. • Linkable - does not require complex installation (store access) 5/15/2018 20© 2015, Perfecto Mobile Ltd. All Rights Reserved. Source :medium.com
  • 19.
    Progressive Web App– The Offline Network Benefit 5/15/2018 21© 2015, Perfecto Mobile Ltd. All Rights Reserved. Source: VentureBeat
  • 20.
    Progressive Web App- Architecture 5/15/2018 22© 2015, Perfecto Mobile Ltd. All Rights Reserved. • A PWA App consists of few key components • Manifest.Json - That's the file within the PWA that describes the app, provides metadata specific to the app like icons, splash screens, and more • Service Workers – A service worker is a script that your browser runs in the background, separate from a web page, opening the door to features that don't need a web page or user interaction. Today, service workers already include features like push notifications and background sync Service worker push notification
  • 21.
    PWA on iOSis Different than on Android 5/15/2018 23© 2015, Perfecto Mobile Ltd. All Rights Reserved. iOS Android App type shortcut apk Offline data Deleted after idle time No limit Offline storage (browser based) Safari < 50Mb Chrome 6% available storage Sensors support Limited (BT) All access to private information No Yes Payment system No supported Supported Voice support No Yes
  • 22.
    Progressive Web App– Test Plan Strategy: 6 Steps 5/15/2018 24© 2015, Perfecto Mobile Ltd. All Rights Reserved. Validate PWA Manifest Validate PWA Service Workers Validate PWA Specific Capabilities Properly Test Across Platforms (RWD) Object Identification Strategy Google’s PWA Checklist Compliance
  • 23.
    Progressive Web App– Test Plan Strategy: Step 4 5/15/2018 26© 2015, Perfecto Mobile Ltd. All Rights Reserved. • UI and visual/layout testing across multiple form factors • Performance and rendering of the content across platforms • Network related testing – in addition to the offline mode that is covered through service workers, make sure to cover the app behavior throughout various network conditions (packet loss, flight mode, latency %, 3G, 4G, 5G etc.) • Functionality of the entire page user flows across platforms and screen sizes and resolutions • The differences between browsers, platforms must be covered as well (Chrome, Safari, Android, Samsung Browser vs. built-in browsers like Facebook - number of instances or copies a PWA app can have on a device. Properly Test Across Platforms (RWD)
  • 24.
    Progressive Web App– Test Plan Strategy: Step 5 5/15/2018 27© 2015, Perfecto Mobile Ltd. All Rights Reserved. Object Identification Strategy &
  • 25.
    Progressive Web App– Test Plan Strategy: Step 6 5/15/2018 28© 2015, Perfecto Mobile Ltd. All Rights Reserved. • https://developers.google.com/web/progressive-web-apps/checklist Google’s PWA Checklist Compliance
  • 26.
    Headless Browsers Role Motivation •Receive fast feedback (Unit and basic functional tests) – Mostly Dev • Easy environment setup, no IT dependencies • Uses standard JS language • Supported and promoted by browser vendors (Google, Mozilla) • Additional use cases to cover: • HAR File • Performance testing • Basic UI • Security 5/15/2018 29© 2015, Perfecto Mobile Ltd. All Rights Reserved.
  • 27.
    Responsive Web Design(RWD) – Tools are already here
  • 28.
    Responsive Web App– Test Plan Strategy: 6 Steps 5/15/2018 31© 2015, Perfecto Mobile Ltd. All Rights Reserved. Choose The Right Platforms Assure Visual Web Pages Across Platforms Functionality of Navigation Flows and Menus Client Side Performance Testing Test Accessibility for Mobile and Web Test Environment Conditions (Networks, Sensors)
  • 29.
    • Identify yourobject in a robust fashion that fits all digital Platforms • Build object repository and use smart locators Responsive Web Design (RWD) - Objects The average website includes nearly 400 different objects. Now try locating them on each and every DIGITAL platform in your lab…
  • 30.
    • Take screenshotand use Visual Checkpoint/assertion to validate responsive aspects Responsive Web Design (RWD) – visual validation w/ Screenshots
  • 31.
    Web Testing Tools(Open-Source) - Download Trends 5/15/2018 34© 2016, Perfecto Mobile Ltd. All Rights Reserved. JSDom & Mocha Selenium Protractor
  • 32.
    Web Market Testing– Selection Criteria 5/15/2018 35© 2016, Perfecto Mobile Ltd. All Rights Reserved. Read More Here
  • 33.
    Free Giveaway toMeetup Attendees 10 Test Automation Frameworks for Cross Browser Testing 5/15/2018 36© 2015, Perfecto Mobile Ltd. All Rights Reserved. https://info.perfectomobile.com/10-test- automation-frameworks-for-cross-browser- testing_mot.html
  • 34.
    Q & A ©2015, Perfecto Mobile Ltd. All Rights Reserved.
  • 35.