Stop Testing (Only) The Functionality of Your Mobile Apps!

Stop Testing (Only) The
Functionality of Your Mobile Apps!
Anand Bagmar & Justin Ison
@BagmarAnand & @isonic1

Why do we need to think diﬀerently
about Mobile App Testing?
2

Diﬀerences in Testing approach for
Web & Native Apps Testing
3

● Interactions
● Device features & capabilities
○ Form factors (sizes, rotations)
○ M-Browsers Vs Native
○ OS variations & compatibility with hardware
○ Hardware
■ Battery
■ CPU / GPU
■ Memory
● Impact due to External factors
○ Network ﬂuctuations & variations when on the move
○ Temperature / Humidity / Moisture
○ Interruptions
● Release Approach
Diﬀerences In Web & App Testing Approach
4

Diﬀerences in Release approach for
Web & Native Apps Testing
5

Web Mobile
Rollback Can be achieved Not possible
Upgrade scenarios (OS flavors) Easy to validate Complex
Hot-fix Easy to deploy Can’t be forced on users
External Approvals Not required
Apple / Google App verification
process
Deployment size Does not matter to users App size has impact on users
Compliance Browser compliance
Stock / Custom OS + Hardware
compatibility + Permissions
Diﬀerences in Web & App Release approach
6

Web Mobile
Upgrade scenarios (OS flavors) Easy to validate Complex
process
7

https://www.npr.org/sections/goatsandsoda/2014/12/01/364749313/ebola-in-the-air-what-science-says-about-how-the-virus-spreads
8

Web Mobile
Upgrade scenarios (OS ﬂavors) Easy to validate Complex
process
9

Web Mobile
Hot-ﬁx Easy to deploy Can’t be forced on users
process
10

Web Mobile
Apple / Google App veriﬁcation
process
11

Web Mobile
process
Deployment size
Does not (necessarily) matter to
users
App size has impact on users
12

Web Mobile
process
Deployment size
Does not (necessarily) matter to
users
App size has impact on users
13

So, how can you add more value to the
quality of your Native Apps?
STRATEGY
14

Real Device Vs Emulator-based Testing
https://askanydifference.com/difference-between-emulator-and-simulator/
STRATEGY
15

● When to use Emulators / Simulators?
○ Able to validate all application functionality
○ No performance impact on the application-under-test
● Why use Emulators / Simulators?
○ Reduce cost
○ Scaled as per needs, resulting in faster feedback
When & Why to use Emulators for Testing?
16

● When to use Real Devices for testing?
○ If Emulators / Simulators are used, then “Sanity” / Focussed testing on real devices before
release
○ If Emulators / Simulators cannot validate all application functionality reliably, then invest
in Real-Device testing
○ If Emulators / Simulators cause performance issues or slowness of interactions with the
application-under-test
When to use Real Devices for Testing?
17

● Cases when Emulators / Simulators may not help
○ Streaming content
○ Applications relying on hardware capabilities
○ Applications dependent on customised OS version
Real Device Vs Emulator-based Testing
18

● It’s not enough to develop an app or any digital content for
that matter. You should strive to make your application
accessible to everyone.
● When should you start? The thought of accessibility should
start at the design or discovery phase or any product
and/or features.
● On this topic, we’re going to talk about a few of them and
how you can test your app for accessibility. Some of the
tools available, and even how to automate some of these.
Mobile Accessibility Testing
STRATEGY
19
● According to the World Health Organization, there are
roughly 285 million people worldwide that suﬀer from low
vision, and/or color blindness.

Accessibility
The Good, The Bad & The Ugly
20

Text To Speech - Voiceover
● Both iOS and Android platforms oﬀer tools to verify voiceover logic.
○ Hearing the text-to-speech on the device/emulator/simulators.
○ Validating the accessibility labels via the layout or various hierarchy viewers apps.
“content-desc” for Android and “Label” for iOS. Both will result in “accessibility-id” in
appium for instance..
21
● Your goal is to ensure people with low vision can utilize your
application, it functions correctly and it makes sense audibly.

Detecting Accessibility Labels
ANDROID
22

Detecting Accessibility Labels
iOS
23

● The Android Studio and Xcode frameworks now for the most part handle layout
changes automatically due to diﬀerent text sizes.
○ However, problems can occur when designers or developers use custom
fonts. They like any humans, make subconscious decisions that may adversely
impact how your app looks, functions and accessible.
Large Text Font & Display Density Size
24
● For those with low vision, increasing the text and even the screen density size is
imperative. Normally these settings are enabled at the devices system level.
○ If these settings are changed from the default your app must be able to
support it.

Default Large Text Large Text & Display
25

● Somewhat controversial when it comes to accessibility. But nonetheless, if
there is a population of people needing this, you should support this
feature and make sure it works, and the content of your app is readable.
● A few beneﬁts:
○ Reduces ﬂicker problems
○ Easier to use in poorly lit rooms
○ Less blue light exposure
26

Bugs CAN Happen...
28

● Nowadays there are many digital platforms such as phones/tablets, smart TV’s,
smartwatches, it’s important we have set standards to ensure our apps are accessible
to users with color blindness.
● And we do! The W3C - Web Content Accessibility Guidelines (WCAG 2.0 / 2.1) are a set
of standards and guidelines for mobile accessibility.
● Along with Levels AA and AAA.
○ Level AA (contrast minimum): Requires a contrast of at least 4.5:1 or 3:1 for large scale text.
○ Level AAA (enhanced): Requires a contrast of at least 7:1 or 4.5:1 for large scale text.
Contrast & Color Blindness
29
● Many people throughout the world suﬀer from color blindness.

● The Accessibility Scanner app scans your screen and provides suggestions to improve the
accessibility of your app. This app uses the Accessibility Test Framework and provides speciﬁc
suggestions after looking at content labels, clickable items, contrast, and more.
● You can automate this with Espresso Accessibility Checking libraries.
○ https://developer.android.com/training/testing/espresso/accessibility-checking
Android - Tools Available to Validate Contrast
30

● Apple provides the Accessibility Inspector testing tool that can be launched from Xcode > Open
Developer Tool > Accessibility Inspector. The Accessibility Inspector presents a utility window
that displays the information properties (and values), action methods, and position in the
accessibility hierarchy.
● Automation options are limited, however.
○ https://mobilea11y.com/guides/xcui/
○ https://github.com/google/GSCXScanner
○ https://www.deque.com/ios-accessibility/
iOS - Tools Available to Validate Contrast
32

Applitools Contrast Advisor - All Platforms
34

Applitools - Validate Contrast from Code!
35

Accessibility
Now let’s tie everything we just learned
with automation example using Applitools!
Default - Contrast - Large Text - Device Density - Dark Mode
36

Android Device Bridge (ADB) APIs
37

Example Appium Script
38

Let’s look at the results in Applitools!
39

● Consumption Monitoring:
○ CPU
○ Memory
○ Network
○ Battery/Energy
● It's not only important to capture this information, but also storing it
for historical lookup, benchmarking, and detecting trends!
● Android SDK Tools & Xcode provide libraries to test these. We will walk
through the various options available.
Devices Resources & Consumption Testing
STRATEGY
40
● Nowadays, it's not enough to only have tests to ensure quality and
catch regressions. It's imperative to know more about what’s
happening under-the-hood of your application.

Android Studio - Proﬁler
41

42
Credit: https://appiumpro.com/editions/5-performance-testing-of-android-apps

Xcode - Instruments Proﬁler
43

44
Credit: https://appiumpro.com/editions/12-capturing-performance-data-for-native-ios-apps

46
Chaos & Monkey Testing

● You can also stress test your application with monkey testing. Monitor
the logs for errors and even use the proﬁling tools we talked about
previously.
● We’ll look at few diﬀerent frameworks available for iOS and Android.
○ These should be used as an additional tool in your toolbox and not to replace
automation.
Chaos & Monkey Testing
STRATEGY
47
● Testing the unexpected! Regressions tests (automated and manual) are
a must-have but they only take you so far. Using a monkey tester can
discover bugs you never knew existed.

iOS - Zalando’s SwiftMonkey
48
Credit Source: https://github.com/zalando/SwiftMonkey

● Perhaps one of the easiest frameworks to
get started with. You can develop your
own Monkeyrunner scripts in Python and
Java.
● Or use the Android Device Bridge APIs and
run it directly from a terminal.
Android - Monkeyrunner
49

Create Your Own Monkey Tester
51

● Many applications today use location-based
logic where they geolocate you to display
certain UI views.
● It’s important to validate these cases by
changing the device location.
● In some cases, this is not enough and a VPN
is needed if using network-based location.
Or use additional spoofer / mock location
apps to achieve this.
Location Testing
52

● Many applications today need to handle diﬀerent networks
and bandwidths. Especially in developing countries where
networks can be spotty.
● Apps need to logically detect when certain network
conditions are met and what view contents need to be
downloaded or otherwise show static views
● Bandwidth can be adjusted for automation as well. Some
links at the end will be provided for further explain how.
Network Bandwidth Testing
53

Visual Testing
STRATEGY
54

55
Bugs still Escape

56
Spot the Diﬀerence!

57
Spot the Diﬀerence!

59
Pixel Comparisons Waste Time With False Positives

60
Visual AI Highlights Only The Diﬀerences That Matter!

61
Use Visual Assertions Instead of Functional Assertions
● A single assertion for complete
functional coverage
● Bonus: validates the UI
● Does not break when the UI changes
● No coding skills required to maintain
baselines
● Validate UX at scale for all supported
browsers

Beta Testing
● Beta Testing
○ Test Release candidate apps with users willing to use and provide feedback for early
releases of Apps
○ Instrumentation / Reports need to be able to separate between Beta releases Vs
actual release versions of the App
STRATEGY
62

On-field Testing
● On-field Testing
○ Test Release builds of apps with real users identified in the field
○ Share specific criteria and scenarios for validations
Examples
■ Network and network conditions on which testing is to be done
■ Device criteria on which testing is to be done
■ Test scenarios to be executed with relevant Test Data
○ Leverage companies that offer on-field testing as a service
STRATEGY
63

Staged Rollouts
● Staged Rollouts
○ Release apps to a % of users
○ Rollout % can be increased over time based on conﬁdence on quality
○ App can be pulled out if problems are noticed / reported
○ Google PlayStore Staged Rollouts:
https://support.google.com/googleplay/android-developer/answer/6346149?hl=en
○ Apple Phased Releases:
https://help.apple.com/app-store-connect/#/dev3d65fcee1
STRATEGY
64

Observability - Logging / Monitoring / Analytics
https://www.splunk.com/en_us/data-insider/what-is-observability.html
STRATEGY
65

● Observability - The ability to dig into unknowns on the ﬂy
● To enable Observability, ensure:
○ Capture Crash logs with context
○ Build Instrumentation & Correlation with Context in your app
○ Capture user interactions in form of Business and Technical Analytics Events
○ Enhanced Real-time Monitoring & Alerting for your system components and
infrastructure
○ Build Meaningful, Real-time and Contextual Dashboards
● Test for Observability before a full release
66

67

Sources
● Accessibility:
○ https://www.w3.org/WAI/standards-guidelines/mobile
○ https://uxdesign.cc/accessibility-and-dark-ui-themes-f01001339b65
○ https://developer.android.com/guide/topics/ui/accessibility/testing
○ https://gist.github.com/ChiftKey/3901ff7702fec1f07c21f22e54533cde
○ https://github.com/rwapp/A11yUITests
○ https://github.com/google/GSCXScanner - for iOS
○ https://www.deque.com/ios-accessibility/
○ https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/OSXAXT
estingApps.html
○ https://applitools.com/docs/features/contrast-accessibility.html
● Device Resources & Consumption:
○ https://developer.android.com/studio/profile/android-profiler
○ https://appiumpro.com/editions/5-performance-testing-of-android-apps
○ https://help.apple.com/instruments/mac/current
○ https://appiumpro.com/editions/12-capturing-performance-data-for-native-ios-apps
○ https://medium.com/@andrew.tishchenko/how-to-create-fake-locations-for-ios-207950ee2410
○ https://nshipster.com/network-link-conditioner/
○ https://appiumpro.com/editions/72-simulating-slow-internet-connections-on-android-emulators-with-appium
○ https://appiumpro.com/editions/104-simulating-different-network-conditions-for-virtual-devices
● Monkey Testing:
○ https://developer.android.com/studio/test/monkeyrunner
○ https://github.com/zalando/SwiftMonkey
○ https://medium.com/mobile-quality/how-to-use-chaos-in-your-ios-tests-b49281a1349a
○ https://github.com/isonic1/Appium-Native-Crawler#example-usage
68

Thank you
Anand Bagmar & Justin Ison

Stop Testing (Only) The Functionality of Your Mobile Apps!

More Related Content

What's hot

Similar to Stop Testing (Only) The Functionality of Your Mobile Apps!

More from Applitools

Recently uploaded

Stop Testing (Only) The Functionality of Your Mobile Apps!