In the world of mobile app testing, data is your friend. So harness your data to your advantage to create an automation and testing strategy. Satyajit Malagu acknowledges that multiple devices, platforms, languages, crashes, bugs, and app stores make the scope of mobile app testing humongous. When you add in inherent human biases and team dynamics, the problems you face and prioritization challenges can be overwhelming. The data collected from analytics, bug trends, monitoring tools, test results, and other sources can help illuminate a clearer path. Join Satyajit as he provides an overview of the tools available to ensure a quality iOS/Android app. Discover a systematic way to determine which tool is suitable for which phase and context of each mobile project. Leave with a rubric on how you can direct your testing and automation efforts based on your collected data.

1. T2
Session - Mobile Testing
4/27/17 10:00 AM
A Data-Driven Approach for Mobile
Testing and Automation
Presented by:
Satyajit Malugu
GoDaddy
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - https://www.techwell.com/

2. Satyajit Malugu
GoDaddy
A senior SDET at GoDaddy, Satyajit Malugu is focused on mobile testing. His work
involves automation of native and hybrid apps, and providing test perspective to
his team that recently converted from waterfall to agile. As a testing leader in a
company that is deploying a suite of native apps, Satyajit is involved with
strategizing, executing, and mentoring other testers and stakeholders on best
practices for native mobile testing. Previously, Satyajit worked at Urbanspoon
and gained SDET knowledge at Microsoft. He frequently presents on mobile
topics. Read his blog and follow him on Twitter at @malugu.

3. 5/2/17
1
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Utilizing data for mobile testing
and automation
by Satyajit Malugu
Sr.SDET at Godaddy
04/27/2017
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
About me
• SDET for 10 years, focused on mobile apps
• Primary tester for mobile apps at
• Urbanspoon
• Godaddy domain finder/ legacy apps/ Investor apps
• Godaddy Smartline

4. 5/2/17
2
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Challenges for quality decisions
• Multiple stakeholders and their viewpoints
• Product
• UX
• Marketing
• Developers
• Executives (high visibility)
• Multiple sources of bug reports
• Internal
• External
• Project deadlines
These are common for any software project
3
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Additional challenges for mobile apps
• UI Automation/build system flakiness
• Test builds distribution troubles (iOS)
• Testflight
• Certificates
• App store deployment wonkiness
• Multiple entry points into app
• Push notifications
• Widgets
• We develop app for an OS, not for an app(browser) on the OS
• Memory management
• Battery etc.
4

5. 5/2/17
3
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
What is a data point
• An artifact that provides information that is:
• Quantifiable
• Measurable
• Actionable
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Why is it important
• Data helps in making objective decisions
• Respect for bug reports
• As the solo critics on the team, data gives you additional support
• Reduce the impact of biases that are ever-present
• Availability(now) bias

6. 5/2/17
4
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Data sources
7
1 Historic
3 During development
2 After deployment
4 External
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Historic
8

7. 5/2/17
5
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Historic
• From other apps within the company
• Previous versions
• Device matrix
• Bug trends
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
After deployment
10

8. 5/2/17
6
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Crash monitoring
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Customer feedback
• App store reviews
• Slack bots
• Summaries
• Customer support
• Other feedback channels
• Emails
• Within apps

9. 5/2/17
7
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Usage monitoring
• Fabric, GA, Google play analytics
• API access logs
• Instrumentations
Flurry flowsFabric analytics
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Adoption of versions
Force upgrade strategy

10. 5/2/17
8
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
During development
15
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Network proxies
• Able to intercept ‘all’ traffic to and from the apps
• Charles/Fiddler
• SSL proxying
• Look/edit network requests
• Rewrite rules

11. 5/2/17
9
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Database/network inspection
• Stetho (android only)
17
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Loggers/Videos
• Android - logcat
• iOS - console
• Vysor – Android videos/screenshare/screenshots
• Quicktime- iOS videos
• Great info for logging bugs

12. 5/2/17
10
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Monitors
19
• Android Monitor
• Xcode Instruments
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
CI pipeline
• Build jobs
• Test results
• Identify regressions
• Find intermittent crashes

13. 5/2/17
11
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
External
21
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
External
• Android official dashboards - https://developer.android.com/about/dashboards/index.html
• iOS official stats - https://developer.apple.com/support/app-store/
• Per country usage from perfecto -
http://info.perfectomobile.com/digital-test-coverage-index.html?utm_source=tci-blog
• Top mobile devices
• Most used apps across categories
• Consolidated release calendar
• Mixpanel - https://mixpanel.com/trends/

14. 5/2/17
12
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Usage scenarios
23
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Device lab planning and budgets
24
• Problem:
• How to plan out device labs
• What and when to acquire devices
• Data sources:
• Historic
• Internal company device matrix
• External
• Android/iOS dashboards
• Release calendar

15. 5/2/17
13
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
CI/CD pipeline
Lint
checks
Unit tests
Espresso
activity
tests
Espresso
smoke
tests
Manual
code
review
Merge to
main
branch
Make decisions automatic
• Problem: Fast feedback during
development
• Data sources:
• Static analyzers
• Unit tests
• UI Tests
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Checklists for releases
• Problem: Are we ready to release?
• Data sources:
• Bugs – priority, trends, open
• Test results – automation, manual
• Code coverage
• Solution: Jira/confluence integration, confluence dashboard

16. 5/2/17
14
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Push notification stability
27
• Problem: How reliable are the push notification delivery
• Data sources:
• Adhoc/ logging
• Solution:
• Collecting and analyzing in a huge excel
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Automation Stabilization
28
• Problem: Finding flaky tests
• Data sources:
• Build jobs
• Solution:
• Test results analyzer
• Slack integration
• Problem: Is this new test flaky
• Data sources:
• Build jobs
• Test runs
• Solution:
• Run tests ’n’ times before promoting

17. 5/2/17
15
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Quality dashboards
• Problem: One pager status report
of quality
• Data souces:
• Builds jobs
• Errors
• Bugs
• Solutions:
New Relic
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Effective bug reports for devs
• Problem: Devs not paying enough attention to bug reports, multiple to-fro conversations
• Data sources:
• Loggers with stack traces
• Network request/responses
• Screenshots/videos
• Database state, memory usage etc.
• Solution: Along with steps to reproduce include other artifacts
30

18. 5/2/17
16
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Effective bug reports for product owners
• Problem: Product folks not paying enough attention to bug reports deprioritizing bugs
• Data sources:
• Customer feedback (reviews, emails)
• Device usage matrix
• Crash reports
• Solution: Along with steps to reproduce include data to help evaluate the impact and risks
31
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Takeaways
• Multiple data points and use cases given
• Analyze your project and come up with a data plan for your project
• Completely depends on your project, company, culture
• What worked for me won't necessarily work for you

19. 5/2/17
17
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Thank you
• Share your experiences
• Comments
• Questions
33
http://mobiletest.engineer
@malugu