Let’s build a mobile app quality and testing strategy together. Whether you have a web, hybrid, or native app, building a quality and testing strategy means (1) knowing what data and tools you have available to make agile decisions, (2) understanding your customers and your competitors, and (3) testing your app under real-world conditions. Jason Arbon guides you through the latest techniques, data, and tools to ensure the awesomeness of your mobile app quality and testing strategy. Leave this interactive session with a strategy for your very own app—or one you pretend to own. The information Jason shares is based on data from Appdiff’s next-gen mobile app testing platform, lessons from Applause/uTest’s crowd, text mining hundreds of millions of app store reviews, and in-depth discussions with top mobile app development teams.

1. ME
Half-day Tutorials
Monday, April 30th, 2018
8:30 AM
Build Your Mobile App Quality and Test
Strategy
Presented by:
Jason Arbon
Appdiff, Inc.
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.stareast.techwell.com/

2. Jason Arbon
Appdiff, Inc.
Jason Arbon is the CEO of Appdiff, which is redefining how enterprises develop,
test, and ship mobile apps with zero code and zero setup required. He was
formerly the director of engineering and product at Applause.com/uTest.com,
where he led product strategy to deliver crowdsourced testing via more than
250,000 community members and created the app store data analytics service.
Jason previously held engineering leadership roles at Google and Microsoft and
coauthored How Google Tests Software and App Quality: Secrets for Agile App
Teams.

3. 4/17/17
1
App Quality and Testing
Build Your Strategy
Jason Arbon, CEO Appdiff.com

4. 4/17/17
2
Problem
Fix?

5. 4/17/17
3
Solution
Who Are You?

6. 4/17/17
4
Test Engineer
Developer
Dir of Product/Eng
Founder/CEO

7. 4/17/17
5
How Google
Tests Software
App Quality

8. 4/17/17
6
Context
Millions of App Store reviews
Chats with Top App Teams
Automating 1000’s of Apps
Building Apps

9. 4/17/17
7

10. 4/17/17
8
App Stores
Standardization
Helps Testing

11. 4/17/17
9
Standardization
Public User Feedback
Per-App Cross Metrics
Categories
Expectations
Structure
~45 Minute Sessions
25 min Learning
15 min Practicing
QnA As long as folks like

12. 4/17/17
10
Topics
Monsters
Attributes
Pillars and Actions
Performance
Automation
Toolbelt
Putting it all Together
Surprise Inside : Test Strategy Document

13. 4/17/17
11
LARP
Be a real app Tester
Break into teams of 2-5

14. 4/17/17
12
Online? http://AppTestStrategy.com
Online?

15. 4/17/17
13
Offline?
Larp: Pick app and Read up

16. 4/17/17
14
jason@appdiff.com
Next Topic:
Monsters

17. 4/17/17
1
App Quality and Testing
Monsters
Jason Arbon, CEO Appdiff.com
Mobile
Monsters

18. 4/17/17
2
Mobile is Different
Apps are Different

19. 4/17/17
3
Monsters
Top Sources of
Mobile App Issues
Users
No Targeting
On the go
Expectations

20. 4/17/17
4
UX
Tiny
Two-Tap
Gestures
Security
Leaks
Treasure Trove
Expectations

21. 4/17/17
5
Reviews
Reflect on you
Public
Can’t please all
Agile/CI/CD
No Time
Too many builds
Tech Debt

22. 4/17/17
6
Real World
Network
Battery
IoT
Metrics
Too Many
Accountable
Evidence

23. 4/17/17
7
Devices
Matrix
Services
Expectations
Distribution
Store Front
Updates
INTL

24. 4/17/17
8
Competition
Many
Avoid
Steal
LARP: Monsters
Pick Top 3 Monsters for Your App
Pick Top 2 Mitigations

25. 4/17/17
9
LARP: Monsters
jason@appdiff.com
Next Topic:
Attributes

26. 4/17/17
1
App Quality and Testing
Attributes
Jason Arbon, CEO Appdiff.com
Mobile
Attributes

27. 4/17/17
2
Attributes
What Users
Care About
Source
Analysis of 200M+ Reviews
1M+ Bugs
10K+ Test Cycles

28. 4/17/17
3
Elegance
“Design is Garbage”
Usability
“Difficult to Navigate”

29. 4/17/17
4
Stability
“Crashes with new
Update!”
Privacy
“Will not remove
financial information”

30. 4/17/17
5
Security
“Can’t login anymore”
Performance
“Slow and frustrating”

31. 4/17/17
6
Satisfaction
“Not cool anymore”
Interop
“Won’t Connect…”

32. 4/17/17
7
Content
“Bots and fake
profiles”
Pricing
“Charged for driver
who didn’t show!”

33. 4/17/17
8
What to Focus on?
What to Focus on?

34. 4/17/17
9
For your App’s Category?
What to Focus on?

35. 4/17/17
10
What to Focus on?
LARP: Attributes
Pick Top 3 Attributes for Your App
Pick Top 2 Mitigations

36. 4/17/17
11
LARP: Attributes
jason@appdiff.com
Next Topic:
Pillars and Actions

37. 4/17/17
1
App Quality and Testing
Pillars and Actions
Jason Arbon, CEO Appdiff.com
Soft Science of Quality

38. 4/17/17
2
Pillar: Users
Software Exists to
Serve Users

39. 4/17/17
3
Users: Actions
Beta
Regression
Automation
Exploratory
Beta
Infrastructure
Enroll
Feedback
Support

40. 4/17/17
4
Regression
Define
Execute
Respond
maintain
Exploratory
Define tours
Explore
Share Findings

41. 4/17/17
5
Automation
Identify
Build
Respond
Maintain
Pillar: Data
Objective Insight
into Quality and
Value

42. 4/17/17
6
Data: Actions
Telemetry
Analysis
Impact
Projection
Telemetry
Identify
Add
report/alert
maintain

43. 4/17/17
7
Analysis
Hypothesis
Measure
Findings
Apply and share
Impact
Identify
What-if
Preventative
Capitalization

44. 4/17/17
8
Projection
Metrics
Low/High
Key Predictions
Share
Pillar: Biz
Why invest in
quality?

45. 4/17/17
9
Biz: Actions
KPIs
Competitive
E2E
Cost and speed
KPIs
Relate Test to KPIs
Measure
Trend
Impact

46. 4/17/17
10
Competitive
Identify
Use
Predict
Borrow
E2E
Upstream
Downstream
Before
After

47. 4/17/17
11
Cost + Speed
Features
Processes
Grease
Tooling
Pillar: Team
Team is Everything
Team does
Everything
Nothing happens
without the team

48. 4/17/17
12
Biz: Actions
Care
Learn
Visibility
Reward
Care
Hire passion
Customers
Team
Career

49. 4/17/17
13
Learn
Cross-train
Technical
Customer visit
Support
Visibility
Kudos (public)
Kudos (private)
Dashboard
Monetary

50. 4/17/17
14
Reward
20%
Gift Certificates
Raise/Promo
Opportunities
LARP: Most Important Pillars
Pick Top 2 Pillars
Pick Top 2 Actions

51. 4/17/17
15
jason@appdiff.com
Next Topic:
Performance

52. App Quality and Testing
Application Performance Monitoring
Jason Arbon, CEO Appdiff.com

53. Production APM

54. APM
Server Performance Monitoring
In-App Performance Monitoring
Crash Collection
Engagement/Usage

55. SDKs
Image from http://www.analyticshero.com/

56. SDKs
Install SDK + Code: App and Server
Maintain/Update
Can cause crashes and perf issues themselves
Several in same app can conflict

57. AppDynamics

58. Twitter Fabric

59. New Relic

60. Runscope

61. Limitations of Production APM
Issues found *after* user impact
Difficult to interpret graphs, charts, dials
Measure API and Code--Not User Perception

62. Pre-Production, User-Focused APM

63. Why Pre-Production, User-Focused APM?
Catch issues *before* user impact
Measure user-perceived performance
Zero-Code, Zero SDK.
With CI, know which code change impacted performance

64. Appdiff: Automatically walk app as user

65. Catch Performance Regressions

66. Performance Perf Feature Area

67. Appdiff: Trends Over Time

68. End User Production Monitoring

69. LARP: Automation
Decide whether to use APM
Which APMs?

70. jason@appdiff.com
Next Topic:
Automation

71. 4/17/17
1
App Quality and Testing
Automation
Jason Arbon, CEO Appdiff.com
Many Try
Many Fail
..and try again

72. 4/17/17
2
Overall
Some teams (~30%) have unit tests (run in <1 minute)
Few teams build and maintain UX-based tests
Biggest App teams often have <50 automated tests
Talent
Unicorns: Hard to Find
Developer+Tester
Tornado: Harder to retain as
they move into Development
roles
Expensive!

73. 4/17/17
3
Automation Breaks when app Changes
Record and Playback?
Demos well
False Hope
Brittle
Doesn’t scale
Often Proprietary

74. 4/17/17
4
I was warned and accept
responsibility for trying
because it sounds cool.
Current Generation

75. 4/17/17
5
Calabash
Cross Platform
Mobile only
Ruby-only
Cucumber
More Layers
Debugging
Not for Testing!

76. 4/17/17
6
Creator Doesn’t Want you To Use it!
Cucumber reached a million downloads in the first three years and 5 million
downloads 3 years later. I'm happy to have created such a popular tool, but
saddened to see how it's misused and misunderstood.
If you think Cucumber is a testing tool, please read on, because you are wrong.
Espresso
Android Only
New/Fresh
Sync with UX

77. 4/17/17
7
Appium
Cross Platform
Web Too!
Open
Any language
Still Fragile
Test Execution

78. 4/17/17
8
Devices vs Emulators
Expensive/Painful
Scaling issues (even cloud)
Sleep well at night
Finally ‘fast enough’
Almost Functionally
Equivalent
Cheap
Scales
Your own device lab

79. 4/17/17
9
Your own device lab
Feel “in control”
Scratch
As a Service
Cloud Device Lab

80. 4/17/17
10
Cloud Devices via API
Cloud Simulators

81. 4/17/17
11
Device Matrix
Usage and Crash Data Will Tell you
Don’t Stress it
Reviews
What to Automate?
Machines: Good at Quantitative
Human: Good at Qualitative
Don’t just automate manual test cases

82. 4/17/17
12
Next Generation
Test Development
and Execution
as a service
Type Test Cases into their Web UX
They write and Run the tests for
you
Not mobile today, but soon?

83. 4/17/17
13
Still a Fundamental Problem

84. 4/17/17
14
Automated
App
Automation
No Code. No Setup.
Auto-Crawl
Focus on what changes build to
build
UX + Performance
Appdiff: Automatically walk app as user

85. 4/17/17
15
Appdiff: Auto-Automation Coverage
Appdiff: Auto-Automation Coverage

86. 4/17/17
16
Appdiff: Auto-Automation Coverage
Appdiff: Test Results

87. 4/17/17
17
LARP: Automation
Automate or Not?
Cloud or Local?
Devices or Emulators?
jason@appdiff.com
Next Topic:
Toolbelt

88. App Quality and Testing
Toolbelt
Jason Arbon, CEO Appdiff.com

89. Other Recommended
Tools and Infrastructure

90. Build

91. BuddyBuild

92. CircleCI

93. Beta Management

94. AppSee

95. BuddyBuild

96. Review Management

97. Applause
Analytics
Auto Quality
Attributes
Clustering
Competitive
Analysis
Review Search

98. Applause Analytics

99. AppBot

100. User Feedback

101. Apptentive

102. Engagement / Analytics

103. Mixpanel

104. Heap

105. Crowd/Panel Testing

106. Applause

107. UserTesting.com

108. LARP: Toolbelt
Chose 2 Tools

109. jason@appdiff.com
Next Topic:
Stitching

110. 4/17/17
1
App Quality and Testing
Stitching Strategy
Jason Arbon, CEO Appdiff.com
Putting it all together

111. 4/17/17
2
A Balanced App Test/Quality Strategy
Address several Monsters
Address several Attributes
Address several Pillars and Actions
Pick pre and post production APM solution
Pick Automation Strategy/Tools
Pick 2 General Quality Tools
Decide on Testing
Cadence
Testing Cadence
Continuous Integration?
Agile?
Keys:
● Focus on Changes
● Rolling Passes
● Automation
● Metrics

112. 4/17/17
3
You have a Test Strategy now
jason@appdiff.com
linkedin.com/in/jasonarbon
signup@
appdiff.com/mdt