Mobile Device
Testing Strategies
Stephen Frein
Comcast
5/8/2014
About Me
• Director of QA for Comcast.com
• Adjunct @ Drexel University
• https://www.linkedin.com/in/stephenfrein
• steph...
• Largest mass media and communications
company in the world (by $)
• Largest cable company / ISP in USA
• Largest converg...
Stuff We'll Talk About
• Importance of mobile
• What should I test on?
• Where do I keep devices?
Mobile Is BIG
Mobile Matters (More Than Ever)
• US retail mobile ecommerce sales increased
68% from 2012 to 2013 (1) *
• US mobile web u...
But Desktop Still Rules Web
• Most mobile Internet usage (80+%) is
driven by apps (4)
• Mobile usage accounts for about 30...
Punchline
Mobile is growing in importance, but your
web testing efforts should probably
emphasize desktop for the foreseea...
When Will That Change?
Not in the next year . . .
Where Should I Focus?
7.1
7.0
6.1
Gingerbread
KitKat
Jellybean
What My Team Does
• Establish coverage thresholds (browser /
OS / device type must have x% traffic)
• Tune test coverage b...
Balancing Stats
Your Stats
(the focus)
Industry Stats
(a counterweight)
Google Analytics, Adobe, Piwik, etc
StatCounter, C...
Industry Stats Useful Links
http://gs.statcounter.com/#mobile+tablet-
browser-US-monthly-201304-201404
http://www.netmarke...
General Observations
• Today, iOS and Android are the real show
• New iOS versions have rapid adoption
• Android versions ...
Where to Test?
Physicals
Cloud
Emulators
Help Me Choose
Approach Pros Cons
Physicals • Realism
• Audio / Video
• Geo-lock
• Management
Cloud • Geo-flex
• Managemen...
We Use Cloud (Mostly)
• International team
• “Hide and seek” got old
• Audio / video not a priority for us
• Good automati...
Whose Cloud Offering?
Start by looking at these two . . .
. . . but there are many others.
Remember
• Mobile matters, but desktop owns web
• Stats key to your device selection
– Android phones and iPhone / iPad ar...
Wrap-up
Questions?
Thanks!
Endnotes
1. http://www.forbes.com/sites/chuckjones/2013/10/02/ecommerce-is-growing-nicely-while-mcommerce-is-on-a-tear/
2....
Development and testing By: Stephen Frein
Upcoming SlideShare
Loading in...5
×

Development and testing By: Stephen Frein

324

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
324
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Development and testing By: Stephen Frein

  1. 1. Mobile Device Testing Strategies Stephen Frein Comcast 5/8/2014
  2. 2. About Me • Director of QA for Comcast.com • Adjunct @ Drexel University • https://www.linkedin.com/in/stephenfrein • stephen.frein@gmail.com • www.frein.com
  3. 3. • Largest mass media and communications company in the world (by $) • Largest cable company / ISP in USA • Largest converged network in the world (data / voice / video) • Comcast.com in April of 2014: o 200 million + page views o 25+ million unique visitors
  4. 4. Stuff We'll Talk About • Importance of mobile • What should I test on? • Where do I keep devices?
  5. 5. Mobile Is BIG
  6. 6. Mobile Matters (More Than Ever) • US retail mobile ecommerce sales increased 68% from 2012 to 2013 (1) * • US mobile web usage rose by a similar amount (2) • More than half of Amazon holiday shoppers used a mobile device in 2013 (3) * Psst! The (1) is an endnote.
  7. 7. But Desktop Still Rules Web • Most mobile Internet usage (80+%) is driven by apps (4) • Mobile usage accounts for about 30% of web traffic (5)
  8. 8. Punchline Mobile is growing in importance, but your web testing efforts should probably emphasize desktop for the foreseeable future. (Apps are their own animal.) Desktop > Mobile
  9. 9. When Will That Change? Not in the next year . . .
  10. 10. Where Should I Focus? 7.1 7.0 6.1 Gingerbread KitKat Jellybean
  11. 11. What My Team Does • Establish coverage thresholds (browser / OS / device type must have x% traffic) • Tune test coverage based on usual factors: o Historical defect discovery rates o Nature of changes o Risk of leakage o Schedule pressure • Pray we got it right
  12. 12. Balancing Stats Your Stats (the focus) Industry Stats (a counterweight) Google Analytics, Adobe, Piwik, etc StatCounter, Chitika, Android, etc
  13. 13. Industry Stats Useful Links http://gs.statcounter.com/#mobile+tablet- browser-US-monthly-201304-201404 http://www.netmarketshare.com/ (some $) http://developer.android.com/about/dashb oards/index.html http://david-smith.org/iosversionstats/
  14. 14. General Observations • Today, iOS and Android are the real show • New iOS versions have rapid adoption • Android versions more fragmented • Android world is heavily phone-skewed • “Default” browsers the focus • Mobile Chrome creeping up in usage • Don’t ignore Windows entirely, but it is a lower priority • Other flavors may not be worth it
  15. 15. Where to Test? Physicals Cloud Emulators
  16. 16. Help Me Choose Approach Pros Cons Physicals • Realism • Audio / Video • Geo-lock • Management Cloud • Geo-flex • Management • Automation • Cost • Audio / Video • Speed Emulator • Cheap • Fast • Realism • Realism!!!
  17. 17. We Use Cloud (Mostly) • International team • “Hide and seek” got old • Audio / video not a priority for us • Good automation support • Private cloud helps o Automation o Test environment access “Who has the &%*@! iPad?”
  18. 18. Whose Cloud Offering? Start by looking at these two . . . . . . but there are many others.
  19. 19. Remember • Mobile matters, but desktop owns web • Stats key to your device selection – Android phones and iPhone / iPad are must-do – Set percentage standards, but then tune based on other factors • Don’t rely on emulators – consider cloud solutions for device hosting
  20. 20. Wrap-up Questions? Thanks!
  21. 21. Endnotes 1. http://www.forbes.com/sites/chuckjones/2013/10/02/ecommerce-is-growing-nicely-while-mcommerce-is-on-a-tear/ 2. http://bgr.com/2013/11/08/mobile-devices-internet-traffic-q3-2013/ 3. http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1886961&highlight= 4. http://www.smartinsights.com/mobile-marketing/mobile-marketing-analytics/mobile-marketing-statistics/ 5. http://www.walkersandsdigital.com/Walker-Sands-Mobile-Traffic-Report-Q3-2013
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×