Building the Ultimate Device Matrix
•
2 likes•5,314 views
Building a useful set of devices for testing apps requires significant knowledge of the Android ecosystem. Once assembled, the device matrix provides broad, efficient coverage with minimal investment.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Building the Ultimate Device Matrix
Similar to Building the Ultimate Device Matrix (20)
Recently uploaded
Recently uploaded (20)
Building the Ultimate Device Matrix
- 1. Picking The Right Set of Mobile Devices By Brian Kitchener So;ware Quality Architect bkitchener@prototest.com
- 2. Overview • About me • Some Background • The Problem • Understanding Android • How Apps Work • Building a Device Matrix • Example Matrices • Conclusion
- 3. About Me • So;ware Quality Architect at ProtoTest • We're a mobile test lab that combines usability tesMng with quality assurance to culMvate a great user experience • Project Architect, Technical Lead, Trainer. • Started in QA in 2001 • BA in Applied CompuMng from University of Denver • TesMng background : FuncMonal, Performance, UAT, Security, API, Database. • AutomaMon : Selenium, WebDriver, WaMN, MonkeyTalk, SOASTA, Fitnesse, QTP, EggPlant, Squish • Languages : C#, Java, Ruby, Javascript
- 5. Some Stats for 2012 • Mobile Apps achieved $17 billion in sales • 5.2 Mobile Subscribers – 1.2 Billion PC’s – 4.2 Billion people use a toothbrush – 1 Billion Smartphones • 722 Million Smartphones sold • 1.4 Million iOS + Android Apps • 25 developers = half of app revenue
- 6. iPhone -‐ June 2007
- 7. About the iPhone • Steve Ballmer : Microso; CEO – “There’s no chance the iPhone is going to gain significant market share. No chance.” • Patrick Stewart: – “Last Wednesday, I stupidly dropped my iPhone in the bath, and my life has sort of spiraled almost out of control.” • Jon Rubinstein – Palm CEO – Is there a toaster that also knows how to brew coffee? There is no such combined device, because it would not make anything be;er than an individual toaster or coffee machine. It works the same way with the iPod, the digital camera or mobile phone: it is important to have specialized devices. • Mike Lazaridis – Blackberry CEO – And so what [the iPhone] has actually done is increased our sales.
- 8. ANDROID IS THE PROBLEM
- 9. And Then There Were Two… • Android unveiled November 2007 • First device was sold in October 2008. • Over 11,000 models have been released. • 48 Billion app installs • Over 1 Billion Android devices acMvated • 8 OS Revisions
- 10. OS FragmentaMon
- 12. Let’s do some math! • 16 device display categories • 20 different common resoluMons • 8 OS versions • 6 Hardware Manufacturers • 4 Major cellular networks • 16 x 20 x 8 x 6 x 4 = 76,800 permutaMons • Pairwise approach = over 30 permutaMons • Who can afford to increase tesMng by 30X?
- 13. Our Approach • Efficiency, not coverage • Flexible: support small or large number of devices • Understand how apps work to logically select criteria • Use Market research to pick most common configuraMons • Pick minimum and maximum boundary values for each criteria • Choose a value that matches an edge case or abnormal configuraMon. • Pick values that stress or tax the system
- 15. Android • Built and Maintained by Google • Open Source • Built on Linux kernel • ARM • X86 Ports • Built to support almost any type of device – Phones, tablets, phablets, media players, tv’s, watches, etc. • Device Manufacturers customize code.
- 16. Example: Kindle Fire • Forked Android 2.3 – Not updateable • Customized UI • Separate App store • Not all android apps work • Custom web browser
- 17. The OperaMng System • Google Releases “stock” versions • 10 Major Releases since 2008 – API Level, not Version • Device manufacturers like to customize the OS – Drivers, libraries, UI • “Stock” OS available in Nexus devices or an Emulator
- 18. Simulators / Emulators • Simulator imitates the so;ware layer – OS and Libraries – Apple provides a simulator in xCode IDE • Emulator duplicates the hardware and so;ware – Processor and Memory – Cannot mimic GPU, GPS, accelerometer • Always run stock OS • Can be used to test some funcMonality • Should always test on a physical device too
- 19. The Processor • ARM RISC-‐based instrucMon set • SpecificaMon defined by ARM holdings • 32 bit • Same as iOS • X86 patches and ports • It’s only a spec, can be modified.
- 20. SoC – System On A Chip • Main Board • Processor • RAM Bus • GPU • May include : – Cellular – WiFi – NFC – GPS – Bluetooth
- 21. 2 Samsung Galaxy S4s Quallcomm Snapdragon • Quallcomm Krait 300 – Quad core ARMv7 Cortex A15 Architecture • Adreno 320 GPU • Dual Channel 533Mhz Bus • Integrated LTE Samsung Exynos 5 Octa • Samsung Big.livle processor – Quad core Cortex A 15 – Quad core Cortex A7 • PowerVR SGX 544 GPU • Dual Channel 800Mhz Bus • No Integrated LTE
- 22. Common SoC Manufacturer Device Name Cores Processor GPU Qualcomm Snapdragon S4 2 or 4 ARM Cortex-‐ A15 Adreno Nvidia Tegra 3 4+1 ARM Cortex-‐ A9 Geforce Samsung Exynos 4 2 or 4 ARM Cortex-‐ A9 Mali Intel Medfeld 1 Intel x86 PowerVR Texas Instruments OMAP 4 2+2 ARM Cortex A9 PowerVR ST-‐Ericcson NovaThor 2 ARM Cortex-‐ A9 PowerVR
- 23. ResoluMon is not enough • Unlimited number of screen sizes available • Screens range from 3” to 11” • Each screen has a resoluMon, same as a monitor – If you increase the resoluMon everything shrinks! • Pixels per Inch = Density • Screen Size + Density = Display Bucket • ResoluMon is not enough!
- 24. The Display Buckets Size : • Xlarge : 8” -‐ 10” tablet. • Large : 5” -‐ 7” tablet. • Normal : 3.5” -‐ 5” phones. • Small : 3” -‐ 3.5” phones. Density : • ldpi = Low DPI (~120) • mdpi = Medium DPI (~160) • hdpi = High DPI (~240) • xhdpi = Extra High DPI (~320) Low density (120), ldpi Medium density (160), mdpi High density (240), hdpi Extra high density (320), xhdpi Small screen QVGA (240x320) 480x640 Normal screen WQVGA400 (240x400) WQVGA432 (240x432) HVGA (320x480) WVGA800 (480x800) WVGA854 (480x854) 600x1024 640x960 Large screen WVGA800* * (480x800) WVGA854* * (480x854) WVGA800* (480x800) WVGA854* (480x854) 600x1024 Extra Large screen 1024x600 WXGA (1280x800) † 1024x768 1280x768 1536x1152 1920x1152 1920x1200 2048x1536 2560x1536 2560x1600
- 25. Display Buckets • Galaxy S3 – 1280 x 720 – Xhdpi density (331ppi) – Normal screen (4.7”) • Galaxy Tab 10.1 – 1280 x 800 – ldpi density (149ppi) – Xlarge sceren (10.1”) • Galaxy Note LTE – 1280 x 800 – hdpi density (285ppi) – Large Screen (5.5”)
- 26. Market Analysis
- 27. Aspect RaMo • UI is manipulated from code • Density Pixels adjust for screen size – But can use regular pixels! • Need to take both X and Y into account! – Easy to overlap or hide things • Includes orientaMon • Some devices include an aspect raMo changer! (LG OpMmus Vu)
- 28. Cellular Carrier • Four Major US Networks – Verizon, Sprint, AT&T, T-‐Mobile – Some phone interoperability – 2 protocols • GSM – T-‐Mobile AT&T • CDMA – Verizon and Sprint – Carriers assigned specific frequency bands – LTE will be new standard -‐ But spectrum issues will prevent cross-‐network phones • So if the phone supports the carrier’s protocol and band it can theoreMcally connect.
- 29. HOW APPS WORK
- 30. How Apps work • Apps need to work on all screen sizes – May not be funcMonal – May be wasted space – May not make sense • Apps define XML layouts similar to HTML – Node structure – StaMc Content – Images, etc – Dynamic Content – Color, Text, etc.
- 31. Layouts and Fragments • XML Fragments are reusable components • Layouts sMtch together fragments for a specific sized device • App may need different flow for tablet vs phone
- 32. BUILDING THE DEVICE MATRIX
- 33. Our Criteria • OperaMng System – OS customizaMons, missing libraries, driver issues, • Screen Size – Rendering issues, usability, missing layouts • Pixel Density – Density Independence, missing layouts. • Aspect RaMo – X,Y calculaMons, overlapping panels, display issues • SoC – Hardware performance, InstrucMon set, bavery, signal • Carrier – Network protocol, speed, responsiveness, packet loss
- 34. The Goal • Efficiency, not coverage! • Build a set of devices to be used for app and website tesMng. • Know when to update them • Define a list of simple categories of devices • Pick devices that offer broad coverage • Adjust the number of devices based upon needed coverage
- 35. Categorical Approach • Define scope – Android, iOS, phone, tablet, etc. • Understand TesMng requirements • Self-‐descripMve Names • Help to broaden coverage • Will adjust devices chosen to cover our criteria • Should be apparent when to update a device • Spread coverage : – Usage -‐> Edge Cases -‐> Strange -‐> Stress
- 36. Example Categories • Common – Matches most common display configuraMon • Newest – Latest OS version, largest screen, highest resoluMon • Oldest – Oldest, slowest, smallest device. • Abnormal – Non-‐standard OS, aspect raMo, orientaMon, size • Popular – Most popular device in terms of sales • Budget – Low-‐priced new model. Tend to have strange specs • Flagship – Nexus device running stock Android OS • Catch-‐All – Cover any missing criteria
- 37. Android Phone Matrix March 2012 Device Name OS Display Aspe ct SoC Carrier Newest HTC Droid DNA 4.2 Normal-‐xhdpi 9:16 Snapdragon S4 Verizon Oldest HTC Tavoo 1.6 Small-‐ldpi 3:4 Snapdragon S1 AT&T Common Motorola Droid 3 2.3 Normal-‐hdpi 9:16 TI OMAP 4 Verizon Popular Samsung Galaxy S3 4.1 Normal-‐xhdpi 9:16 Exynos 4 Sprint Abnormal LG OpMmus VU 4 Large-‐hdpi 3:4 Nvidia Tegra 3 Tmobile Flagship LG Nexus 4 4.2 Normal-‐xhdpi 3:5 Snapdragon S4 TMobile Budget Dell Venue 2.2 Normal-‐mdpi 3:5 Snapdragon S3 AT&T Catch-‐All Sony Xperia P 2.3 Normal-‐hdpi 9:16 Sony NovaThor AT&T
- 38. iOS Matrix March 2012 Device Name OS Display Aspect SoC Carrier Newest iPhone 5S 7 4” 1136 x 640 326ppi 9:16 Apple 64bit A7 T-‐Mobile Oldest iPhone 3g 6 3.5” 320 x 480 165ppi 2:3 Apple A3 AT&T Common iPhone 5 6 4” 1136 x 640 326ppi 9:16 Apple A5 Verizon Popular iPhone 4 6 3.5” 640x960 330ppi 2:3 Apple A4 Sprint iPad (ReZna) iPad 3 7 10” 1536x2048 264ppi 3:4 Apple A5X Verizon iPod iPod Touch (4th gen) 5 3.5” 640x960 326ppi 2:3 Apple A4 WiFi Mini iPad Mini 6 7” 1024 x 768 162ppi 3:4 Apple A5 AT&T
- 39. Conclusion • Understanding how everything works allows us to logically select devices. • A large number of permutaMons can be covered in few devices • If addiMonal coverage is needed addiMonal devices can be added • White Paper : – hvp://www.prototest.com : – Building the UlMmate Device Matrix