Building the Ultimate Device Matrix

  • 1,199 views
Uploaded on

Building a useful set of devices for testing apps requires significant knowledge of the Android ecosystem. Once assembled, the device matrix provides broad, efficient coverage with minimal investment.

Building a useful set of devices for testing apps requires significant knowledge of the Android ecosystem. Once assembled, the device matrix provides broad, efficient coverage with minimal investment.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • I have built something similar
    www.testingoncloud.com
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
1,199
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
31
Comments
1
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Picking  The  Right  Set  of   Mobile  Devices   By  Brian  Kitchener   So;ware  Quality  Architect   bkitchener@prototest.com  
  • 2. Overview   •  About  me   •  Some  Background   •  The  Problem   •  Understanding  Android   •  How  Apps  Work   •  Building  a  Device  Matrix   •  Example  Matrices   •  Conclusion  
  • 3. About  Me   •  So;ware  Quality  Architect  at  ProtoTest   •  We're  a  mobile  test  lab  that  combines  usability  tesMng  with  quality   assurance    to  culMvate  a  great  user  experience   •  Project  Architect,  Technical  Lead,  Trainer.       •  Started  in  QA  in  2001   •  BA  in  Applied  CompuMng  from  University  of  Denver   •  TesMng  background  :  FuncMonal,  Performance,  UAT,   Security,  API,  Database.   •  AutomaMon  :  Selenium,  WebDriver,  WaMN,   MonkeyTalk,  SOASTA,  Fitnesse,  QTP,  EggPlant,  Squish   •  Languages  :  C#,  Java,  Ruby,  Javascript  
  • 4. BACKGROUND   INFORMATION  
  • 5. Some  Stats  for  2012   •  Mobile  Apps  achieved  $17  billion  in  sales   •  5.2  Mobile  Subscribers   – 1.2  Billion  PC’s   – 4.2  Billion  people  use  a  toothbrush   – 1  Billion  Smartphones   •  722  Million  Smartphones  sold     •  1.4  Million  iOS  +  Android  Apps   •  25  developers  =  half  of  app  revenue  
  • 6. iPhone  -­‐  June  2007  
  • 7. About  the  iPhone   •  Steve    Ballmer  :  Microso;  CEO   –  “There’s  no  chance  the  iPhone  is  going  to  gain  significant  market   share.    No  chance.”   •  Patrick  Stewart:     –  “Last  Wednesday,  I  stupidly  dropped  my  iPhone  in  the  bath,  and  my   life  has  sort  of  spiraled  almost  out  of  control.”   •  Jon  Rubinstein  –  Palm  CEO   –  Is  there  a  toaster  that  also  knows  how  to  brew  coffee?  There  is  no  such   combined  device,  because  it  would  not  make  anything  be;er  than  an   individual  toaster  or  coffee  machine.  It  works  the  same  way  with  the   iPod,  the  digital  camera  or  mobile  phone:  it  is  important  to  have   specialized  devices.   •  Mike  Lazaridis  –  Blackberry  CEO   –  And  so  what  [the  iPhone]  has  actually  done  is  increased  our  sales.  
  • 8. ANDROID  IS  THE   PROBLEM  
  • 9. And  Then  There  Were  Two…   •  Android  unveiled  November  2007   •  First  device  was  sold  in  October   2008.   •  Over  11,000  models  have  been   released.   •  48  Billion  app  installs   •  Over  1  Billion  Android  devices   acMvated   •  8  OS  Revisions    
  • 10. OS  FragmentaMon  
  • 11. Device  FragmentaMon  
  • 12. Let’s  do  some  math!   •  16  device  display  categories   •  20  different  common  resoluMons   •  8  OS  versions   •  6  Hardware  Manufacturers   •  4  Major  cellular  networks   •  16  x  20  x  8  x  6  x  4  =  76,800  permutaMons   •  Pairwise  approach  =    over  30  permutaMons   •  Who  can  afford  to  increase  tesMng  by  30X?    
  • 13. Our  Approach     •  Efficiency,  not  coverage   •  Flexible:  support  small  or  large  number  of  devices   •  Understand  how  apps  work  to  logically  select  criteria   •  Use  Market  research  to  pick  most  common   configuraMons   •  Pick  minimum  and  maximum  boundary  values  for  each   criteria   •  Choose  a  value  that  matches  an  edge  case  or  abnormal   configuraMon.   •  Pick  values  that  stress  or  tax  the  system  
  • 14. UNDERSTANDING   ANDROID  
  • 15. Android   •  Built  and  Maintained  by  Google   •  Open  Source   •  Built  on  Linux  kernel   •  ARM   •  X86  Ports   •  Built  to  support  almost  any  type  of  device   –  Phones,  tablets,  phablets,  media  players,  tv’s,   watches,  etc.   •  Device  Manufacturers  customize  code.  
  • 16. Example:  Kindle  Fire   •  Forked  Android  2.3   –  Not  updateable   •  Customized  UI   •  Separate  App  store   •  Not  all  android  apps  work   •  Custom  web  browser    
  • 17. The  OperaMng  System   •  Google  Releases  “stock”   versions   •  10  Major  Releases  since   2008   –  API  Level,  not  Version   •  Device  manufacturers  like   to  customize  the  OS   –  Drivers,  libraries,  UI   •  “Stock”  OS  available  in   Nexus  devices  or  an   Emulator    
  • 18. Simulators  /  Emulators   •  Simulator  imitates  the  so;ware  layer     –  OS  and  Libraries   –  Apple  provides  a  simulator  in  xCode  IDE   •  Emulator  duplicates  the  hardware  and  so;ware   –  Processor  and  Memory   –  Cannot  mimic  GPU,  GPS,  accelerometer   •  Always  run  stock  OS   •  Can  be  used  to  test  some  funcMonality   •  Should  always  test  on  a  physical  device  too  
  • 19. The  Processor   •  ARM  RISC-­‐based  instrucMon  set   •  SpecificaMon  defined  by  ARM  holdings   •  32  bit   •  Same  as  iOS   •  X86  patches  and  ports   •  It’s  only  a  spec,  can  be  modified.  
  • 20. SoC  –  System  On  A  Chip   •  Main  Board   •  Processor   •  RAM  Bus     •  GPU     •  May  include  :     –  Cellular   –  WiFi   –  NFC   –  GPS   –  Bluetooth  
  • 21. 2  Samsung  Galaxy  S4s   Quallcomm  Snapdragon   •  Quallcomm  Krait  300   –  Quad  core  ARMv7   Cortex  A15  Architecture   •  Adreno  320  GPU     •  Dual  Channel  533Mhz   Bus   •  Integrated  LTE   Samsung  Exynos  5  Octa   •  Samsung  Big.livle   processor   –  Quad  core  Cortex  A  15   –  Quad  core  Cortex  A7     •  PowerVR  SGX  544  GPU   •  Dual  Channel  800Mhz   Bus   •  No  Integrated  LTE  
  • 22. Common  SoC   Manufacturer   Device  Name   Cores   Processor   GPU   Qualcomm   Snapdragon  S4   2  or  4   ARM  Cortex-­‐ A15   Adreno   Nvidia   Tegra  3   4+1   ARM  Cortex-­‐ A9   Geforce   Samsung   Exynos  4   2  or  4   ARM  Cortex-­‐ A9   Mali   Intel   Medfeld   1   Intel  x86   PowerVR   Texas   Instruments   OMAP  4   2+2   ARM  Cortex   A9   PowerVR   ST-­‐Ericcson   NovaThor   2   ARM  Cortex-­‐ A9   PowerVR  
  • 23. ResoluMon  is  not  enough   •  Unlimited  number  of  screen  sizes  available   •  Screens  range  from  3”  to  11”   •  Each  screen  has  a  resoluMon,  same  as  a   monitor   – If  you  increase  the  resoluMon  everything  shrinks!   •  Pixels  per  Inch  =  Density   •  Screen  Size  +  Density  =  Display  Bucket   •  ResoluMon  is  not  enough!    
  • 24. The  Display  Buckets   Size  :     •  Xlarge  :  8”  -­‐  10”  tablet.       •  Large  :    5”  -­‐  7”  tablet.       •  Normal  :  3.5”  -­‐  5”  phones.   •  Small  :  3”  -­‐  3.5”  phones.       Density  :   •  ldpi    =  Low  DPI  (~120)   •  mdpi  =  Medium  DPI  (~160)   •  hdpi  =  High  DPI  (~240)   •  xhdpi  =  Extra  High  DPI   (~320)   Low  density   (120),  ldpi   Medium   density   (160),  mdpi   High  density   (240),  hdpi   Extra  high   density   (320),  xhdpi   Small   screen   QVGA   (240x320)   480x640   Normal   screen   WQVGA400   (240x400)   WQVGA432   (240x432)   HVGA   (320x480)   WVGA800   (480x800)     WVGA854   (480x854)     600x1024   640x960   Large   screen   WVGA800* *  (480x800)     WVGA854* *  (480x854)   WVGA800*   (480x800)     WVGA854*   (480x854)     600x1024   Extra  Large   screen   1024x600   WXGA   (1280x800) †   1024x768   1280x768   1536x1152   1920x1152     1920x1200   2048x1536   2560x1536     2560x1600  
  • 25. Display  Buckets   •  Galaxy  S3   –  1280  x  720   –  Xhdpi  density  (331ppi)   –  Normal  screen  (4.7”)   •  Galaxy  Tab  10.1   –  1280  x  800     –  ldpi  density  (149ppi)   –  Xlarge  sceren  (10.1”)   •  Galaxy  Note  LTE     –  1280  x  800   –  hdpi  density  (285ppi)   –  Large  Screen  (5.5”)  
  • 26. Market  Analysis  
  • 27. Aspect  RaMo   •  UI  is  manipulated  from  code   •  Density  Pixels  adjust  for  screen  size   – But  can  use  regular  pixels!   •  Need  to  take  both  X  and  Y  into  account!   – Easy  to  overlap  or  hide  things   •  Includes  orientaMon   •  Some  devices  include  an  aspect  raMo  changer!   (LG  OpMmus  Vu)  
  • 28. Cellular  Carrier   •  Four  Major  US  Networks   –  Verizon,  Sprint,  AT&T,  T-­‐Mobile   –  Some  phone  interoperability   –  2  protocols     •  GSM  –  T-­‐Mobile  AT&T   •  CDMA  –  Verizon  and  Sprint   –  Carriers  assigned  specific  frequency  bands   –  LTE  will  be  new  standard  -­‐  But  spectrum  issues  will   prevent  cross-­‐network  phones   •  So  if  the  phone  supports  the  carrier’s  protocol   and  band  it  can  theoreMcally  connect.  
  • 29. HOW  APPS  WORK  
  • 30. How  Apps  work   •  Apps  need  to  work  on  all  screen  sizes     – May  not  be  funcMonal   – May  be  wasted  space   – May  not  make  sense   •  Apps  define  XML  layouts  similar  to  HTML     – Node  structure   – StaMc  Content  –  Images,  etc   – Dynamic  Content  –  Color,  Text,  etc.  
  • 31. Layouts  and  Fragments   •  XML  Fragments  are   reusable  components   •  Layouts  sMtch  together   fragments  for  a  specific   sized  device     •  App  may  need  different   flow  for  tablet  vs  phone  
  • 32. BUILDING  THE     DEVICE  MATRIX  
  • 33. Our  Criteria   •  OperaMng  System   –  OS  customizaMons,  missing  libraries,  driver  issues,     •  Screen  Size   –  Rendering  issues,  usability,  missing  layouts   •  Pixel  Density   –  Density  Independence,  missing  layouts.   •  Aspect  RaMo   –  X,Y  calculaMons,  overlapping  panels,  display  issues   •  SoC   –  Hardware  performance,  InstrucMon  set,  bavery,  signal   •  Carrier   –  Network  protocol,  speed,  responsiveness,  packet  loss  
  • 34. The  Goal   •  Efficiency,  not  coverage!   •  Build  a  set  of  devices  to  be  used  for  app  and   website  tesMng.   •  Know  when  to  update  them   •  Define  a  list  of  simple  categories  of  devices   •  Pick  devices  that  offer  broad  coverage   •  Adjust  the  number  of  devices  based  upon   needed  coverage  
  • 35. Categorical  Approach   •  Define  scope   –   Android,  iOS,  phone,  tablet,  etc.   •  Understand  TesMng  requirements   •  Self-­‐descripMve  Names   •  Help  to  broaden  coverage   •  Will  adjust  devices  chosen  to  cover  our  criteria   •  Should  be  apparent  when  to  update  a  device   •  Spread  coverage  :   –  Usage  -­‐>  Edge  Cases  -­‐>  Strange  -­‐>  Stress  
  • 36. Example  Categories   •  Common     –  Matches  most  common   display  configuraMon     •  Newest   –  Latest  OS  version,  largest   screen,  highest  resoluMon   •  Oldest   –  Oldest,  slowest,  smallest   device.   •  Abnormal   –  Non-­‐standard  OS,  aspect   raMo,  orientaMon,  size   •  Popular   –  Most  popular  device  in   terms  of  sales   •  Budget   –  Low-­‐priced  new  model.     Tend  to  have  strange  specs   •  Flagship   –  Nexus  device  running  stock   Android  OS   •  Catch-­‐All   –  Cover  any  missing  criteria  
  • 37. Android  Phone  Matrix     March   2012   Device  Name   OS   Display   Aspe ct   SoC   Carrier   Newest   HTC  Droid  DNA   4.2   Normal-­‐xhdpi   9:16   Snapdragon  S4   Verizon   Oldest   HTC  Tavoo   1.6   Small-­‐ldpi   3:4   Snapdragon  S1   AT&T   Common   Motorola  Droid   3   2.3   Normal-­‐hdpi   9:16   TI  OMAP  4   Verizon   Popular   Samsung  Galaxy   S3   4.1   Normal-­‐xhdpi   9:16   Exynos  4   Sprint   Abnormal   LG  OpMmus  VU   4   Large-­‐hdpi   3:4   Nvidia  Tegra  3   Tmobile   Flagship   LG  Nexus  4   4.2   Normal-­‐xhdpi   3:5   Snapdragon  S4   TMobile   Budget   Dell  Venue   2.2   Normal-­‐mdpi   3:5   Snapdragon  S3   AT&T   Catch-­‐All   Sony  Xperia  P   2.3   Normal-­‐hdpi   9:16   Sony  NovaThor   AT&T  
  • 38. iOS  Matrix     March   2012   Device   Name   OS   Display   Aspect   SoC   Carrier   Newest     iPhone  5S   7   4”  1136  x  640  326ppi   9:16   Apple  64bit  A7     T-­‐Mobile   Oldest     iPhone  3g   6   3.5”  320  x  480  165ppi   2:3   Apple  A3   AT&T   Common   iPhone  5   6   4”  1136  x  640  326ppi   9:16   Apple  A5   Verizon     Popular   iPhone  4   6   3.5”  640x960  330ppi   2:3   Apple  A4   Sprint   iPad   (ReZna)   iPad  3     7   10”  1536x2048  264ppi   3:4   Apple  A5X   Verizon   iPod   iPod  Touch   (4th  gen)   5   3.5”  640x960  326ppi   2:3   Apple  A4   WiFi   Mini   iPad  Mini   6   7”  1024  x  768  162ppi   3:4   Apple  A5   AT&T  
  • 39. Conclusion   •  Understanding  how  everything  works  allows   us  to  logically  select  devices.   •  A  large  number  of  permutaMons  can  be   covered  in  few  devices   •  If  addiMonal  coverage  is  needed  addiMonal   devices  can  be  added   •  White  Paper  :     – hvp://www.prototest.com  :     – Building  the  UlMmate  Device  Matrix