Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing Sucks, But It Doesn't Have To


Published on

Writing an app for Android takes a lot of work. One of the greatest pains developers suffer from is the testing process, which many do badly or not at all. Why? Testing sucks - but it's a necessary evil.

In this class, you'll learn about test methods available to developers today, in particular Monkey, the user interface testing tool. You'll learn why Monkey is such an effective tool, and how to use it to test your own apps. You'll also learn about the limitations of Monkey, especially when it comes to making sure your app runs on any Android device. Finally, you'll learn about Apkudo, a testing solution that lets developers see how their apps install, execute, and perform on every (yes, every) Android device, assuring interoperability and customer satisfaction.

Testing doesn't have to suck. We'll show you why.

  • Be the first to comment

Testing Sucks, But It Doesn't Have To

  3. 3. 54% WON’T INSTALL 61% ©  2011  -­‐   ©  2011  -­‐   ©  2011  -­‐  CONFIDENTIAL   CONFIDENTIAL   CONFIDENTIA CRASHES 3 ©2011 Apkudo Inc. ©  2011  -­‐   ©  2011  -­‐   ©  2011  -­‐  
  4. 4. “I TEST MY APP!” 99%4 ©2011 Apkudo Inc.
  5. 5. “I TEST MY APP!” 99% O RLY? 85% MANUAL  5 ©2011 Apkudo Inc.
  6. 6. OH NOES6 ©2011 apkudo, inc. Confidential
  7. 7. AD HOC TESTING Sucks Keeping up with device versions Testing across various devices AUTOMATED TESTING Tedious JUnit, Robotium Test plan7 ©2011 apkudo, inc. Confidential
  8. 8. MONKEY! A good start. Simulates inputs. Completely automated.8 ©2011 apkudo, inc. Confidential
  9. 9. MONKEY OVERVIEW    Package  Manager      Install  and  uninstall  packages      Get  list  of  package  ac:vi:es      Ac$vity  Manager      Start  /  stop  ac:vi:es      Get  list  of  running  ac:vi:es      Window  Manager      Send  touch  and  key  events        to  foreground  app   9 ©2011 apkudo, inc. Confidential
  10. 10. MONKEY OVERVIEW    Package  Manager   queryIntentActivities()    Ac$vity  Manager      setActivityController()    Window  Manager      inject*Event() 10 ©2011 apkudo, inc. Confidential
  11. 11. MONKEY SEEMONKEY DO•  Monkey injects “events” - what is an event? –  KeyEvent: Hardware buttons and keyboard –  MotionEvent: X, Y, and touches –  FlipEvent, NetworkEvent, PowerEvent (mostly legacy)11 ©2011 apkudo, inc. Confidential
  12. 12. Security   SECURITY •  Monkey uses standard Android permissions... that no user apps can use!public static final String SET_ACTIVITY_WATCHERSince: API Level 1Allows an application to watch and control how activities are started globally in the system. Only for is indebugging (usually the monkey command).public static final String INJECT_EVENTSSince: API Level 1Allows an application to inject user events (keys, touch, trackball) into the event stream and deliver themto ANY window. Without this permission, you can only deliver events to windows in your own process. Veryfew applications should need to use this permission. 12 ©2011 apkudo, inc. Confidential
  13. 13. MONKEYvVS onkey  Runner   Monkey   s  MMONKEY RUNNER•  Monkey is for randomized testing, and runs on the device•  Monkey Runner is for scripting your own tests, and runs on a laptop / desktop•  In fact, Monkey Runner uses Monkey internally
  14. 14. Monkey  on  Angry  Birds   THE BIRDS, THEY ARE ANGRY!adb shell monkey –v -p com.rovio.angrybirds -s 39 --throttle 10 --pct-touch 100 5000 14 ©2011 apkudo, inc. Confidential www.apkudo.comadb shell monkey -p com.rovio.angrybirds 5000
  15. 15. What  can  Monkey  find?   MONKEY RESULTS •  Key and motion events •  Crash information if applicable, as traceback:AllowPackage:  com.rovio.angrybirds  :IncludeCategory:  android.intent.category.LAUNCHER  :IncludeCategory:  android.intent.category.MONKEY  :Switch:  #Intent;;category=android.intent.category.LAUNCHER;launchFlags=0x10000000;component=com.rovio.angrybirds/com.rovio.ka3d.App;end        //  Allowing  start  of  Intent  {  cat=[android.intent.category.LAUNCHER]  cmp=com.rovio.angrybirds/com.rovio.ka3d.App  }  in  package  com.rovio.angrybirds  :Sending  Pointer  ACTION_DOWN  x=554.0  y=357.0  :Sending  Pointer  ACTION_UP  x=558.0  y=350.0        //  Rejec:ng  start  of  Intent  {  }  in  package  :Sending  Pointer  ACTION_DOWN  x=696.0  y=280.0  :Sending  Pointer  ACTION_UP  x=709.0  y=271.0  :Sending  Pointer  ACTION_MOVE  x=-­‐1.0  y=1.0  :Sending  Pointer  ACTION_DOWN  x=348.0  y=10.0   15 ©2011 apkudo, inc. Confidential
  16. 16. WHAT CAN Monkey  find?   What  can  MONKEY FIND?•  Weird UI sequences –  Press button 2 before button 1•  Unexpected inputs –  E.G. keyboard input during a Toast•  User-interface race conditions•  Activity life-cycle errors16 ©2011 apkudo, inc. Confidential
  17. 17. Weird  UI  sequences  WEIRDUI SEQUENCES•  Monkey presses UI controls in an unexpected way•  Example: Comic Rack: selecting “Options” immediately after application start•  Resolution: Always check preconditions in UI callback functions17 ©2011 apkudo, inc. Confidential
  18. 18. UNEXPECTED inputs   Unexpected  INPUTS•  Monkey simulates keypresses that the application doesn’t expect•  Example: multiple keypresses on the same button•  Resolution: Again, be paranoid when entering UI callback functions18 ©2011 apkudo, inc. Confidential
  19. 19. RACE Race  condi:ons  CONDITIONS•  Monkey can press UI elements very quickly, triggering race conditions•  Example: I Heart Radio: PlayerController not initialized•  Resolution: limit use of threads. When they are necessary, perform appropriate locking.•  Same applies to networking.19 ©2011 apkudo, inc. Confidential
  20. 20. Ac:vity  life-­‐cycle  errors  ACTIVITY LIFE CYCLEERRORS•  Monkey performs unexpected activity life- cycle actions•  Example: Shazam: Unable to pause activity•  Resolution: Become tiresomely familiar with the Android activity life cycle.
  21. 21. MULTI-DEVICE Monkey   Mul:-­‐device  MONKEY•  Monkey is really good at testing across multiple devices:•  Correct Market permissions –  appropriate-architecture native code, eg armv7•  Unexpected hardware properties•  “Weird” hardware failures
  22. 22. Mul:-­‐device  Monkey  CORRECT MARKETFILTERS•  OpenGL ES version and features•  Dependencies on system libraries•  Incorrect or missing minSdkVersion
  23. 23. Unexpected  hardware  proper:es  UNEXPECTED HARDWAREPROPERTIES•  The device has an unexpected hardware feature (e.g. a strange screen resolution)•  Examples: RDefense Free, Zombie Killer•  Best resolution: Test on multiple devices
  24. 24. Weird  hardware  failures  WEIRD HARDWREFAILURES•  Not all devices “just work”•  Example: PaperCamera doesn’t handle RuntimeError from takePicture()•  Resolution: Test on multiple devices. Be wary of Java unchecked errors.24 ©2011 apkudo, inc. Confidential
  25. 25. MONKEYLimita:ons  LIMITATIONS•  False positives –  “Impossible” sequences of actions / speeds•  Very limited testing strategy –  What if most of your app is hidden behind a login screen?•  Some Android pain points not covered –  E.G. screen rotation, layout issues25 ©2011 apkudo, inc. Confidential
  26. 26. AND Apkudo  APKUDO•  “See your app run on every Android device” –  Monkey test on 289 phones and tablets –  Free –  Private beta right now: email if interested26 ©2011 apkudo, inc. Confidential
  27. 27. Thank you. NFD@ .COM