CodeFest 2014. Eduardo Bravo — Google+, A Deep Dive into Mobile Testing


Published on

Published in: Internet, Technology, Business
  • Be the first to comment

  • Be the first to like this

CodeFest 2014. Eduardo Bravo — Google+, A Deep Dive into Mobile Testing

  1. 1. Eduardo Bravo (Software Engineer in Test @ Google) Google+, A Deep Dive into Mobile Testing Codefest 2014, Novosibirsk, Russia
  2. 2. Lets start with a question….. (‘cause life is full of them)
  3. 3. Is your product a “Mobile first” product?
  4. 4. Google+ became “Mobile first” overnight .. Testing teamManagers “Yeah! Mobile first” Mobile testing ... what?
  5. 5. The Google+ Testing Picture Google Apis Server Foo backend Bar backend Many more Google+ Backend
  6. 6. The Google+ Mobile Testing Picture Google Apis Server Foo backend Bar backend Many more Google+ Backend Mobile Testing
  7. 7. Google+ Android Mobile Test Infrastructure Unit tests Hermetic UI tests Monkeyrunner tests Feature “Leak” tests Android tooling
  8. 8. Android Unit Testing Robolectric Why? Robolectric is designed to allow running android tests on the JVM, by using shadow objects and allowing resource loading. Robolectric tests can be run as normal java tests. Advantages: ● It allows us to shadow methods that can’t be mocked on the traditionally android framework.(Exceptions apply, see powermock) ● Decoupled from an android device or emulator.
  9. 9. Android Unit Testing Robolectric All of a sudden your unit tests just got faster! ● There is no need to start an emulator (our tests run in the cloud) ● Shorter build times ● No need to install apks
  10. 10. Android UI Testing The flakiness problem Flaky tests are worse than having no tests
  11. 11. Android Hermetic UI Tests Google+ uses Espresso Why? ● Backwards compatibility: Froyo, Gingerbread, Ice Cream Sandwich, Jelly Bean, Kit kat ● Handles Synchronization: Instrumentation tests run on a different (instrumentation) thread than UI operations (processed on the UI thread).
  12. 12. Android Hermetic UI Tests Google+ uses Espresso Why? ● Rich debugging information, cleaner api, 95% test coverage, etc Without synchronization of test operations with UI updates the tests will be prone to flakiness; Espresso takes care of this for us.
  13. 13. Android Emulator Espresso Hermetic Tests Fake Login Server Normal flow Espresso + Hermetic flow Google+ Backend Login Server Android Emulator UI Tests Fake Google+ Backend Android Hermetic UI Tests
  14. 14. Android Monkey runner What is it? ● The Android Monkey is a tool that generates pseudo- random user events such as clicks, touches, gestures, as well as a number of system-level events. Purpose ● Stress test applications
  15. 15. Android Monkey runner Some details about our setup: ● 10 devices with different API levels running 24/7 ● Exceptions clustering ● Over 100+ bugs found (yes, human reproducible ones’)
  16. 16. Android “Feature” Leak Testing Why? ● Because people are curious and like reverse engineering android apk’s to search for unreleased features. Example from Android Police: [APK Teardown] Google Games, Anyone? Play Services Is Getting Real-Time And Turn-Based Multiplayer, Invitations, In-Game Chat, Lobbies, Leaderboards, And Achievements!
  17. 17. Android “Feature” Leak Testing Google+ 3-step strategy: 1. Flags library that removes unreleased features at compile time. (Provides separation between a developer build and a release one.) 2. Proguard every version sent to production. 3. Continuously run a leak test that reverse-engineers the release apk, looking for “Feature” leaks.
  18. 18. Android Tooling ● Android lint errors displayed at changelist review time. ● Continuous monitoring of APK size and alert system. ● Monitor method count (Don’t get caught by the 64K method reference limit, two days before shipping.)
  19. 19. iOS Mobile Test Infrastructure for Google+ app Unit tests Hermetic UI tests Monkeyrunner tests Feature “Leak” tests
  20. 20. iOS Unit Testing GTM + OCMock GTM (google-toolbox-for-mac) ● Unit test suite based on GTMSenTestcase (psst, we are considering switching to xctest in the future). Why OCMock? ● Very powerful framework for mocking Objective-C code and also very flexible.
  21. 21. iOS Hermetic UI Testing KIF (Keep it functional) Why KIF? ● More natural than UIAutomation (Javascript based) ● Since tests are run in-process, it allows tests to be more tightly coupled with the app under test, making the tests inherently more stable.
  22. 22. iOS Hermetic UI Testing Hermetic environment We swizzle methods that fire HTTP requests in the app, and instead have them read off of an in-memory NSDictionary to get response data. @interface FakeGoogleApisServer : NSObject { … NSMutableDictionary *_sequentialResponses; … NSMutableDictionary *_fixedResponses; ... } KIF + Swizzling = Hermetic Google+ iOS tests
  23. 23. iOS Monkey Runner ● Tool developed to randomly navigate an application in order to test for crashes. ● In house tool, but we are planning to open source it. ● KIF Based. ● The Google+ team runs it mostly on simulators. ● More than 50+ bugs catched by monkey ● Experimenting with a smarter monkey that can leverage randomized and deterministic actions.
  24. 24. iOS “Feature” Leak Testing Looking for unreleased features in iOS apps is not a common practice like it is in android, we like to take precautions We have a test that: - unzips a provided .zip or .ipa - Checks for leak terms in filenames - Uses the class-dump tool and makes sure no leak terms are found.
  25. 25. Backend Server Testing Server and Client have different release cycles Google+ server releases new versions of services multiple times per week. Mobile clients (iOS and Android) usually twice a month. Different mobile clients app versions Google+ servers changes needs to be compatible against older mobile clients.
  26. 26. Backend Server Testing We decided to test the protocol We follow a replay-test approach. The approach simulates mobile clients requests and makes sure the Google+ backend server returns the response the mobile clients were expecting yesterday, and the response they were expecting 6 months ago.
  27. 27. Backend server testing Fake Data
  28. 28. Спасибо Вопросы?