Yelp Tech Talks: Mobile Testing 1, 2, 3

1. @YelpEngineering YelpEngineers engineeringblog.yelp.com github.com/yelpyelp.com/careers

3. Building Yelp for Apple Watch Bill Meltsner wmeltsner@yelp.com @billmeltsner

4. Today Initial Scoping / Planning Yelp.app on Apple Watch Deep Dive Lessons Learned

5. Who I Am iOS Technical Lead Yelp on Apple Watch project lead Worked largely on Watch app logic

6. Who We Were Designer Engineer Product Manager Engineer (Intern)

7. Why Build Yelp Watch App? Yelp everywhere Day-one advantage Knew we could build a great experience

8. Initial Scoping time = features

9. Initial Scoping unlimited time* = unlimited features *this never happens

10. Initial Scoping finite time = finite features

11. Initial Scoping Product team defined features Engineering team estimated available time Worked together to define MVP as cost of time versus feature

12. Initial Scoping UI & Logic could be parallelized - 1 engineer for each Milestones for MVP, MVP+1, MVP+2, … Flexibility in our schedule to add / remove features and stay agile

13. Demo

14. Yelp Watch App Technical Overview

15. WatchKit - A Changing Landscape Brand new platform with docs & APIs changing drastically between betas No defined best practices Focused on making best-effort technical decisions with the ability to refine later

16. Overview of a WatchKit app Parent App API Requests Watch App Storyboard WatchKit Extension Location Logic Interface Control Images iPhone Apple Watch

17. Interface Controllers View Controller analog Interface hierarchy is fixed YPWKSearchResultsInterfaceController

18. Storyboard Overview

28. Before After

29. Networking API requests owned by parent app Image loading owned by extension

30. Location Owned by extension – runs in foreground, parent app runs in background We only request foreground access

31. Location Permissions belong to parent app, must be granted on phone

32. Phone ↔ Watch Communications All calls in one iteration of the run loop coalesced together Communication between watch and phone is rate-limited serial queue Overhead is high – batch your calls!

33. Images Key part of our search UI Naive approach: send each image to the watch as it’s loaded Result: traffic jam of communications, unresponsive app

34. Images Solution: Wait x seconds, send all images loaded in that timeframe at once Problem: that can be a lot of data Solution Part 2: crush the heck out of ‘em UIImageJPEGRepresentation(image, 0.0) // max compression

35. Lessons Learned

36. Think like a Startup Priority 1 was being there on launch day MVP comes first, everything else can wait Technical debt is not inherently bad

37. Plan Ahead Define designs and scope before writing any code New platforms are hard to predict effectively Bend but don’t break

38. Questions?

39. Testing The Yelp App iOS & Android

40. Who We Are Mason Glidden ● iOS Engineer ● iOS Testing o KIF & Jenkins ● mglidden@yelp.com Tim Mellor ● Android Engineer ● Android Testing o Espresso & Jenkins

41. How we develop new mobile APIs iOS & Android - Tests & Testing Strategy Today

42. Building New Mobile APIs How we use our documentation to test new APIs

43. Mobile APIs @ Yelp ● API shared by iOS & Android ● New APIs start with documentation and examples ● Client and API can be developed simultaneously ● API team manages backwards compatibility tests

44. /* h2. Photo (full) |_. Name |_. Type |_. Description | | id | string | Identifier | | time_created | time | Timestamp for when photo was uploaded | | url_prefix | string | Prefix for image url | | user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) | | caption | string | Caption | | feedback_positive_count | integer | Number of likes for the photo | | */ { "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3- media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19, } Documentation

45. /* h2. Photo (full) |_. Name |_. Type |_. Description | | id | string | Identifier | | time_created | time | Timestamp for when photo was uploaded | | url_prefix | string | Prefix for image url | | user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) | | caption | string | Caption | | feedback_positive_count | integer | Number of likes for the photo | | */ { "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3- media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19, } Documentation Textile

46. /* h2. Photo (full) |_. Name |_. Type |_. Description | | id | string | Identifier | | time_created | time | Timestamp for when photo was uploaded | | url_prefix | string | Prefix for image url | | user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) | | caption | string | Caption | | feedback_positive_count | integer | Number of likes for the photo | | */ { "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3- media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19, } Documentation JSONDoc Textile

47. Documentation /* h2. Photo (full) |_. Name |_. Type |_. Description | | id | string | Identifier | | time_created | time | Timestamp for when photo was uploaded | | url_prefix | string | Prefix for image url | | user_passport | "Passport":{{site.url}}/v1/objects/passport/index.html | Passport for user who uploaded photo (not provided for photos added by biz owners to their own biz from their biz owner account) (Optional) | | caption | string | Caption | | feedback_positive_count | integer | Number of likes for the photo | | */ { "id": "PHOTO_ID123", "time_created": 1370985832, "user_passport": {% include_jsondoc "v1/objects/passport/default.json" Passport %}, "url_prefix": "http://s3- media4.ak.yelpcdn.com/bphoto/_Xwa-lWTpivnlB1tgX-sJw/", "caption": "Korean tacos ($5.75)", "feedback_positive_count": 19, }

48. Documentation -> JSON ● Included as submodule in client repos ● Build step to flatten documentation into JSON (e.g. v1- -objects--photo+full.json) ● Code requests specific mocks

49. Why This Approach Works for Us ● API & client contract ● Fewer dependencies for developers & Jenkins ● Improved test speed & reliability on iOS & Android

50. iOS Testing @ Yelp Mason Glidden

51. ● Prevent Regressions ● Give developers confidence ● Run quickly ● Reliable results ● Easy to write Test Goals

52. ● Unit Tests ● Integration Tests ● Acceptance Tests Test Types

53. ● Prevent UI & Logic Regressions ● ~150 logic unit tests ● ~100 network request contract tests ● ~650 view tests ● Continuous Integration on Jenkins Unit Tests

54. ● Generally pretty simple ● Test-Driven Development ● Super fast to run Logic Tests

55. Example: Business Hours Logic - (void)testOpensSoon { // Test that "Opens soon" appears with the correct time interval [NSDate yk_setDate:[NSDate dateWithTimeIntervalSince1970:1356040364]]; NSArray *openHoursArray = @[@[@5160, @5460]]; OpenHours *openHours = [OpenHours openHoursFromJSON:openHoursArray timeZoneString:@"America/Los_Angeles"]; STAssertEqualObjects(@"Opens in 8 min", [openHours openOrClosedStringUsingMinutes:YES], nil); } Logic Tests

56. ● Makes sure client can still parse documented API changes ● Example: ReviewsListRequestTest - (void)testList { ReviewsListRequest *request = [[ReviewsListRequest alloc] init]; [OHHTTPStubs yp_receiveFromPath:@"v1--reviews+reviews.json" statusCode:200 MIMEType:@"application/json" afterDelay:0.1]; [request listWithBusinessId:@"BIZID" selectedReviewId:nil offset:0 limit:10 delegate:self]; [self waitForStatus:YPAsyncTestWaitStatusSuccess timeout:10.0 requestToCancelOnTimeout:request]; } Parsing Tests

57. View Tests ● View with mock data ● Screenshot of view ● Compares with previous versions ● Based off GHUnit

58. View Tests

59. View Tests ● Example: contribution buttons view test - (void)testBasicButtons { Business *business = [Business businessFromJSONDictionary: [YPDebug JSONFromResource:@"v1--objects--business+full.json"] request:nil context:nil]; YPBusinessContributeButtons *buttons = [[YPBusinessContributeButtons alloc] init]; [buttons setBusiness:business]; YPVerifyView(buttons); }

60. View Tests ● Pros: ○ Easy way to catch regressions ○ Invaluable when refactoring or updating to new OS versions ● Cons: ○ Slow: ~¾ seconds per test ○ Lots of false-positive failures

61. Integration Tests ● Testing that application behaves as expected ● Interaction between view controllers ● Primary signals of a problem: ○ Non-visual - analytics & network requests ○ Visual - button or label ● ~225 Integration tests

62. KIF ● ~150 KIF tests ● Uses accessibility labels to navigate ● Custom hooks for analytics, requests ● Continuous integration on Jenkins ● Separate iPad and iPhone tests github.com/kif-framework/KIF

63. Integration Test Example

64. Integration Test Example ● Example: ReviewCompositionIntegrationTest - (void)testReviewWrite { [YPAPI yp_addMockSessionConfirmed:YES sendNotification:YES]; [self yp_openBusinessViewController]; [tester tapViewWithAccessibilityLabel:@"Write a Review"]; [tester yp_waitForModalViewControllerOfClass:[ReviewComposeViewController class]]; [tester yp_waitForExpectedAnalytics:@[@{@"iri": kAnalyticsReviewWriteIRI, @"params": @{@"intended_compose_type": @"add"}]]; [tester tapViewWithAccessibilityLabel:@"Rating"]; [tester clearTextFromAndThenEnterText:@"My review" intoViewWithAccessibilityLabel:@"Write your review here."]; [tester tapViewWithAccessibilityLabel:@"Post"]; [tester yp_waitForRequestWithPath:@"/review/save" queryParams:nil postParams:@{ @"text": @"My review", @"rating": @"3" }]; }

68. Sandboxing Mocked During Tests: ● Networking ● Date, Time, Timezone ● Device permissions ● Singletons Between Test Runs: ● Clean caches ● Reset user defaults ● Reset navigation stack ● Device orientation

69. Other Tooling ● OHHTTPStubs to block & mock network requests ● OCMock for mocking ● XCTool to run our tests

70. Acceptance Tests ● Test overall look and feel ● ~50 manual test cases ○ Moving some to KIF ● iOS7 & 8, iPad & iPhone ● Run by Engineers + PM during release process

71. Closing Thoughts ● API mocks make it easy for us to reliably grow our testing suite ● Different types of tests for different problems ● Sandboxes to create consistent environments ● KIF <3

72. Android Testing @ Yelp Tim Mellor tfmellor@yelp.com

73. tests = tools + code

75. Simple, right? ● AndroidTestCase ● InstrumentationTestCase ● ApplicationTestCase ● ActivityTestCase ● ActivityUnitTestCase ● ActivityInstrumentationTestCase ● ActivityInstrumentationTestCase2

76. More decisions

78. Problem: Devices ● Devices are necessary ● Devices suck ● Virtual devices are bearable

79. Solution: Devices ● Genymotion’s gmtool ● Clone image into new device ● Speed of Genymotion $ python gmtool_wrapper.py start --vms '{"18":1, "19":1, "21":1}'

80. Problem: Flakes! ● Part 1: Instrumentation + Device ● Part 2: Test library

81. Android Instrumentation

82. Instrumentation consequences ● Uncaught exceptions halt test suite ● Activities/Services/etc. stay open

83. Solution: Instrumentation Flakes ● One test per instrumentation run! $ adb shell pm clear com.yelp.droid

84. Flakiness and Test libraries Robotium and its solo.waitFor* methods

85. Android test kit to the rescue!

86. Main Thread

87. click() Espresso blocked Main Thread Test thread assertions Test

88. click() Espresso blocked Main Thread Test thread assertions Test Task Thread Background task

89. Problem: slow test suites ● Consequence of needing devices ● Long = impractical

90. Solution: test sharding $ adb shell am instrument -w -e numShards 4 -e shardIndex 1 ● github.com/shazam/fork ● Resources are the limit!

91. Yelp Testing Process ● ~300 unit tests ● ~100 integration tests ● ~150 UI integration tests ● Manual testing against production ● Beta group ● 50% roll-out in Play Store

92. UI Integration test toolkit @ Yelp ● Espresso! ● Home-rolled MockHttpClient o MockResponse o MockRequestMatcher ● Spoon

95. public void test_ClickingBookmark_SendsAddRequestAndUpdatesUi() { mBusiness.setBookmarked(false); setActivityIntent(ActivityBusinessPage.intentForBusiness(getYelpContext(), mBusiness)); getActivity(); takeScreenshot("business detail unbookmarked"); // Check that the bookmark button has the text “Bookmark” and click on it // This should trigger a bookmark add request. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.action_bookmark)))) .perform(click()); takeScreenshot("bookmarked"); // Make sure we hit bookmarks/add with the proper params. assertThat(mAddBookmarkResponse.getRequestCount(), is(1)); // Make sure the "Bookmarked" button now shows. onView(withId(R.id.bookmark)) .check(matches(allOf(isDisplayed(), withText(R.string.bookmarked)))); }

101. public void test_WriteTip_SendsProperRequest() { setMockSession(); MockResponse tipSaveResponse = mMockHttpClient.addDefaultMock( ApiPath.QUICKTIPS_SAVE, getTipSaveParams()); openTipPage(); onView(withId(R.id.edit_text)).perform(typeText(TIP_TEXT)); takeScreenshot("tip typed"); onView(withId(R.id.done_button)).perform(click()); takeScreenshot("business page"); // We should show a "Thanks for the tip!" dialog on the business page. onView(withText(R.string.thanks_for_the_tip)).check(matches(isDisplayed())); assertThat(tipSaveResponse.getRequestCount(), is(1)); }

107. ● Library choices matter ● Address the issues at the source! ● Tests don’t have to be a pain Lessons learned

108. Questions?

Yelp Tech Talks: Mobile Testing 1, 2, 3

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Yelp Tech Talks: Mobile Testing 1, 2, 3

Similar to Yelp Tech Talks: Mobile Testing 1, 2, 3 (20)

More from Yelp Engineering

More from Yelp Engineering (13)

Recently uploaded

Recently uploaded (20)

Yelp Tech Talks: Mobile Testing 1, 2, 3