The document discusses LinkedIn's transition to a 3x3 release cycle for their flagship mobile app, releasing new versions every three hours. Previously they released monthly. The new process involves continuous integration pipelines for code review, static analysis, unit testing, and UI testing on alpha and beta builds. Tests are automated and run in parallel on emulators to speed the process. Partner teams now validate through automation. Risk is minimized through feature flags and A/B testing. Areas for future improvement include further automating the release process and adding performance testing.

1. 3x3: Speeding Up Mobile Releases
Drew Hannay (@drewhannay)

2. Project Voyager
● New version of
flagship LinkedIn app
● 250+ committers
across Android & iOS
● ~1 year of
development
● Investment in mobile
infrastructure at
LinkedIn

3. Before Voyager
● 12 releases per year
● RC build + manual regression suite
● Mad dash to commit code before the RC cutoff
○ Missing the cutoff meant a long wait for the next release
● Product & marketing plans were made around the monthly releases
● Hard to iterate on member feedback

4. 3x3
Release three times per day, no more than three
hours from code is commit to member availability

5. Why three hours?
● Not enough time for manual testing steps
● Not enough time to test everything
○ The goal isn’t 100% automation, it’s faster iterations
○ We don’t want engineers spending most of their time maintaining tests that break whenever a
design changes
● UI tests are chosen based on production-critical business flows
● Faster iteration helps emphasize craftsmanship
○ Devs can take the extra time to write quality code since the next release is soon

6. Commit Pipeline
Code
Review
Static
Analysis
Unit
Tests
Build
Release
Artifacts
UI
Tests
Alpha
Release
Feature
Development
Production
Release
Beta
Release

7. Commit Pipeline
Code
Review
Static
Analysis
Unit
Tests
Build
Release
Artifacts
UI
Tests
Alpha
Release
Feature
Development
Production
Release
Beta
Release

8. Static analysis
● Compile-time contract with API server using Rest.li
○ Rest.li data templates are shared between API server & clients
○ Provides static analysis checks that guarantee backwards compatibility
○ Client models are code generated for further safety
● Java Checkstyle
● Android Lint
○ Over 200 checks provided by Google
○ Several custom checks written for LinkedIn-specific patterns
● Swift Lint
○ Forked version of Realm’s SwiftLint
○ Added custom checks for LinkedIn patterns

9. Building the code
● Over 500k lines of code between Android & iOS
● Building production binaries for a large codebase is slow
● iOS & Swift
○ At one point early on, Swift compilation took over two hours
○ Refactoring into sub-projects and modules lead to a more than 50% speed up
● Android Split APKs
○ Separate binary for each combination of screen density and CPU architecture
● Distributed builds
○ Build the release binaries on separate machines while tests are running

10. What do we test?
● Unit tests
● Layout tests
○ Unit tests for views
○ Stress test views with long strings, short strings
○ Make sure views don’t overlap, and render properly in right-to-left mode
● Scenario tests
○ Validate that key business metric flows are working properly
○ Usually flows that span multiple screens in the app
○ App gets mock data from a local fixture server
○ Not an exhaustive suite

11. Test stability
● UI tests use Android Espresso & iOS KIF frameworks
● Needed to create a consistent test environment across dev & build machines
● Android
○ Self-contained, standardized Android emulator bundle
○ Custom test harness that runs one test per application launch
● iOS
○ KIF contained hard-coded waits that passed on dev boxes, but failed on slower build servers
○ Forked KIF to convert to a checkpoint-based system,
where the code tells the test when to proceed to the next step

12. Test speed
● Android
○ Use Espresso’s IdlingResource API to avoid sleeps and waits
○ Custom test harness allows optimal test parallelization
○ Run up to 16 Android emulators in parallel on a single build machine
● iOS
○ Refactoring KIF cut UI testing time by more than 80%
● Distributed testing -> Shard tests across multiple machines
○ Significantly faster, but led to greater exposure to any tooling instability
○ Nontrivial overhead in starting a child job

13. Android multi-emulator test run

14. Partner teams
● Historically, several partner teams validated the
build before a release
● For example, we needed sign off from the
localization team
● Lint checks catch hardcoded or improperly
formatted strings
● Layout tests catch strings that are too long and
RTL layout bugs
● Semantic correctness of translations is still
validated by translators manually

15. Getting to members
● Every three hours, internal alpha testers get a new build
○ Mainly members of the Flagship team
○ Product managers, devs, and execs who want to see the latest code ASAP
● Every week, the rest of the company gets a new beta build
○ iOS build is submitted to Apple for review
● After a week of beta, the build is promoted to production
○ Assuming Apple’s review is complete, iOS is released
○ Take advantage of Google Play staged rollout for Android

16. Dogfooding
● Android: Google Play alpha/beta channel
○ Easy upgrades for employees, even while off the corporate network
○ Somewhat difficult to get set up, but easy once registered
● iOS: TestFlight
○ Nice, but limited number of users
● iOS: Custom enterprise distribution
○ Scales to our number of users, but employees must be on corporate wifi to upgrade
● Splash screen in the production app encourages employees to use beta builds

17. Minimizing risk & enabling experiments
● Take advantage of LinkedIn’s existing A/B testing infrastructure
● New features are developed behind feature flags
○ Code can be ramped dynamically to different groups of members
○ Performance of new features or changes can be monitored
● Dynamic configuration
● Server-controlled kill switch
○ Crashing or buggy code can often be disabled without a new build

18. 3x3 after 3 months: areas to improve
● Release automation
○ Production uploads to the app stores are still a manual process
○ Getting release notes & translations is painful
● Automated performance testing
○ We can sample performance of the app in production,
but don’t have a great way of catching issues before release
● iOS speed improvements
○ Keep up with Swift evolution
○ Multi-simulator testing?
● Bring 3x3 framework to other LinkedIn apps

19. Questions

20. 3x3 blogs & videos
● 3x3: Speeding up mobile releases
● Test Stability - How We Make UI Tests Stable
● UI Automation: Keep it Functional - and Stable!
● Consistent Android Testing Environments with Gradle (slides)
● Effective Layout Testing Library for iOS
● Managing iOS Continuous Integration at Enterprise Scale