The document discusses the implementation of continuous integration and delivery (CI/CD) for HERE mobile apps, focusing on the challenges and solutions related to mobile app development. It outlines key objectives, metrics, tools used, and the importance of feedback during the development process to ensure app quality and user satisfaction. The presentation also addresses future plans for automation and improvements in testing and release cycles.

1. Continuous Integration and Delivery
for the HERE Mobile Apps
How we deliver mainline to millions of
Android and iOS users every other
week.
DevOps Meetup Berlin
Stefan Verhoeff, 28 September 2016
@stefan_verhoeff
All opinions expressed in this deck are the author's own and do not
necessarily represent the official view of HERE

2. Intro
• Hi, I’m Stefan
• I run a team called Wookiee
who build and run CI/CD for
Android and iOS
• We build the HERE WeGo
App, give it a try ;-)

3. HERE WeGo App

4. Agenda
• CI/CD for Mobile?
• Our approach
– Key objectives + metrics
– Pipelines
– Platform architecture
– Testing
– Dashboards
– Releasing
• Challenges & plans
• Q&A

5. DevOps + CI/CD
• Who here does CI/CD? On mobile?
• Wikipedia says:
– Continuous Integration
Continuous integration (CI) is the practice of merging all developer
working copies to a shared mainline several times a day
– Continuous Delivery
Continuous delivery (CD) is a software engineering approach in which
teams produce software in short cycles, ensuring that the software can
be reliably released at any time. It aims at building, testing, and
releasing software faster and more frequently.

6. DevOps + CI/CD
• CI is feedback to the developer
– Did I build it right?
– Did I break something?
• CD is feedback + deliver value to
the user
– Did we build the right thing?
– Did it the changes cause any
issues?

7. State of Mobile CI/CD
• We know about CI/CD for web
and cloud, what is different for
Mobile?
• Delivery = App Store / Play
Store -> Brick wall
• You can still keep your App
shippable at all times, even if
you don’t ship every build

8. Challenges of Mobile CI/CD
• Dealing with many devices
and OS versions
• App store approval cycle
takes time. Apple, looking at
you!
• High cost of failure. Can’t roll
back easily.
• CI tools for Mobile lacking in
the past, but starting to pop
up lately.

9. CI related tools on Mobile
• Platforms
– Google Firebase, Fabric, AWS Mobile
• App store automation
– Fastlane
• Distribution
– HockeyApp, TestFlight, Play Store channels
• Cloud devices
– Amazon Device Farm, Google Test Cloud, Bitbar TestDroid
• Cloud CI
– Travis, CircleCI, Bitrise, Build Buddy

10. Monolith, anyone?

11. Objectives + key metrics
• Maximize developer feedback
– Coverage: lines + test cases > 80% coverage
– Speed: feedback time Pre-Submit < 15 min
– Reliability: red pipeline % of day > 90%
• Maximize user delight
– App Store rating > 4.5 stars
– NPS score > 30 score
– Bug limit < 30 open bugs + SLA
– Limit # critical bugs on production < 5 per year
• Visual dashboards to track

12. HERE implementation of CI
Pre-submit Testing
• Improves code quality
• Enables cross-team collaboration
To make the shared “Mainline” work, we introduce 2 enablers
Code Review
• Protects the mainline
• Provides direct feedback on each change
One Mainline per
Product
• Reduces time to market
• Creates responsibility
• Creates focus
• Improves predictability

13. Code reviews through Gerrit
• Double combo:
– Pre-submit: CI feedback
isolated from other developers
– Review: Peer developer
feedback
• Reviews help knowledge
sharing across teams

14. Code Review and Pre-submit Testing
Developer‘s
machine
Download code
Code Hosting Server
1
New change
created locally
Create/modify files
2
Upload change for
review with peers
3 No
Yes
Rework
5.a
Submit5.b
Accepted
Pre-submit
4
Code
Review
Automated
Tests

15. HERE CI stages
Mainline
Pre-submit
Verification
Pre-submit
Verification
Pre-submit
Verification
Pre-submit
Verification
Pre-submit
Verification
Pre-submit
Verification
Pre-submit
Verification
Submit
Verification
Submit
Verification
Submit
Verification
Submit
Verification
Submit
Verification
Submit
Verification
Submit
Verification
Submit
Verification
Longer
Verification < 1h
Verification
> 1h
Can be released
Optional Manual
Tests
Longer
Verification < 1h
Longer
Verification < 1h
Pre-submit
Verification

16. Why we have stages

17. Platform architecture

18. Our device labs

19. Testing
UI test with mocks
UI test end to end
Exploratory + Regression + Compatibly
Drive in the field
Unit tests class level
Unit tests component level
1 day
2 days
100s
500+
100s
1000s

20. Avoiding flaky tests
• Flaky tests are a huge source of waste,
we fight them with fire
• Mocking dependencies and network
• Moving tests down to component level
• Re-run after failing
• Stable testing culture. Google’s Testing
of the toilet series

21. Visualizing test results – history matrix

22. Test status dashboard

23. Dashboards
• Gathering data and creating dashboards probably one
of the best things we've done to grow awareness
• Key metrics: these are in everyone's (dev + ops) annual
objectives!
– Coverage
– Performance
– Stability
• Graphite + Granafa

24. CI performance dashboard

25. CI stability dashboard

26. Team dashboard

27. Jenkins wall monitor

28. Keeping CI stable
• Wall monitor and team
dashboard visible in area
• Stop the line if bug limit
reached
• Build Police: rotating
developer who spots
failures and finds
who/what caused it
• HERO*: CI Engineer
available on chat for
support
* Helpful Emergency Responsible Operator

29. Releasing
• Release train
– Ship on a regular schedule
– Always release master*
– New features hidden behind flag
• Gradual rollout
– Activate features without App Store release!
– Roll out percentage wise to measure impact
– Works great combined with A/B testing
– Reduces risk and enables learning
* We do create a release branch but just to avoid regressions while we manually test

30. How we release
• Push to App Store / Play Store still manual but automation
planned
• Weekly: Alpha internal release
• Every other week
– Beta
– Production
• Release checklist
– Comms, legal, privacy, platform, ...
• Release results tracking

31. Release schedule

32. Production feedback
• Metrics: analytics, logs and crash data
• User feedback, NPS, store reviews

33. CI as Code
• Jenkins Job DSL plugin
• Treat it as code:
– Version control
– Abstractions, re-use and
refactoring
– Tests
– CI for CI code (infinite
recursion error)
– Code reviews
– Local testing
[
Constants.Branches.MASTER,
Constants.Branches.RELEASE
].each { branch ->
def pipelineBuilder = new NightlyPipelineBuilder(this)
pipelineBuilder.with {
defaultConfiguration([
client : Constants.Description.Contacts.WOOKIEE,
pipelineType: Constants.Description.PipelineType.NIGHTLY
])
baseFolders([Constants.Folders.PROJECT, branch, Constants.Folders.NIGHTLY])
triggerManifestRepoBranch = branch == Constants.Branches.MASTER ? 'master' :
'release/heresuite'
triggerManifestRepoFile = 'heresuite-android.xml'
triggerPublishArtifacts = ['manifest.xml']
defaultBuildJobName =
"${Constants.Folders.PROJECT}/${branch}/${Constants.Folders.NIGHTLY}/" +
'build_heresuite-android_arm_dev_debug_play'
addTrigger()
buildPhase.predefinedParams += [BRANCH: branch]
testPhase.predefinedParams += [BRANCH: branch]
addBuildJobs()
addTestEndurance()
addTestEndurance10()
addTestOfflineWithRerun()
addTestUnitTestCoverage()
}

34. Challenges & solutions
• Long release cycles for App -> Move to release train + gradual rollout
• Long SDK integration cycle -> Early manual testing and connected CI systems
• Constant bug fighting App teams -> Tech improvements, refactoring, test automation, RCA
• Flaky tests -> Visualize, prune test suite, mock, component tests
• No insight into state of CI KPIs -> Collect metrics and create dashboards
• Manual job maintenance hell -> CI as Code
• Taking action on failures -> Visible screens, HERO and build police roles
• Maintenance and capacity devices -> Buy more hardware, optimize use, move to cloud

35. Next plans / on the radar
• More tests for CI code
• Jenkins Pipeline plugin
• Fastlane for iOS
• App Store / Play Store
automation
• TestDroid cloud testing
• Multiplatform end to end tests

36. Thank you!

37. Bonus

38. Pipelines for Mobile
• The stages of our CI system
– Pre-Submit
– Post-Submit
– Hourly
– Nightly

39. Mobile specific concerns
• Signing certificates
• Distributing Dev and Beta builds
– HockeyApp
– TestFlight App Store
– Google Play Store Alpha/Beta channel
• App Store / Play Store automation

40. CI node dashboard

41. Device lab dashboard

42. CI building blocks
• Builds
• Static Analysis
• Unit tests
• Functional test

43. Test types and life cycle
• Gateway
– Set of representative test run in Pre-Submit
– Very stable and fast
• Hourly
– Big set up regression tests. Take an hour to run so timed
• Unproven
– New tests first need to prove that they are stable enough before being added to the
hourly test set
• Unstable
– Known unstable tests. Either fix or delete
• End to end
– All above tests use mocks. But there is a set of . These are more brittle so high
maintenance costs. They do find more issues.

44. Non-functional tests
• Performance KPI
• Monkey
• Power consumption
• Memory