Building CI from scratch

Artem Nikitin
Heisenbug | May, 2018
New CI from scratch

These slides are already available!
https://bit.ly/2rOa0Ah
© 2018 HERE | PublicHeisenbug | May, 2018

About SDK
https://developer.here.com/develop/mobile-sdks

Specifics of our existing SDK
• Huge codebase in C++/Java/ObjC
• 40 min checkout sources from scratch
• 6 GB size of source code after checkout
• 11M+ LOC (including dependencies)

Problems of existed SDK
• Build on top of huge internal monolithic core
• Big size of SDK
• No modularity in SDK
• Low developer productivity

Problems of existed CI
• Big Jenkins setup
https://www.youtube.com/watch?v=cT8GjzM1DXk
https://conferences.oreilly.com/velocity/vl-eu-2017/public/schedule/detail/61951
• Highly opinionated CI setup
• Hard to customize
• Apple support is third class citizen

01
Solving architecture
problems first

Huge and complex codebase
• Huge internal codebase (10M+ LOC)
• Very modular (1000+ Git repos)
• Our project depends on almost 100 internal
dependencies

Solution
• Using `repo` for working with codebase
https://source.android.com/setup/develop/
• Gerrit for code reviews
https://news.ycombinator.com/item?id=8605293
• Block changes to our dependencies if they are failing
our builds/tests

Lots of boilerplate code
• Complex codebase structure with C++/Java/ObjC
• Code written by different people from different teams
• No consistency in implementation
• Lots of manually written boilerplate code

Solution
• Using IDL to describe public API
• Using code generation to generate all the boilerplate
code based on IDL description
• We are using Djinni from Dropbox
https://github.com/dropbox/djinni

Djinni, IDL

Djinni, generated C++

Djinni, generated Java

Djinni, generated ObjC

Complex build system
• Project setup is complicated and unfriendly
• Separate build setups for local development and for CI

Solution
• Universal build system/setup for local development
and CI
• Make project setup as friendly as possible

Script

Differences in test coverage
• Platform level tests written by different people from
different teams
• As a result, sometimes test coverage differ between
platforms

Solution
• Using BDD as a tool
• Specify test scenarios with Gherkin
• It forces to have exactly the same tests on every
platform

Foundation of our new CI
• Jenkins (TeamCity? GitLab? Travis CI? AWS CodeBuild?)
• AWS (GCP? Azure? Virtualization? Own hardware?
Kubernetes? Mesos?)
• Docker (Configuration management?)
• Mac (Own hardware? Managed hosting? Virtualization?)

CI Overview
EC2 + EFS
Mac nodes
Jenkins
Artifactory
Gerrit
S3 Device Farm

Workflow: Blocking change before merging
Developer
Gerrit
Chang
e

Workflow: Verify after merging
Gerrit
Developer
S3

Workflow: Release
Gerrit
Artifactory
User
User

Jenkins
EC2 + EFS
Mac nodes
Jenkins
Artifactory
Gerrit
S3 Device Farm

Jenkins
Freestyle jobs Declarative pipeline
Easy to use + -
Maintenance - +
Configuration as Code -* +
Shared lib/code -* +
Reaction on job
status
-* +
Complex scenarios +/- +
WTF per minute min a lot

AWS
EC2 + EFS
Mac nodes
Jenkins
Artifactory
Gerrit
S3 Device Farm

AWS
• Spot Instances - key element for scalable CI
• A way to get resources with discount up to 90%
• But no guarantee of availability!
• We paid around 65 USD for 500+ hours in April
• 0.12 USD per hour -> x3 cheaper then On-Demand price
of the same instance!

AWS
• EFS - it’s a NAS (Network-attached storage)
• Very tricky throughput management
https://docs.aws.amazon.com/efs/latest/ug/performance.html
• You should monitor BurstCreditBalance metric
• Around 25 USD per ~100GB of cache in April

AWS
Use cases:
• We are using it for sharing ccache cache to speed up
C++ compilation
• You can use it to share something between EC2
instances

AWS
• S3 – object storage
• S3 isn’t a file system!
• Not just for storing files
• 2.52 USD in April for 1M+ of requests and 400GB+ of
output traffic

AWS
Use cases:
• We are using it to store temporary build artifacts
(demo apps, reports, logs, screenshots, etc…)
• We are delivering releases to public using S3 with
CloudFront
• I’m hosting my personal blog on S3
• Once I used it like a DB for a hackathon project,
because it was easy to use and cheap J

AWS

AWS
• Lambda – new paradigm
• Run code on events without thinking about
infrastructure
• Pay for amount of resources and time
• Lambda’s from Java to Go -> 2x money savings
• 250K+ requests and 180K seconds in April for 3.16 USD*

AWS
Use cases:
• Posting review feedback to code changes in Gerrit
• Killing unused EC2 instances
• Updating AMI (like Packer but with our specifics)
• Collecting metrics from test apps
• In general, it’s a great glue for AWS services

AWS
• Device Farm – mobile devices in cloud
• Supports fuzzing (random input events), mobile web,
android and iOS testing
• Supports Appium, Calabash and “native” tools
• Remote access to devices
• https://github.com/artemnikitin/devicefarm-ci-tool

AWS
Must Do in the Cloud:
• Immutable infrastructure
• Infrastructure as Code
• Prepare for failures
• Monitor your spending's

AWS
https://github.com/open-guides/og-aws
https://www.expeditedssl.com/aws-in-plain-English
https://read.acloud.guru/
https://www.slideshare.net/hornsby/10-lessons-from-10-years-of-aws
https://seleniumcamp.com/talk/scaling-execution-of-protractorjs-on-aws-
lambda-with-selenoid/
https://aws.amazon.com/blogs/devops/ui-testing-at-scale-with-aws-
lambda/
https://www.youtube.com/watch?v=EDZBYbEwhm8
https://www.youtube.com/watch?v=73-G2zQ9sHU
https://www.youtube.com/watch?v=ZgxZCXouBkY
https://www.youtube.com/watch?v=w7X4gAQTk2E

Docker
EC2 + EFS

Docker
• Docker – tool for operating-system-level virtualization,
aka containerization
• We are using it to encapsulate environment for builds
• We are ”backing” images in AMI
• It creates a problem. We need to keep AMI up to date

Docker

Mac nodes
EC2 + EFS
Mac nodes
Jenkins
Artifactory
Gerrit
S3 Device Farm

Mac nodes
• It’s ok to manage 1 or 2 node(s) manually
• It’s become a nightmare on a higher scale
• Not everything can be automated easily

Mac nodes
• Ansible – tool for configuration management
• Works for Linux, MacOS, Windows
• What was done: managing operations via CLI
• What is still missing: how to manage system stuff?
(MacOS updates, Xcode updates, etc…)

Mac nodes

In the end…
• We have our own CI setup what fits our needs
• CI is relatively cheap
• It can scale until we have money
• We found some ways of managing Mac nodes
automatically

Thank you
Contact
Artem Nikitin hi@artemnikitin.com artemnikitin artemnikitin

Building CI from scratch

More Related Content

What's hot

Similar to Building CI from scratch

Recently uploaded

Building CI from scratch