How to build scalable CI system that will be able to verify each commit to repository.

1. Running tests for
every commit
Gerrit, Jenkins, Docker, AWS
Alexander Akbashev
DevOps Specialist
16/04/2016

2. Nice to meet you

3. HERE
• employs 6,500 people in 55 countries.
• produces maps for almost 200 countries.
• enables four out of five in-car nav systems in North America and Europe
• enables mobile, web, and enterprise solutions for global industry leaders.
HERE is a leading location cloud providing superior mapping,
services and location intelligence across the automotive,
enterprise and internet industries.

5. 55
Stack
• CI: Jenkins
• Tests: RobotFramework/Cucumber, GoogleTest, etc
• Target platforms: Windows 32/64, Linux 32/64, QNX, Android/iOS
Copyright © HERE 2016

6. 66
Where we were
• Build every 40 minutes for several commits
• Review on-demand
• Each team could have own Jenkins
• Each team could have CI pipeline
Copyright © HERE 2016

7. 77
Desired
• Build on every commit
• Review for every commit
• One Jenkins
• Unified CI pipeline
Copyright © HERE 2016

8. 88
Numbers*
• 55 - 65k builds/day
• 5 - 6k builds/hour
• 1.8 - 2k commits/day
• 1100 -1200 executors
• 19 minutes
* - for the main productCopyright © HERE 2016

9. </cut>

10. 1010
Big picture
Code Gerrit Jenkins Gerrit Repository
Copyright © HERE 2016

11. 1111
Gerrit
• Git
• Open source
• Plugins
• Scale
• JVM
Copyright © HERE 2016

12. 1212
Gerrit
• Code Review ( +2 )
• Test passed ( +1 )
Feedback time: 2 minutes for build fail, 19 minutes for success
Copyright © HERE 2016

13. 1313
Jenkins
• OpenSource
• Plugins
• Community
• JVM
Copyright © HERE 2016

14. 1414
AWS EC2 + EC2 Plugin
• Pool
• Maintenance
• Scalability
• Price
Copyright © HERE 2016

15. 1515
Docker
• DRY
• Isolation
• Limits
• Repeatability
Copyright © HERE 2016

16. 1616
Jenkins is not for
• Artifacts
• Statistics
• Managers
• Lazy people
Copyright © HERE 2016

17. 1717
Pipeline plugin*
• Reduce monkey work
• Review
• Configuration as code
* - by default in Jenkins 2.0Copyright © HERE 2016

18. 1818
In-house dashboard + Groovy Event Listener Plugin
• Nice UI
• Quick UI
• No action buttons
• Results for release scope
Copyright © HERE 2016

19. 1919
AWS S3 + S3 plugin
• Keep artifacts longer
• Save money
• Faster
Copyright © HERE 2016

20. Room for improvements

21. 2121
Divide et empera
Copyright © HERE 2016

22. 2222
Divide et empera
Copyright © HERE 2016

23. 2323
Divide et empera
Copyright © HERE 2016

24. 2424
Profiling
• IO
• CPU
• Memory
• Time budget
• Cut explicitly
Copyright © HERE 2016

25. 2525
Jenkins
• Yourkit
• GC logs
• FlightRecorder
“All magic comes with a price”
Copyright © HERE 2016

26. 2626
Android
• TestDroid
• Battery Life
• OOM Killer
• Data updates
Copyright © HERE 2016

27. Stability

28. 2828
Angry Devs
0
20
40
60
80
100
100 90 80 70 60 50 40 30 20 10 0
Copyright © HERE 2016

29. 2929
Verification jobs
• Same tests
• Schedule
• Pass rate - 100%
Copyright © HERE 2016

30. 3030
Dashboards
• Queue Size
• Chance to submit
• Feedback
• Executors
• Flaky tests/jobs
• etc
Copyright © HERE 2016

31. 3131
Automatization
• Sanity checks
• Dead nodes
• Configuration
Copyright © HERE 2016

32. Summary

33. 3333
Plugins
• S3 Plugin
• EC2 Plugin
• Build Failure Analyzer Plugin
• Groovy Event Listener Plugin
• Hierarchy Killer Plugin
Copyright © HERE 2016

34. 3434
Summary
• Automate automatization
• Be strong
• Visibility
• Relax&enjoy
Copyright © HERE 2016

35. QA
alexander.akbashev@here.com
github: Jimilian
habrahabr: Jimilian