The document discusses MongoDB's development of their own continuous integration (CI) system called Evergreen, which streamlined their testing process by allowing faster feedback and efficient resource usage. It highlights the key features of Evergreen such as dynamic multi-platform host allocation, improved failure feedback, and significant reductions in iteration time for developers. Ultimately, the system has led to more builds across multiple platforms, enhancing the overall release cycle and developer experience.

1. 1

2. 2
How We Test MongoDB:
Evergreen
Kyle Erf, MongoDB Inc.

3. 3
Let’s Take a Look

4. 44

5. 5

6. 6

7. 7
Evergreen

8. 8
We revolutionized our workflow
by building our own
Continuous Integration System.

9. 9
Our Automated Tests
Two types
Unit - run right after compile
JS - large suites of integration tests tests, longer
Other tests done as part of the QA process

10. 10
Requirements (Today)
800+ hours of computer time to test one commit across all platforms
34 combinations of OSes, Architectures, and compile flags.
10 - 30 commits per day
Engineers need feedback as quickly as possible.

11. 11
Early Automated Testing
Buildbot
A few static hosts
Run whenever things are available
Not great for long tests
Hard to know exactly who broke what
Complicated Python scripts
“Nightly” and “Weekly” test suites
What worked for a 5 people wasn’t working for 10+

12. 12
Need To Do Better
Scale with load
Faster feedback
Precise feedback
Easy to Configure
Keep Costs Down
?

13. 13
Write Our Own?
Existing CI Tools Evergreen
No first-class cloud support Designed for elastic computing
Annoying to Configure Simple YAML Files
No simple pre-commit build support Pre-commit builds
Bad support for multiple platforms Support all of MongoDB’s platforms
Lots of work Lots of work

14. 14
Prototype
Built on MongoDB + Go + AWS in 2013
Good Initial Results
Went all in on our own CI tool
The CI tool of our dreams

15. 15
Features for our Use Case
Dynamic, Multiplatform
Host Allocation
Easily pinpoint the
sources of failures
Test code before
pushing

16. 16
Dynamic Allocation
As many machines as we need
Scale with demand
task-centric rather than machine-centric
Limited by our longest test suite
Cheaper than dedicated hosts

17. 17
How’s It Work?

18. 18
We spin up enough hosts to try
and get all scheduled tasks for
a commit done within one hour.
Each host runs a statically
compiled Evergreen agent,
which accepts tasks from the
main server.
We shut hosts down when the
workload decreases.
Self-healing in case of system
failures, frozen tests.
Server
Agent
Agent
New
Host

19. 19
For Multiple Environments
Cloud Services + Go give us access to all required
OS + Architecture combinations.
Linux, Windows,
Solaris, OSX
32 & 64-bit
AWS/Spot
Mac Stadium
Digital Ocean
Static Hosts

20. 20
Benefits
Orders of magnitude speed-up
Feedback in a couple hours rather than days
We only use the resources we need
AWS Costs Per Day

21. 21
Failure Feedback

22. 22
Failure Feedback
Evergreen only runs a commit to a project every few hours.
On failure, we run previous commits to pinpoint its source.
Like git bisect (but linear)
Commit #3 Commit #2 Commit #1

23. 23

24. 24
Spawn Hosts
Start up a duplicate host for more complex debugging

25. 25
Benefits
Eliminates guesswork from failures
Less stress
Less time

26. 26
Patch Builds
Test the most likely suites before commit

27. 27

28. 28
Benefits
Expected as part of any code review
Fewer platform-related failures
Faster iteration

29. 29
Successes
• More Builds on more platforms
• Better, More Precise Feedback
• Faster Iteration
• Simpler Release Cycle
• General Purpose
• Looks Nice, Too!

30. 30
Successes
• More Builds on more platforms
• Better, More Precise Feedback
• Faster Iteration
• Simpler Release Cycle
• General Purpose
• Looks Nice, Too!
1. Stop Commits
2. Run Long Tests
3. Wait :(

31. 31
Successes
• More Builds on more platforms
• Better, More Precise Feedback
• Faster Iteration
• Simpler Release Cycle
• General Purpose
• Looks Nice, Too!
Build GCC
Build Machine
Image
Build MongoDB

32. 32
Successes
• More Builds on more platforms
• Better, More Precise Feedback
• Faster Iteration
• Simpler Release Cycle
• General Purpose
• Looks Nice, Too!

33. 33
Worth It?

34. 34
Evergreen
See our builds at
evergreen.mongodb.com

35. 35
Evergreen
Or try it out at
github.com/evergreen-ci/evergreen
Apache License - 0.9.0 Alpha

36. 36
Questions?
Why Go?
Our use of MongoDB?
Technical Challenges?
How can I use Evergreen?