How We Test MongoDB: Evergreen
•
3 likes•5,780 views
Comprehensive testing is a critical part of how we develop MongoDB. Because we support myriad features on multiple operating systems and architectures, it takes hundreds of hours to fully test a single commit to the MongoDB server repository. So how do we keep up? In this talk, we will detail our automated testing process and introduce Evergreen, the distributed continuous integration system enabling our engineers to get feedback on their code like never before.
More Related Content
What's hot
What's hot (20)
Similar to How We Test MongoDB: Evergreen
Similar to How We Test MongoDB: Evergreen (20)
More from MongoDB
More from MongoDB (20)
Recently uploaded
Recently uploaded (20)
How We Test MongoDB: Evergreen
- 1. 1
- 2. 2 How We Test MongoDB: Evergreen Kyle Erf, MongoDB Inc.
- 3. 3 Let’s Take a Look
- 4. 44
- 5. 5
- 6. 6
- 7. 7 Evergreen
- 8. 8 We revolutionized our workflow by building our own Continuous Integration System.
- 9. 9 Our Automated Tests Two types Unit - run right after compile JS - large suites of integration tests tests, longer Other tests done as part of the QA process
- 10. 10 Requirements (Today) 800+ hours of computer time to test one commit across all platforms 34 combinations of OSes, Architectures, and compile flags. 10 - 30 commits per day Engineers need feedback as quickly as possible.
- 11. 11 Early Automated Testing Buildbot A few static hosts Run whenever things are available Not great for long tests Hard to know exactly who broke what Complicated Python scripts “Nightly” and “Weekly” test suites What worked for a 5 people wasn’t working for 10+
- 12. 12 Need To Do Better Scale with load Faster feedback Precise feedback Easy to Configure Keep Costs Down ?
- 13. 13 Write Our Own? Existing CI Tools Evergreen No first-class cloud support Designed for elastic computing Annoying to Configure Simple YAML Files No simple pre-commit build support Pre-commit builds Bad support for multiple platforms Support all of MongoDB’s platforms Lots of work Lots of work
- 14. 14 Prototype Built on MongoDB + Go + AWS in 2013 Good Initial Results Went all in on our own CI tool The CI tool of our dreams
- 15. 15 Features for our Use Case Dynamic, Multiplatform Host Allocation Easily pinpoint the sources of failures Test code before pushing
- 16. 16 Dynamic Allocation As many machines as we need Scale with demand task-centric rather than machine-centric Limited by our longest test suite Cheaper than dedicated hosts
- 18. 18 We spin up enough hosts to try and get all scheduled tasks for a commit done within one hour. Each host runs a statically compiled Evergreen agent, which accepts tasks from the main server. We shut hosts down when the workload decreases. Self-healing in case of system failures, frozen tests. Server Agent Agent New Host
- 19. 19 For Multiple Environments Cloud Services + Go give us access to all required OS + Architecture combinations. Linux, Windows, Solaris, OSX 32 & 64-bit AWS/Spot Mac Stadium Digital Ocean Static Hosts
- 20. 20 Benefits Orders of magnitude speed-up Feedback in a couple hours rather than days We only use the resources we need AWS Costs Per Day
- 22. 22 Failure Feedback Evergreen only runs a commit to a project every few hours. On failure, we run previous commits to pinpoint its source. Like git bisect (but linear) Commit #3 Commit #2 Commit #1
- 23. 23
- 24. 24 Spawn Hosts Start up a duplicate host for more complex debugging
- 25. 25 Benefits Eliminates guesswork from failures Less stress Less time
- 26. 26 Patch Builds Test the most likely suites before commit
- 27. 27
- 28. 28 Benefits Expected as part of any code review Fewer platform-related failures Faster iteration
- 29. 29 Successes • More Builds on more platforms • Better, More Precise Feedback • Faster Iteration • Simpler Release Cycle • General Purpose • Looks Nice, Too!
- 30. 30 Successes • More Builds on more platforms • Better, More Precise Feedback • Faster Iteration • Simpler Release Cycle • General Purpose • Looks Nice, Too! 1. Stop Commits 2. Run Long Tests 3. Wait :(
- 31. 31 Successes • More Builds on more platforms • Better, More Precise Feedback • Faster Iteration • Simpler Release Cycle • General Purpose • Looks Nice, Too! Build GCC Build Machine Image Build MongoDB
- 32. 32 Successes • More Builds on more platforms • Better, More Precise Feedback • Faster Iteration • Simpler Release Cycle • General Purpose • Looks Nice, Too!
- 33. 33 Worth It?
- 34. 34 Evergreen See our builds at evergreen.mongodb.com
- 35. 35 Evergreen Or try it out at github.com/evergreen-ci/evergreen Apache License - 0.9.0 Alpha
- 36. 36 Questions? Why Go? Our use of MongoDB? Technical Challenges? How can I use Evergreen?