Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
•Download as PPT, PDF•
0 likes•759 views
This slide deck was used during the RELENG 2014 conference at MV, California. It describes the issues faced by Mozilla's continuous integration & how they saved large amounts of money.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Similar to Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014 (20)
Recently uploaded
Recently uploaded (20)
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
- 4. Once upon a time…
- 8. THE END
- 9. Fast growth will cause your infrastructure pain and lots of costs.
- 10. How much have we saved? Roughly 60-70%
- 11. • No in-house cost • Higher EC2 usage
- 12. Term definitions Quick intro to Mozilla’s CI Inter data-center issues AWS costs
- 13. TERMS • Push – Commit – Code changes • VPN – Virtual Private Network • AWS – Amazon Web Services •EC2 – Elastic Compute Cloud •EBS – Elastic Block Store •S3 – Simple Storage Service • Buildbot
- 14. Mozilla’s products: •Non-for-profit & users first •Desktop, Android, Thunderbird & Firefox OS •~1/2 billion Firefox desktop users •50M-100M Android installs •90+ locales •Firefox OS phones in 13+ countries Mozilla’s CI
- 15. 1M+ build jobs 10M+ test jobs 90+ wall-time years Every year*: * Back of the envelope Mozilla’s CI
- 16. Mozilla’s CI
- 17. Mozilla’s CI
- 18. RelEng’s CI: •2 in-house data-centers •2 Amazon regions •VPC •VPN link between them •Internet data transfers through SSL Mozilla’s CI
- 19. INTER DATACENTER ISSUES • Loss of stateful connections • Link capacity Note: Fixing these also brings costs down
- 26. AWS COSTS • Spot vs On-Demand instances • Shorten job’s wall time • Smarter bidding • Less EBS
- 27. http://rail.merail.ca/posts/ec2-spot-instances-experiments.html http://vps.glek.net/graphite Spot vs On-Demand instances
- 28. http://atlee.ca/blog/posts/initial-jacuzzi-results.html Shorten job’s wall time Incremental builds
- 29. http://rail.merail.ca/posts/ec2-spot-instances-experiments.html http://vps.glek.net/graphite Spot vs On-Demand instances
- 30. https://bugzilla.mozilla.org/show_bug.cgi?id=972562 Smarter bidding - 75% less on-demand75% less on-demand - More instance typesMore instance types - Choose the right regionChoose the right region - Pricing analysisPricing analysis
- 31. Less EBS EBS cost us 20% of all costs
- 32. What we have covered • Analyze inter data-center flows • Smart and flexible bidding • Reduce wall time • Reduce disk usage