Your SlideShare is downloading. ×
  • Like
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014

  • 90 views
Published

This slide deck was used during the RELENG 2014 conference at MV, California. …

This slide deck was used during the RELENG 2014 conference at MV, California.
It describes the issues faced by Mozilla's continuous integration & how they saved large amounts of money.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
90
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Mozilla’s Hybrid Continuous Integration @armenzg http://armenzg.blogspot.ca
  • 2. @MozReleng http://planet.mozilla.org/releng
  • 3. Once upon a time…
  • 4. THE END
  • 5. Fast growth will cause your infrastructure pain and lots of costs.
  • 6. How much have we saved? Roughly 60-70%
  • 7. • No in-house cost • Higher EC2 usage
  • 8. Term definitions Quick intro to Mozilla’s CI Inter data-center issues AWS costs
  • 9. TERMS • Push – Commit – Code changes • VPN – Virtual Private Network • AWS – Amazon Web Services •EC2 – Elastic Compute Cloud •EBS – Elastic Block Store •S3 – Simple Storage Service • Buildbot
  • 10. Mozilla’s products: •Non-for-profit & users first •Desktop, Android, Thunderbird & Firefox OS •~1/2 billion Firefox desktop users •50M-100M Android installs •90+ locales •Firefox OS phones in 13+ countries Mozilla’s CI
  • 11. 1M+ build jobs 10M+ test jobs 90+ wall-time years Every year*: * Back of the envelope Mozilla’s CI
  • 12. Mozilla’s CI
  • 13. Mozilla’s CI
  • 14. RelEng’s CI: •2 in-house data-centers •2 Amazon regions •VPC •VPN link between them •Internet data transfers through SSL Mozilla’s CI
  • 15. INTER DATACENTER ISSUES • Loss of stateful connections • Link capacity Note: Fixing these also brings costs down
  • 16. http://atlee.ca/blog/posts/aws-networks-and-burning-trees.html
  • 17. AWS COSTS • Spot vs On-Demand instances • Shorten job’s wall time • Smarter bidding • Less EBS
  • 18. http://rail.merail.ca/posts/ec2-spot-instances-experiments.html http://vps.glek.net/graphite Spot vs On-Demand instances
  • 19. http://atlee.ca/blog/posts/initial-jacuzzi-results.html Shorten job’s wall time Incremental builds
  • 20. http://rail.merail.ca/posts/ec2-spot-instances-experiments.html http://vps.glek.net/graphite Spot vs On-Demand instances
  • 21. https://bugzilla.mozilla.org/show_bug.cgi?id=972562 Smarter bidding - 75% less on-demand75% less on-demand - More instance typesMore instance types - Choose the right regionChoose the right region - Pricing analysisPricing analysis
  • 22. Less EBS EBS cost us 20% of all costs
  • 23. What we have covered • Analyze inter data-center flows • Smart and flexible bidding • Reduce wall time • Reduce disk usage
  • 24. @armenzg - @MozReleng