Mozilla’s Hybrid
Continuous Integration
@armenzg
http://armenzg.blogspot.ca
@MozReleng
http://planet.mozilla.org/releng
Once upon a time…
THE
END
Fast growth will cause
your infrastructure pain
and lots of costs.
How much have we saved?
Roughly 60-70%
• No in-house cost
• Higher EC2 usage
Term definitions
Quick intro to Mozilla’s CI
Inter data-center issues
AWS costs
TERMS
• Push – Commit – Code changes
• VPN – Virtual Private Network
• AWS – Amazon Web Services
•EC2 – Elastic Compute Cl...
Mozilla’s products:
•Non-for-profit & users first
•Desktop, Android, Thunderbird & Firefox OS
•~1/2 billion Firefox deskto...
1M+ build jobs
10M+ test jobs
90+ wall-time years
Every year*:
* Back of the envelope
Mozilla’s CI
Mozilla’s CI
Mozilla’s CI
RelEng’s CI:
•2 in-house data-centers
•2 Amazon regions
•VPC
•VPN link between them
•Internet data transfers through
SSL
M...
INTER DATACENTER ISSUES
• Loss of stateful connections
• Link capacity
Note: Fixing these also brings costs down
http://atlee.ca/blog/posts/aws-networks-and-burning-trees.html
AWS COSTS
• Spot vs On-Demand instances
• Shorten job’s wall time
• Smarter bidding
• Less EBS
http://rail.merail.ca/posts/ec2-spot-instances-experiments.html
http://vps.glek.net/graphite
Spot vs On-Demand instances
http://atlee.ca/blog/posts/initial-jacuzzi-results.html
Shorten job’s wall time
Incremental builds
http://rail.merail.ca/posts/ec2-spot-instances-experiments.html
http://vps.glek.net/graphite
Spot vs On-Demand instances
https://bugzilla.mozilla.org/show_bug.cgi?id=972562
Smarter bidding
- 75% less on-demand75% less on-demand
- More instance...
Less EBS
EBS cost us 20% of all costs
What we have covered
• Analyze inter data-center flows
• Smart and flexible bidding
• Reduce wall time
• Reduce disk usage
@armenzg - @MozReleng
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014
Upcoming SlideShare
Loading in...5
×

Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014

168

Published on

This slide deck was used during the RELENG 2014 conference at MV, California.
It describes the issues faced by Mozilla's continuous integration & how they saved large amounts of money.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
168
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mozilla's hybrid continuos integration - RELENG 2014 Conference - April 11th, 2014

  1. 1. Mozilla’s Hybrid Continuous Integration @armenzg http://armenzg.blogspot.ca
  2. 2. @MozReleng http://planet.mozilla.org/releng
  3. 3. Once upon a time…
  4. 4. THE END
  5. 5. Fast growth will cause your infrastructure pain and lots of costs.
  6. 6. How much have we saved? Roughly 60-70%
  7. 7. • No in-house cost • Higher EC2 usage
  8. 8. Term definitions Quick intro to Mozilla’s CI Inter data-center issues AWS costs
  9. 9. TERMS • Push – Commit – Code changes • VPN – Virtual Private Network • AWS – Amazon Web Services •EC2 – Elastic Compute Cloud •EBS – Elastic Block Store •S3 – Simple Storage Service • Buildbot
  10. 10. Mozilla’s products: •Non-for-profit & users first •Desktop, Android, Thunderbird & Firefox OS •~1/2 billion Firefox desktop users •50M-100M Android installs •90+ locales •Firefox OS phones in 13+ countries Mozilla’s CI
  11. 11. 1M+ build jobs 10M+ test jobs 90+ wall-time years Every year*: * Back of the envelope Mozilla’s CI
  12. 12. Mozilla’s CI
  13. 13. Mozilla’s CI
  14. 14. RelEng’s CI: •2 in-house data-centers •2 Amazon regions •VPC •VPN link between them •Internet data transfers through SSL Mozilla’s CI
  15. 15. INTER DATACENTER ISSUES • Loss of stateful connections • Link capacity Note: Fixing these also brings costs down
  16. 16. http://atlee.ca/blog/posts/aws-networks-and-burning-trees.html
  17. 17. AWS COSTS • Spot vs On-Demand instances • Shorten job’s wall time • Smarter bidding • Less EBS
  18. 18. http://rail.merail.ca/posts/ec2-spot-instances-experiments.html http://vps.glek.net/graphite Spot vs On-Demand instances
  19. 19. http://atlee.ca/blog/posts/initial-jacuzzi-results.html Shorten job’s wall time Incremental builds
  20. 20. http://rail.merail.ca/posts/ec2-spot-instances-experiments.html http://vps.glek.net/graphite Spot vs On-Demand instances
  21. 21. https://bugzilla.mozilla.org/show_bug.cgi?id=972562 Smarter bidding - 75% less on-demand75% less on-demand - More instance typesMore instance types - Choose the right regionChoose the right region - Pricing analysisPricing analysis
  22. 22. Less EBS EBS cost us 20% of all costs
  23. 23. What we have covered • Analyze inter data-center flows • Smart and flexible bidding • Reduce wall time • Reduce disk usage
  24. 24. @armenzg - @MozReleng
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×