Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ship happens: A better firefox build and release pipeline


Published on

The Firefox build and release pipeline is crucial to delivering our products to customers. Over the past year we have transformed our pipeline to a more robust and scalable system using Taskcluster, Docker and in-tree scheduling. We have also implemented release promotion, which takes existing continuous integration (CI) binaries and transforms them for release, significantly reducing wall clock time.

Attend this session to hear exciting stories about how to replace components of a large running distributed system using the ominously named strangler application approach. I’ll discuss some metrics regarding the end-to-end time for our release process. I’ll also cover how developers can implement changes to transform builds and tests themselves in-tree.

Published in: Technology
  • If you want to enjoy the Good Life: making money in the comfort of your own home with just your laptop, then this is for YOU... 
    Are you sure you want to  Yes  No
    Your message goes here
  • Unlock Her Legs is your passage way to a life full of loving and sex... read more ... ➤➤
    Are you sure you want to  Yes  No
    Your message goes here
  • FREE TRAINING: "How to Earn a 6-Figure Side-Income Online" ... ■■■
    Are you sure you want to  Yes  No
    Your message goes here
  • The #1 Woodworking Resource With Over 16,000 Plans, Download 50 FREE Plans... ◆◆◆
    Are you sure you want to  Yes  No
    Your message goes here
  • There are over 16,000 woodworking plans that comes with step-by-step instructions and detailed photos, Click here to take a look ♥♥♥
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Ship happens: A better firefox build and release pipeline

  1. 1. Kim Moir (kmoir), Mozilla Release Engineering Ship Happens: A better Firefox build & release pipeline
  2. 2. “I am notorious for making impassioned speeches about things nobody cares about.” ― Mindy Kaling, Why Not Me?
  3. 3. Today’s agenda ● Faster pipelines and what they mean for you ● How to try it yourself! ● Lessons learned and what’s next
  4. 4. Mozilla Releng live here
  5. 5. Release times ● 2013 - 11 hours ● 2017 - 4-5 hours
  6. 6. Continuous integration Land code Unit tests Decision graph Builds x N platforms Performance tests Sign Builds
  7. 7. Nightlies Land code Unit tests Decision graph Builds x N platforms Performance tests Sign Builds Generate updates L10n
  8. 8. Release process using release promotion Use existing build artifacts Generate updates L10n Unit tests Decision graph Sign Builds Performance tests Repackage Builds + Move artifacts Refresh update db rules Update websites with release
  9. 9. About:Taskcluster ● Taskcluster is a task execution framework that supports Mozilla’s continuous integration farm + release pipeline It is a set of components that manages task queuing, scheduling, execution and provisioning of resources.
  10. 10. Why: In-tree and Decision Graph ● Build and test configs are all in tree ○ Good news: Developer autonomy ○ Bad news: Developer autonomy ● Decision graph upon push identifies failures more quickly ● Changes can be tested locally and on try
  11. 11. Testing the graph locally ● Generates the full taskgraph. ○ ./mach taskgraph full > full.txt ● Generates an optimized taskgraph ○ ./mach taskgraph optimized > full.txt ● Generates a target taskgraph ○ ./mach taskgraph target -p parameters.yml > target.txt ● Generates a target taskgraph with json to inspect content of graph ○ ./mach taskgraph target --json -p parameters.yml > target.txt
  12. 12. ● Taskcluster config files are under taskcluster/ in tree ○ Example: taskcluster/ci/build/macosx.yml defines mac builds (which actually run on Linux)
  13. 13. Changing tests ● YAML files in taskcluster/ci/test/ files define tests groups by suite name - e.g. mochitest, reftest, talos etc
  14. 14. Why: Docker Containers ● Docker containers for test and build images (not all platforms) ○ Consistent environment to debug build and test failures via one click loaners ○ More self-serve developer loaners
  15. 15. Why: More autoscaling ● Moved more platforms to AWS enable autoscaling in response to bursty load ○ Moved Macosx builds to Linux cross-compile on AWS ○ Moved many Windows builds/tests to AWS
  16. 16. Why: More security ● Better security - Chain of Trust (CoT) between artifacts as they are built, signed and moved to AWS S3/CDNs for download on releases/nightlies ● CoT is the security model for releases ● Task execution is restricted by taskcluster scopes, but that is only one type of authentication ● CoT allows us to trace requests back to the tree and verify each previous task in the chain. ● If CoT fails, the task is marked as invalid
  17. 17. Why+? ● Team learned new things - Docker, transforms, migration strategies, microservices, monitoring ● Future efficiencies - allow us to continue to scale ● Migrate off technologies that did not scale to our needs ● Re-evaluate existing jobs: Are they still needed? Could they be improved?
  18. 18. Timeline for migration ● Jan 20 - Linux Desktop and Android Firefox nightly builds from Taskcluster ● Mar 13 - Mobile beta in Taskcluster ● July 2 - Mac Nightlies in Taskcluster ● Aug 30 - Windows nightlies in Taskcluster ● Nov 14 - Shipped Firefox Quantum in Taskcluster
  19. 19. Approach to migration ● Incremental portions of pool ● Communication ● Checklist ● Monitor capacity and wait times ● Monitor state after migration ● Rollback plan ● Decommission old ● Migrate more
  20. 20. Strangler Application - Martin Fowler
  21. 21. 56 was a rough release ● We had many automation changes ○ New compression format for updates ○ Watersheds for win32->win64 migration for people on 64 bit hardware ○ Win32/Win64 on taskcluster
  22. 22. Operation: Don’t F*ck up 57 ● Implement missing release automation ● Fix our staging environment ● Smooth our merge day process ● Train team members on merges and staging releases ● Run staging releases and merges to iron out any issues before 57 releases ● Write tests to validate update rules for 57 ● Spreadsheet to coordinate update rules with relman
  23. 23. What have we learned? ● Incrementalism - change one thing, evaluate, then change another ● Expectations change. The faster we build, the faster other groups expect to be able to ship ● Staging environment is important to test new automation ● Communication ● Organizational changes ● Consider the operational side, not just landing code
  24. 24. Upcoming work ● In tree release promotion for beta and release builds ● Release process optimizations: measure our release end- to-end times, common failure points with the aim of providing more predictable and stable releases ● Staging releases on try ● More incremental fixes to make things faster
  25. 25. I embrace mistakes, they make you who you are ―Beyoncé
  26. 26. Questions?
  27. 27. Additional Reading ● Justin Wood’s (Callek’s) talks on transforms ● All your nightlies are belong to Taskcluster ● Nightly builds from Taskcluster taskcluster.html ● 2016 retrospective ● What's So Special About "In-Tree?"
  28. 28. Additional Reading ● Chris Cooper Nightlies in Taskcluster go-team ● Chris Cooper Mobile Betas in TC release-promotion-firefox-530b1 ● So you want to rewrite that - Camille Fournier, GOTO conference, Chicago, 2014