Project Goals• Better and more reliable insight on how the build system is performing • How long jobs wait in the queue before even starting? • How many pushes per month to different branches? • How long does a build run take to complete? • Which platforms/builders take the longest? • How long do individual builds/unittests/talos take? • What is the status of existing jobs? • What is the infrastructure load? • Do we need more machines? What platform? • When is the best time for a downtime?
Dashboards/Reports• Wait Times Report• Pushes Report• End to End Times Report • Build Run Report • TryChooser Report• Average Time per Builder Report • Builder Report
Buildbot Build Pool * 4 Build Masters * 300 Slaves Try Build Pool * 1 Build Master * 200 Slaves Test Build Pool * 7 Test Masters * 371 Slaves
Build Run Life Cycle 169 Build Requests: • Builds: 11 platforms • Unittests: 108 • Talos: 50
End to End Report• How long does a build run take to complete?• e2e time averages: • ~6h 7m, m-c Sep • ~7h 44m, try Sep • ~9h 51m, m-c Aug • ~11 10m, try Aug• Problem: nightly builds have no revision, nightly tests have a wrong revision
End to End Report• Outliers: • 500h build run / failed, wait times 500h • 60h build run -> 1 talos 52h / successful • 19h build run -> 1 talos 17h / exception • 127h build run -> Cancelled jobs
Average Time / Builder• How much time is it spent per builder- type?How much time is itspent on each platform?
Average Time / BuilderHow much time per How much time pereach Linux builder? each Leopart optimized job type?