1. Anamaria Stoica
Release Engineering Intern
Summer 2010

2. Dashboards
for the
build infrastructure

3. Project Goals
• Better and more reliable insight on how the
build system is performing
• How long jobs wait in the queue before even starting?
• How many pushes per month to different branches?
• How long does a build run take to complete?
• Which platforms/builders take the longest?
• How long do individual builds/unittests/talos take?
• What is the status of existing jobs?
• What is the infrastructure load?
• Do we need more machines? What platform?
• When is the best time for a downtime?

4. Dashboards/Reports
• Wait Times Report
• Pushes Report
• End to End Times Report
• Build Run Report
• TryChooser Report
• Average Time per Builder Report
• Builder Report

5. Buildbot
Build Pool
* 4 Build Masters
* 300 Slaves
Try Build Pool
* 1 Build Master
* 200 Slaves
Test Build Pool
* 7 Test Masters
* 371 Slaves

6. Build Run Life Cycle
169 Build Requests:
• Builds: 11 platforms
• Unittests: 108
• Talos: 50

7. The Great Wall of
Mac minis

8. Technologies
• Python
• SQLAlchemy
• Pylons
• Google Chart Tools (Visualization API) &
Data Source Python Library

9. Reports Overview
• Permissive parameters
• startime, endtime: unixtimestamp
• int_size: interval size (seconds)
• mpb: minutes per block (minutes)
• filters: platform, build_type, job_type,
detail_level
• JSON (format=json)
• Visualization API (charts)

10. Wait Times Report
• How much jobs wait before being assigned
to free Slaves (wait to start)
Perfect Day
Buildpool / Sep 29

11. Wait Times Report
Less ideal day: Testpool / Sep 29
Bad day: Trybuildpool / Aug 17

12. Monthly Report
September
Testpool
Quarter Report
June-Oct
Trybuildpool

13. Pushes Report
• How many pushes per branch?
Daily Report
Sep 22

14. Weekly Report
Sep 13-20
m-c vs. try

15. Monthly Report
July
Annual Report
June-Sep

16. End to End Report
• How long does a build run take to
complete?
• e2e time averages:
• ~6h 7m, m-c Sep
• ~7h 44m, try Sep
• ~9h 51m, m-c Aug
• ~11 10m, try Aug
• Problem: nightly builds have no revision,
nightly tests have a wrong revision

17. End to End Report
• Outliers:
• 500h build run / failed, wait times 500h
• 60h build run -> 1 talos 52h / successful
• 19h build run -> 1 talos 17h / exception
• 127h build run -> Cancelled jobs

18. Average Time / Builder
• How much time is it spent per builder-
type?
How much time is it
spent on each platform?

19. Average Time / Builder
How much time per How much time per
each Linux builder? each Leopart optimized
job type?

20. Demo!

21. Thanks!
• Chris AtLee (catlee)
• John O’Duinn (joduinn)
• Lukas Blakk (lsblakk)
• Release Engineering Team
• Erica, Julie, Karen
• Interns!