Off-the-shelf, Perforce is a remarkably fast and stable system. Perforce admins can change that, however, by adding triggers that are slow, crash, or have external dependencies that cause them to fail. Find out how the Pixar team has addressed this challenge with a checkin-driven work queue system that allows for independent, parallel and asynchronous execution of checkin logic. They'll also briefly describe how they notify 10,000 computers when a checkin is completed.

1. Don t Let A Bad Trigger
Ruin Your Checkin!
Mark Harrison
Tech Lead, Data Management Group
Pixar Animation Studios
.

2. Perforce -- part of the Templar Asset System
• Named after original Asset Managers
• Tracks Data and Metadata for Movie, Art Assets
• High Availability
- 50 year charter: all data, metadata available in 2054
• Services:
- thumbnails
- asset servers
- event queue
• Integrates Perforce with Asset Database
• We also manage our traditional source p4 server
• More details in Perforce 2009 presentation

3. Perforce Growth
Insert chart 1 here
Perforce growth over time

4. Perforce Triggers -- a good thing!
• Very useful to us
• Especially check-in triggers
• Guaranteed to run on all files checked in:
no backdoor avoidance!
• Decoupled us from front end applications:
- language
- release schedule

5. Too much of a good thing!
• Other groups needed to add some check-in time
processing.
• No problem, give them a trigger too.
• But:
- sudden combinatorial explosion of triggers/depots
- big management headache -- who was authorized to
update trigger tables?
- how to manage trigger breakage?
- how to manage trigger hangs?
- what about triggers that needed particular mounts?
• Too much of a good thing == a bad thing!

6. Using Triggers to Enqueue Work
• How can we allow groups to benefit from checkin
triggers?
• How can we protect the groups (and ourselves!) from
each other?
• Solution:
• Standard set of triggers for all depots
• One trigger will enqueue work for other add-time
programs; these programs will execute concurrently after
the checkin is complete.

7. Our Queue Implementation
• Note: our queue implementation is not the silver
bullet
• Putting work into queues is the silver bullet
• There seems to be many queue systems available --
pick one that works well in your environment.

8. Templar Queuing System
• Each program has its own work queue
• Checkin trigger adds entries to work queues
• Work queues are asynchronous: a slow queue
processor will not hinder a fast queue processor
• Queues are persistent (stored in database), surviving
crashes, etc
• Queues are distributed (can run on multiple nodes, one
queue can feed processors on multiple nodes, etc)

9. System Evolution
• Small number of triggers
- nice, but then everybody wanted one!
• Lots of triggers
- great functionality, but lots of breakage and admin
overhead!
• Small number of triggers, many work queues
- triggers became simpler
- queue isolation made everybody happier
- took a long time to get just right