[Pixar] Templar Underminer


Published on

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

[Pixar] Templar Underminer

  1. 1. Templar UnderminerTemplar is Pixar’s asset management system. Its charter is to manage assets and assetmetadata for a 50 year period. The Underminer manages Templar backend depot storage. Itis in production use.Phase I implements multi-partion storage for Templar Perforce instances. Specifically, somepartions can be made read-only, enabling the backup team to perform more efficientbackups.Fundamental Underminer IdeaPerforce will write to the main partition in its usual way. The depot must be configured so thatall files are of file type “+F”.The underminer will create directory structures on the target that mirrors the directorystructure on the main partition. Specified files in the main partition will be copied to theircorresponding location on the active target, and corresponding symlinks will replace the filesin the main partition. (note: We don’t use archive triggers as we’re not interested in replacingP4’s extensively tested and validated reading/writing code with new code that couldpotentially fail in such a was as to result in permanent data loss.)In Phase I this operation will always be performed independently of any Perforce operation. Itcan be done automatically (e.g. via a cron job) or manually (as part of a partion addition).In Phase I The database tables will be maintained manually.
  2. 2. LINKATRON links will be updated as part of this process.note: LINKATRON overview is available here:http://www.perforce.com/sites/default/files/harrison-sundy-perforce-pixar-linkatron-paper.pdfThe main partition can consist ofall files. this is identitical to “standard” Perforce storage.all symlinks. this will be the state following a full undermining of a depot with nosubsequent modifications to the depot.mix of files and symlinks. this will be the state following a partial undermining, or of afull undermining followed by subsequent Perforce depot modifications.In all cases, Perforce views the depot as a standard depot and is not aware of nor concernedwith the modifications to the underlying storage mechanism.All state-changing operations check to make sure the program has been run as the diskfarmuser.PhasesPhase I – no Perforce dendenciesPhase II – Perforce “create file” eventPhase III – Perforce “file staged” import (mv instead of cp)Phase IDatabaseWe will store underminer partition configuration information in the Templar database(currently in Oracle). Most operations will go from the “main” partition to the “target” partition.By keeping the configuration in the database, there are far fewer opportunities for mistakes.
  3. 3. Partition Typesmainwhere Perforce writes the data.targetwritable partition that is the default target for underminer operations.archived (r/o)read-only partitions have links pointing to them but (of course) are never written to and donot need to be traversed looking for files to back up.writable (r/w)a read-write partition is a writable partition that is not the active target. It most likely willexist for a short time when a new active target is specified and the old active target hasnot been made r/o. We treat a r/w partition in the same logical manner as a r/o partitionbut note that file system modifications can be made by activities outside the underminer.Deliverablesdocumentationdatabase schema populated for our various depotsunderminer executable as detailed belowcrontab configuration for verification(possible) crontab configuration for automatically updating storageIssues and Investigation+S filesneed to make sure that as we expire revisions we clean up the storage.+S obliterationneed to pester p4 for the obliterate trigger. as a practical matter we don’t do muchobliteration though.+S deduplicationhow does this affect our backend dedupe? can we re-dedupe once the files have beencopied to the target partition?Perforce FeaturesNeed to show this to the perforce guys and get their feedback and see what weneed to do to get the phase 2 and 3 dependencies on their scheduleReporting
  4. 4. These are things which are useful to report from the underminer command line.database dumpshow configuration for a particular depotshow status for a particular depot or p4spec (e.g. what files have been undermined,etc.)Command Lineunderminer [-n] cmd [-a] //p4path ...operation will be similar to p4 files. Only the head version will be used unless the -aflag is specified. P4 paths may be repeated. All path resolution will be performed via p4files.“-n” means “no don’t do it”. Work will be displayed that would have been done.Commands:stdmoveperform a “standard” move. This will move all files moved since the last stdmoveoperation. The changelist of the last stdmove operation is stored in the database and willbe updated upon successful completion of this command.movemove the files specified in the //p4paths from the main partiton to the active target. (Wemay allow specification of -from and -to flags for testing and balancing of partitions.)dbdumpdump the state of the underminer configuration. If no depots are specified, all depots willbe dumped.reportreport on the status of the specified //p4spec.spaceusedreport on space used for all partitions.verifyrwverify the read/write and read/only status of all partitions.dbverifyverify that the database specification is consistent for the specified repositories.verifyverify that all files specfied in the //p4files are functional.inconsistenciesreport “harmless” inconsistencies that would not affect correct operation.
  5. 5. Database DumpHere’s what the database dump looks like. Without going into details on the schema it shouldgive you a general idea of how a depot is configured.$ ./underminer dbdump---(markive, 001, main, /pixar/templar/markive/markive/rep)(markive, 002, target, /pixar/templar/markive-002/markive/rep)(markive, 003, writable, /pixar/templar/markive-003/markive/rep)---(main, the main extent that p4 knows about)(archived, an archived, non-writable extent)(writable, not active, presumably will be archived soon)(target, the extent to which p4 data is being copied)Notes for future workif the decision is made to back up thumbnail partitions (perhaps because of transcodedvideo), the thumbnail partions should likewise be undermined.