Lots of Content? Check. Big assets? Oh yeah. See how Pixar makes movies using Perforce. Find out how to scale your version control system once your data goes from gigabytes to terabytes.

1. BDAM:
Big Data
Asset Management
Mark Harrison - Mike Sundy
{mh,msundy}@pixar.com

2. No Recording

3. What is Asset Management?
• Long-Lived Data
– 50 year charter
• Large Data
– Many TB
• Tight Data/Metadata Integration
– Shot lists, assignments, rights management
• Scalable Data Services
– Human, Render Farm, Build Farm Scale

4. Long Lived Data
• How Templar Project was Started
• Things Change
– Vendors
– Software
– File formats
– Hardware, OS, Storage
• Your Own Requirements Change
– How flexible, “hackable” can you be?

5. Large Data
• Expanding Expectations (include)
• Harrison’s Law of 1 Terabyte (include)
• Harrison’s Time Scale of Data (include)
• Harrison’s law of mentioning Harrison
• Basic Drivers:
– Storage: cheaper
– Expectations: higher
– Time: stays constant

6. Tight Data/Metadata Integration
• Over Time, you lose information about files
• Important Information:
– Assignments, shot lists, rights clearances
• Don’t let data disappear into proprietary hole

7. Scalable Data Services
• Picture of single server
• Applications need to scale appropriately
• Avoid bottleneck of single server (if possible)
• Infrastructure should handle data bandwidth
• Note: Bottlenecks will always move, but
always exist

8. Templar
• Pixar’s Proprietary Asset Management System
• Handles all studio data and metadata
– feature films, shorts, special projects
– artwork, scripts, movie frames, simulation data,
project management data
• 50 year Timeframe
– All metadata, data can be accessed and used
through 2053

9. Templar Asset Management
• Long-Lived Data
– 50 year charter
• Large Data
– Many TB
• Tight Data/Metadata Integration
– Shot lists, assignments, rights management
• Scalable Data Services
– Human, Render Farm, Build Farm Scale

10. Templar: Long Lived Data
• Federated Architecture
– Loosely Coupled
– Software hooks into pipeline
• Pieces can be upgraded incrementally
– Software, file formats
• Exit Strategy Orientation
– Standards, access to internals

11. Templar Large Data
• Large, Fast Storage
– File system caching, etc.
• Scalable Storage Software
– proprietary system for non-revisioned files
– Perforce
• Both horizontal and vertical scalability

12. Templar Data/Metadata Integration
• “Federated” System
– No monolithic application that “does everything”
• Instead, “best in class” programs that
interoperate
– modeling, rendering, storage, etc.
• Lightly Coupled Applications to Metadata
• Metadata in Relational DB, eg Oracle
• Expandable Metadata Schema

13. Templar: Scalable Data Services
• Multiple Access Methods for Assets
– File system, HTTP, direct Perforce
• Load Balancer, multiple servers (e.g. HTTP)
• File System optimizations (clusters, caching)
• Perforce: use LINKATRON
• Asynchronous Queuing

14. Perforce
• In use at Pixar since 2000 for code only
• File revision history goes back to 1983
• First Perforce-managed film: Toy Story 3

15. Perforce: Long Lived Data
• Matches “exit strategy” requirements
– All data, metadata extractable, hackable
– ,d magic – direct flat file storage access on back-end
• Types of Data – not just code!
– art – reference and concept art – inspirational art for film
– tech – show-specific data. e.g. models, textures, pipeline
– studio – company-wide reference libraries. e.g. animation
reference, configuration files, Flickr-like company photo site
– tools – code for our central tools team, software projects
– dept – department-specific files. e.g. marketing images
– exotics – patent data, casting audio, data for live action shorts,
story gags, theme park concepts, intern art show

16. Perforce: Large Data
• Vertical Scalability
– 900 GB single file
– 6.5 TB checkin
– 47 TB largest single depot
– 160 TB total Perforce storage across all depots
• Leverage Perforce features to reduce data:
– Used +S auto-purge filetype to save 40% of
storage on Toy Story 3 (1.2 TB)
– Wrote a script to de-duplicate files, using p4
checksum data. Saved 1 million files and 1 TB

17. Perforce: Data/Metadata Integration
• How does it integrate with Templar?
– stores the files
– version control
– the “authority” for source writes
– triggers for synchronous operations (e.g.
LINKATRON)

18. Perforce: Scalable Data Services
• Horizontal Scalability
– 190+ depots
– 58 VMWare servers
– 26 million submitted changelists
• Server architecture
– Scale out
• Performance on one depot won’t affect another
• Easier administration/downtime scheduling
– Virtualization
• 95% of physical hardware performance with greater flexibility
• 15 minutes to build new server
• Automated p4 server setup (squire)
– 8 seconds to run script to create new p4 instance

19. Conclusion
• Templar and Perforce met our four requirements:
– Long-Lived Data
• 50 year charter
• confidence in retrieving data due to access to internals
– Large Data
• Hundreds of TB
• 500 TB depot on horizon
– Tight Data/Metadata Integration
• Rock solid file management
• users trust it
– Scalable Data Services
• 190 depots
• hundreds more to come – we keep finding new uses