The document outlines the infrastructure and architectural challenges faced by NetherRealm Studios, highlighting their need for reliability, speed, flexibility, and effective management in game development processes. It discusses various strategies and solutions implemented, including the transition to NFS for better performance, the establishment of a VM farm, and emphasizing the importance of redundancy and management tools. Ultimately, it concludes that while Perforce can scale to meet the demands of data-intensive development, ongoing investments in engineering and infrastructure are essential.

1. 1
Game Studio Perforce Architecture
Ryan Mensching
Senior Network Administrator
NetherRealm Studios/WB Games Logo area

2. 2
The Setup
• Established Studio with track record
• Mortal Kombat (1992 – 2011)
• DC Comics – Injustice (2013), Arkham City Lockdown
• AAA Games on multiple platforms
• 360
• PS3
• WiiU
• Vita
• iOS

3. 3
Historical Perspective
• Team has been making games for 20 plus years.
• “Things will get done.”
• Outages cost money, ship dates, and sanity.
• “Crunch” is a reality.
• Multi-Year development cycle, two years
• Used other tools in the past, Source Safe

4. 4
Statistics
• Games are Code and Binary Art Assets
• Textures
• Animations
• Cinematics
• Lighting
• Sync of current project at head is 20G
• Sync of source art assets is 545G

5. 5
Visuals
• 460+ CL’s per day at peek intervals (1000’s of builds)
• Some 20G of changes per day at peek intervals

6. 6
Products
• As a studio we have 3 software products that are
being produced at any given time.
• Engine
• Games
• Art Tool/Game Editor

7. 7
Challenges
• As IT we had several hurdles to overcome
• Aging Hardware
• Aging processes
• Management woes
• Insight into systems
• Lack of agility to meet business demand

8. 8
Strategic Goals

9. 9
Strategic Goals
• Game development support is a balancing of
immediate gratification and resources.
• IT business driven goals/pillars
• Space
• Speed
• Flexibility
• Reliability
• Management

10. 10
Space
• Available, can’t be used if its locked up, i.e. thin
provisioning
• Abstraction, array layer should not need
configuration to allocate space and at volume
layer should not require reconfig for volume
expansion.

11. 11
Speed
• Entire VM platform, OS, and mass storage.
• Performance needs to be maintained in relation
to consumption.

12. 12
Flexibility
• Works in conjunction with other pillars.
• Stands alone as a goal in games environment.
• Freedom of protocol choices.
• Ease of VM and hardware additions.

13. 13
Reliability
• Redundancy should be inherent.
• Should not require special plans or procedures.
• If the reliability path is to complex issues will
arise.

14. 14
Management
• IT Glue
• Proper management makes all the pillars come
together.
• Self documenting

15. 15
Solutions
How as an IT organization do we accomplish these goals and
vision?

16. 16
Solutions
• Solid Layer 2 and 3 core
with 10G distribution.
• Previous experience with
VM, Storage over IP and
infrastructure testing.

17. 17
Research
• Built VM farm with
existing hardware and
repurposed array for
iSCSI.
• Benchmarks were in the
top on the Perforce
Benchmark DB.
• Ran real world project
on systems.

18. 18

19. 19
Storage Platform
• Key building block of infrastructure design.
• iSCSI management proved not agile enough,
LUN management chores. Redundancy requires
complex configurations.
• NFS provided equal or better performance.

20. 20
NFS
• True thin provisioning, space consumed and given
back.
• No LUN changes.
• Volumes size can be adjusted on the fly.
• Strong VMware support.
• NetApp engine
• Snapshots
• Native NFS among other protocols
• Key to P4 revision file storage design.

21. 21
Virtualization
• VM hardware needs to be “wide and fast”.
• As much RAM as possible to improve VM operations
and better disk caching.
• Cores should have highest clock speed possible.
• Perforce uses 2 vCPUs.
• Cisco UCS
• Hardware abstraction
• Native Unified Fabric, 10G
• VM Passthrough

22. 22
OS
• RHEL for all Perforce servers and many
applications.
• Linux provides best usage of resources for VM’s
and disk caching.
• Current project Perforce servers have 32GB of
RAM.
• Able to achieve 440+Mb sustained transfer to p4
clients.

23. 23
Reliability
• Redundancy at array level with NetApp SnapMirror.
• Volume and VM level snapshots.
• Provides ability to single file restore directly back to
depots/servers.
• UCS blades multi-homed to core; dual chassis
uplinks, dual fabric uplinks, dual core switches.
• VMware HA clustering
• Perforce uses proxies for load balancing and
checkpoints for integrity.
• Moving to replicas.

24. 24
Management Tools
• NetApp OnCommand
• UCS Manager
• VMware
• Plugins, SMVI
• Active monitoring of P4
services and disk health.

25. 25

26. 26
Conclusion
• Perforce is very capable of scaling to speed and
data intense development.
• Requires investments in engineering and
infrastructure.
• Each environment is different, but can share
philosophies.
• Abstraction, abstraction, abstraction

27. 27
Thank you
Questions?
Ryan Mensching
rmensching@wbgames.com