Get an in-depth look at how NetherRealm Studios, creator of the Mortal Kombat franchise, addresses the challenges that modern game development poses to network architecture and resources.

1. 1
Game Studio Perforce Architecture
Ryan Mensching
Senior Network Administrator
NetherRealm Studios/WB Games Logo area

2. 2
The Setup
• Established Studio with track record
• Mortal Kombat (1992 – 2011)
• DC Comics – Injustice (2013), Arkham City Lockdown
• AAA Games on multiple platforms
• 360
• PS3
• WiiU
• Vita
• iOS

3. 3
Historical Perspective
• Team has been making games for 20 plus years.
• “Things will get done.”
• Outages cost money, ship dates, and sanity.
• “Crunch” is a reality.
• Multi-Year development cycle, two years
• Used other tools in the past, Source Safe

4. 4
Statistics
• Games are Code and Binary Art Assets
• Textures
• Animations
• Cinematics
• Lighting
• Sync of current project at head is 20G
• Sync of source art assets is 545G

5. 5
Visuals
• 460+ CL’s per day at peek intervals (1000’s of builds)
• Some 20G of changes per day at peek intervals

6. 6
Products
• As a studio we have 3 software products that are
being produced at any given time.
• Engine
• Games
• Art Tool/Game Editor

7. 7
Challenges
• As IT we had several hurdles to overcome
• Aging Hardware
• Aging processes
• Management woes
• Insight into systems
• Lack of agility to meet business demand

8. 8
Strategic Goals

9. 9
Strategic Goals
• Game development support is a balancing of
immediate gratification and resources.
• IT business driven goals/pillars
• Space
• Speed
• Flexibility
• Reliability
• Management

10. 10
Space
• Available, can’t be used if its locked up, i.e. thin
provisioning
• Abstraction, array layer should not need
configuration to allocate space and at volume
layer should not require reconfig for volume
expansion.

11. 11
Speed
• Entire VM platform, OS, and mass storage.
• Performance needs to be maintained in relation
to consumption.

12. 12
Flexibility
• Works in conjunction with other pillars.
• Stands alone as a goal in games environment.
• Freedom of protocol choices.
• Ease of VM and hardware additions.

13. 13
Reliability
• Redundancy should be inherent.
• Should not require special plans or procedures.
• If the reliability path is to complex issues will
arise.

14. 14
Management
• IT Glue
• Proper management makes all the pillars come
together.
• Self documenting

15. 15
Solutions
How as an IT organization do we accomplish these goals and
vision?

16. 16
Solutions
• Solid Layer 2 and 3 core
with 10G distribution.
• Previous experience with
VM, Storage over IP and
infrastructure testing.

17. 17
Research
• Built VM farm with
existing hardware and
repurposed array for
iSCSI.
• Benchmarks were in the
top on the Perforce
Benchmark DB.
• Ran real world project
on systems.

18. 18

19. 19
Storage Platform
• Key building block of infrastructure design.
• iSCSI management proved not agile enough,
LUN management chores. Redundancy requires
complex configurations.
• NFS provided equal or better performance.

20. 20
NFS
• True thin provisioning, space consumed and given
back.
• No LUN changes.
• Volumes size can be adjusted on the fly.
• Strong VMware support.
• NetApp engine
• Snapshots
• Native NFS among other protocols
• Key to P4 revision file storage design.

21. 21
Virtualization
• VM hardware needs to be “wide and fast”.
• As much RAM as possible to improve VM operations
and better disk caching.
• Cores should have highest clock speed possible.
• Perforce uses 2 vCPUs.
• Cisco UCS
• Hardware abstraction
• Native Unified Fabric, 10G
• VM Passthrough

22. 22
OS
• RHEL for all Perforce servers and many
applications.
• Linux provides best usage of resources for VM’s
and disk caching.
• Current project Perforce servers have 32GB of
RAM.
• Able to achieve 440+Mb sustained transfer to p4
clients.

23. 23
Reliability
• Redundancy at array level with NetApp SnapMirror.
• Volume and VM level snapshots.
• Provides ability to single file restore directly back to
depots/servers.
• UCS blades multi-homed to core; dual chassis
uplinks, dual fabric uplinks, dual core switches.
• VMware HA clustering
• Perforce uses proxies for load balancing and
checkpoints for integrity.
• Moving to replicas.

24. 24
Management Tools
• NetApp OnCommand
• UCS Manager
• VMware
• Plugins, SMVI
• Active monitoring of P4
services and disk health.

25. 25

26. 26
Conclusion
• Perforce is very capable of scaling to speed and
data intense development.
• Requires investments in engineering and
infrastructure.
• Each environment is different, but can share
philosophies.
• Abstraction, abstraction, abstraction

27. 27
Thank you
Questions?
Ryan Mensching
rmensching@wbgames.com