# 
Frank Compagner 
Guerrilla Games
# 
How we learned to stop worrying and love 
the development cycle 
• Background 
• Iteration 
• Teamwork 
• Stability 
• Releasing 
Iterate 
Launch
# 
• Killzone Shadow Fall – PS4 launch title 
• 200 local users, 150 remote 
• Head revision: 1.8 TB (1.6 M files) 
• Depot size: 8 TB 
• Peak change: 180 GB / week 
• 60 dedicated build servers 
• All in Perforce since 2004
# 
Server 
Proxy Proxy Proxy Proxy 
Build Farm (60) 
Users (200) 
Proxy 
Cambridge 
Proxy 
Guilford 
Proxy 
San Diego 
Proxy 
Amsterdam Shanghai
# 
Commit 
Server 
Edge 
Server 
Build Farm (60) Users (200) 
Cambridge 
Proxy 
Guilford 
Proxy 
San Diego 
Proxy 
Amsterdam Shanghai 
Edge 
Server 
Edge 
Server
# 
Submits 
GB / week
# 
• Monitors all vital stats of server and proxies 
• Simple Python script, produces simple html 
• Works for Windows and Linux servers 
• Proxies cleaned by another Python script 
• Both available in the Perforce Workshop
#
#
# 
Code 
Compile 
Test Programmer 
Link
# 
Code 
Compile 
IDE and Tools 
Test Programmer 
Link 
Distribute 
Incremental linker 
Code 
Hotloading
# 
Model 
Artist 
Test Process 
Hotloading
# 
Model 
Tools 
Artist 
Test Process 
Hotloading Distribute
# 
Programmer 
Rigger 
Tester 
Team 
Designer Animator
# 
Programmer 
Rigger 
Animator 
Animator 
Tester 
Designer 
Modeler 
Programmer 
Programmer 
VFX artist 
Designer 
Audio Engineer 
Modeler 
Designer 
Animator
#
# 
• Does not look like this
# 
• It’s more like this
# 
• OK, like this
# 
• Personal branch is fine (sandbox / streams / Git) 
• Feature branches sometimes work 
• Give control over your environment 
• But create distance to others 
• Team work requires frequent branch switching
# 
• For us, branch switching is always expensive: 
– Amount of data and change 
– Unmergeable files 
– Code-Data dependencies 
• Branches add complexity, we already have that 
• Team effort, we don’t want distance 
• Almost everybody works on the trunk 
• So why isn’t it always broken?
#
# 
Programmer 
Rigger 
Tester 
Team 
Designer Animator
# 
Programmer 
Submit Sync 
Rigger 
Tester 
Team 
Submit 
Submit 
Designer Animator 
Submit 
Submit 
Sync 
Sync 
Sync 
Sync
# 
Programmer 
Test 
Submit Sync 
Rigger 
Tester 
Team 
Submit 
Submit 
Designer Animator 
Submit 
Submit 
Sync 
Sync 
Sync 
Sync 
Test 
Test 
Test 
Test
# 
• Test pre submit (user) 
• Test post submit (builder) 
• Test in parallel 
• Test as fast as possible 
• Still accidents do happen 
User B 
Sync 
Sync Build Game Submit 
Sync Test Game 
Sync Build Tools Submit 
Sync Cook Data 
< 30 minutes 
User A 
Submit Sync 
Test Game 
… 
Sync 
Upload 
Cook Data Upload 
Test
# 
Build machines 
write 
Label 42: change 1200 
Label 41: change 1180 
Label 40: change 1155 
Label 39: change 1150 
… 
Labels.xml 
read 
Safe-Sync Tool 
• Each game build creates a label 
• Stored in Labels.xml file in depot 
• Builds add their result to the label 
• Builds can add artifacts to a label 
• Nobody gets latest (not safe) 
• Everybody uses safe-sync tool
Select 
tests 
Sync to 
get label 
# 
Labels that 
pass tests 
Labels that 
fail tests 
• Custom safe-sync tool shows labels and test results 
• Use it to sync to a known good label 
– This includes artifacts from other builds
#
# 
• We branch for every release (often every week) 
• Nobody works on the release branch 
• Everybody works on the trunk 
• Changes are merged up from main to release 
release 
main 
= release 
= change
• Cherry picking rules! 
• Gives you a lot of control 
• But also lots of confusion 
• And mistakes are easy 
• Good tools can fix this 
• Give flexibility 
• And control 
# 
(See whitepaper for details)
# 
• Iterate as fast as you can 
• Don’t branch too much 
• Test everything 
• Sync should be safe 
• Release all the time 
• Control what you release 
Speed 
Control
# 
Frank Compagner 
frank@guerrilla-games.com
# 
RESOURCES 
Perforce Workshop (P4 monitor, cache cleaner): 
https://swarm.workshop.perforce.com/files/guest/frank_compagner 
ReleaseManager: 
http://www.perforce.com/resources/presentations/user-conference-talks/ 
tools-for-smooth-release-cycle 
http://www.perforce.com/sites/default/files/guerrilla-games-whitepaper.pdf

Working Well Together: How to Keep High-end Game Development Teams Productive

  • 1.
    # Frank Compagner Guerrilla Games
  • 2.
    # How welearned to stop worrying and love the development cycle • Background • Iteration • Teamwork • Stability • Releasing Iterate Launch
  • 3.
    # • KillzoneShadow Fall – PS4 launch title • 200 local users, 150 remote • Head revision: 1.8 TB (1.6 M files) • Depot size: 8 TB • Peak change: 180 GB / week • 60 dedicated build servers • All in Perforce since 2004
  • 4.
    # Server ProxyProxy Proxy Proxy Build Farm (60) Users (200) Proxy Cambridge Proxy Guilford Proxy San Diego Proxy Amsterdam Shanghai
  • 5.
    # Commit Server Edge Server Build Farm (60) Users (200) Cambridge Proxy Guilford Proxy San Diego Proxy Amsterdam Shanghai Edge Server Edge Server
  • 6.
  • 7.
    # • Monitorsall vital stats of server and proxies • Simple Python script, produces simple html • Works for Windows and Linux servers • Proxies cleaned by another Python script • Both available in the Perforce Workshop
  • 8.
  • 9.
  • 10.
    # Code Compile Test Programmer Link
  • 11.
    # Code Compile IDE and Tools Test Programmer Link Distribute Incremental linker Code Hotloading
  • 12.
    # Model Artist Test Process Hotloading
  • 13.
    # Model Tools Artist Test Process Hotloading Distribute
  • 14.
    # Programmer Rigger Tester Team Designer Animator
  • 15.
    # Programmer Rigger Animator Animator Tester Designer Modeler Programmer Programmer VFX artist Designer Audio Engineer Modeler Designer Animator
  • 16.
  • 17.
    # • Doesnot look like this
  • 18.
    # • It’smore like this
  • 19.
    # • OK,like this
  • 20.
    # • Personalbranch is fine (sandbox / streams / Git) • Feature branches sometimes work • Give control over your environment • But create distance to others • Team work requires frequent branch switching
  • 21.
    # • Forus, branch switching is always expensive: – Amount of data and change – Unmergeable files – Code-Data dependencies • Branches add complexity, we already have that • Team effort, we don’t want distance • Almost everybody works on the trunk • So why isn’t it always broken?
  • 22.
  • 23.
    # Programmer Rigger Tester Team Designer Animator
  • 24.
    # Programmer SubmitSync Rigger Tester Team Submit Submit Designer Animator Submit Submit Sync Sync Sync Sync
  • 25.
    # Programmer Test Submit Sync Rigger Tester Team Submit Submit Designer Animator Submit Submit Sync Sync Sync Sync Test Test Test Test
  • 26.
    # • Testpre submit (user) • Test post submit (builder) • Test in parallel • Test as fast as possible • Still accidents do happen User B Sync Sync Build Game Submit Sync Test Game Sync Build Tools Submit Sync Cook Data < 30 minutes User A Submit Sync Test Game … Sync Upload Cook Data Upload Test
  • 27.
    # Build machines write Label 42: change 1200 Label 41: change 1180 Label 40: change 1155 Label 39: change 1150 … Labels.xml read Safe-Sync Tool • Each game build creates a label • Stored in Labels.xml file in depot • Builds add their result to the label • Builds can add artifacts to a label • Nobody gets latest (not safe) • Everybody uses safe-sync tool
  • 28.
    Select tests Syncto get label # Labels that pass tests Labels that fail tests • Custom safe-sync tool shows labels and test results • Use it to sync to a known good label – This includes artifacts from other builds
  • 29.
  • 30.
    # • Webranch for every release (often every week) • Nobody works on the release branch • Everybody works on the trunk • Changes are merged up from main to release release main = release = change
  • 31.
    • Cherry pickingrules! • Gives you a lot of control • But also lots of confusion • And mistakes are easy • Good tools can fix this • Give flexibility • And control # (See whitepaper for details)
  • 32.
    # • Iterateas fast as you can • Don’t branch too much • Test everything • Sync should be safe • Release all the time • Control what you release Speed Control
  • 33.
    # Frank Compagner frank@guerrilla-games.com
  • 34.
    # RESOURCES PerforceWorkshop (P4 monitor, cache cleaner): https://swarm.workshop.perforce.com/files/guest/frank_compagner ReleaseManager: http://www.perforce.com/resources/presentations/user-conference-talks/ tools-for-smooth-release-cycle http://www.perforce.com/sites/default/files/guerrilla-games-whitepaper.pdf