Scaling Source Control for the NextGeneration of Game Development     Mike Sundy            Toby Roberts   2007 Perforce U...
Overview• Current Gen vs. NextGen• Test Methodology• Test Results• Lessons Learned• Future of P4 at EA        Scaling Sour...
Challenges in Game Industry • CurrentGen vs. NextGen   • 9 GB for DVDs for CurrentGen. (Xbox, PS2)   • 30-50 GB for NextGe...
EA Redwood Shores (EARS)             P4 Installation Metrics•   11 Perforce Servers, 1200 users (90+ servers, 4,000+ users...
Problems with Windows •   2 GB per process memory limit •   Filehandle leak (crashed up to 6 times per day) •   Performanc...
Goals of Project •   Improve stability •   Improve scalability •   Solution that works for our dev environment,     infras...
Parameters of Project •   Dell and STK hardware     • Intel •   RedHat •   Windows Desktops     • 95% of user machines •  ...
Test Methodology •   Single operation vs. load •   P4 benchmarking tools     • fstst and locktest •   Homegrown scripts   ...
Configuration Overview •   Tested 21 different server configurations.     • Windows vs. Linux     • 32-bit vs. 64-bit     ...
db locktest          6          5          4Seconds          3          2          1          0              1   2   3   4...
db 8k read-write       600       500       400MBs       300       200       100         0             1    2   3   4   5  ...
rcs 64k write       900       800       700       600       500MBs       400       300       200       100         0      ...
rcs 64k read       3500       3000       2500       2000MBs       1500       1000        500          0              1    ...
1 force sync avg.         21         19         17         15         13config         11          9          7          5...
3 force syncs avg.         21         19         17         15         13config         11          9          7          ...
5 force syncs avg.         21         19         17         15         13config         11          9          7          ...
% degradation from 1 to 3 syncs         21         19         17         15         13config         11          9        ...
edit loop avg. (176 files)         21         19         17         15         13config         11          9          7  ...
Test Results Summary•   DB operations 5-10x faster on 64-bit Linux on XFS than    Windows 32-bit.•   Linux sync performanc...
dm-CommitSubmitconfig   linux-gf1_1666         win-gf1_1777         win-gf1_1666                          0   0.2   0.4   ...
avg. editconfig   linux-gf1_1666         win-gf1_1777         win-gf1_1666                          0   0.2   0.4   0.6   ...
avg. fstatconfig   linux-gf1_1666         win-gf1_1777         win-gf1_1666                          0   0.1    0.2       ...
avg. sync compute phaseconfig   linux-gf1_1666         win-gf1_1777         win-gf1_1666                          0   0.2 ...
Configuration Lessons Learned  •   OS      •   2.6 kernel was 2 – 4x faster than 2.4 in some tests.  •   RAID      •   RAI...
Performance Principles   •   Keep your db’s on their own disks   •   Proxy heavy sync users such as build farms or remote ...
Success Stories  •   New standard has had a huge impact on our local team      productivity and P4 performance across EA. ...
NextGen EA P4 Architecture  • P4 Server setup  • Distributed Development  • Proxy Build Farm (30% gain)  • Case-insensitiv...
The Future of Perforce at EA  • RAID 50 (2x faster locking than RAID 10)  • SAS SFF drives  • AD2P4  • Multi-Node Cluster ...
Thanks  • Michael Shields  • Brett Taylor  • Kenny Wu         Scaling Source Control for NextGen Game Development   29
Trivia for SchwagQuestions?       Scaling Source Control for NextGen Game Development   30
Upcoming SlideShare
Loading in …5
×

Scaling Source Control for the Next Generation of Game Development

469 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
469
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Scaling Source Control for the Next Generation of Game Development

  1. 1. Scaling Source Control for the NextGeneration of Game Development Mike Sundy Toby Roberts 2007 Perforce User’s Conference Scaling Source Control for NextGen Game Development 1
  2. 2. Overview• Current Gen vs. NextGen• Test Methodology• Test Results• Lessons Learned• Future of P4 at EA Scaling Source Control for NextGen Game Development 2
  3. 3. Challenges in Game Industry • CurrentGen vs. NextGen • 9 GB for DVDs for CurrentGen. (Xbox, PS2) • 30-50 GB for NextGen (Blu-Ray, HD-DVD – XBox 360, PS3) - 3-5x increase in storage capacity • Asset Volume Explosion • 50k clientspec grew to 500k clientspec • Build Farms • 20-30 machines grew to 100+ • Team Sizes • Outsourcing Scaling Source Control for NextGen Game Development 3
  4. 4. EA Redwood Shores (EARS) P4 Installation Metrics• 11 Perforce Servers, 1200 users (90+ servers, 4,000+ users at EA)• 7 million files on one server, 80 GB db.have• Store all source code and source data in P4• 1.5 TB on one server, 7 TB total P4 storage• Avg. client workspace – 80-120GB and 250k-500k files• 2,500 – 66,000 syncs per day• Titles include Sims, Godfather, Simpsons, and Spore. Scaling Source Control for NextGen Game Development 4
  5. 5. Problems with Windows • 2 GB per process memory limit • Filehandle leak (crashed up to 6 times per day) • Performance during large concurrent syncs • 40 GB sync went from 40 min. to 3+ hours • NTFS • Threads and troubleshooting • Patching Scaling Source Control for NextGen Game Development 5
  6. 6. Goals of Project • Improve stability • Improve scalability • Solution that works for our dev environment, infrastructure, and team cultures Scaling Source Control for NextGen Game Development 6
  7. 7. Parameters of Project • Dell and STK hardware • Intel • RedHat • Windows Desktops • 95% of user machines • Gig network Scaling Source Control for NextGen Game Development 7
  8. 8. Test Methodology • Single operation vs. load • P4 benchmarking tools • fstst and locktest • Homegrown scripts • Force sync of 39 GB data, 180k files • p4.exe on Windows Scaling Source Control for NextGen Game Development 8
  9. 9. Configuration Overview • Tested 21 different server configurations. • Windows vs. Linux • 32-bit vs. 64-bit • NTFS v. ext3 v. ReiserFS v. XFS • 10k vs. 15k drives Scaling Source Control for NextGen Game Development 9
  10. 10. db locktest 6 5 4Seconds 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 10
  11. 11. db 8k read-write 600 500 400MBs 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 11
  12. 12. rcs 64k write 900 800 700 600 500MBs 400 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 12
  13. 13. rcs 64k read 3500 3000 2500 2000MBs 1500 1000 500 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 config Scaling Source Control for NextGen Game Development 13
  14. 14. 1 force sync avg. 21 19 17 15 13config 11 9 7 5 3 1 0 5 10 15 20 25 30 35 40 45 50 minutes Scaling Source Control for NextGen Game Development 14
  15. 15. 3 force syncs avg. 21 19 17 15 13config 11 9 7 5 3 1 0 10 20 30 40 50 60 70 80 minutes Scaling Source Control for NextGen Game Development 15
  16. 16. 5 force syncs avg. 21 19 17 15 13config 11 9 7 5 3 1 0 10 20 30 40 50 60 70 80 minutes Scaling Source Control for NextGen Game Development 16
  17. 17. % degradation from 1 to 3 syncs 21 19 17 15 13config 11 9 7 5 3 1 0 0.1 0.2 0.3 0.4 0.5 0.6 percent Scaling Source Control for NextGen Game Development 17
  18. 18. edit loop avg. (176 files) 21 19 17 15 13config 11 9 7 5 3 1 0 0.2 0.4 0.6 0.8 1 1.2 seconds Scaling Source Control for NextGen Game Development 18
  19. 19. Test Results Summary• DB operations 5-10x faster on 64-bit Linux on XFS than Windows 32-bit.• Linux sync performance under load degraded only 5% instead of the 450% we saw with Windows.• XFS outperforms ext3 and even edges ReiserFS for data syncs from the repository volume. Scaling Source Control for NextGen Game Development 19
  20. 20. dm-CommitSubmitconfig linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 20
  21. 21. avg. editconfig linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 21
  22. 22. avg. fstatconfig linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.1 0.2 0.3 0.4 0.5 0.6 seconds Scaling Source Control for NextGen Game Development 22
  23. 23. avg. sync compute phaseconfig linux-gf1_1666 win-gf1_1777 win-gf1_1666 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 seconds Scaling Source Control for NextGen Game Development 23
  24. 24. Configuration Lessons Learned • OS • 2.6 kernel was 2 – 4x faster than 2.4 in some tests. • RAID • RAID 5 best for RCS due to the high number of syncs/reads. • DB benefited from RAID 1 or RAID 10. • Filesystem • XFS proved to be significantly faster than ext3 and beat Reiser by 30% in some tests. • Drive Speed • 20% less degradation with 15k drives during multiple concurrent forced syncs due to higher I/O capacity. Scaling Source Control for NextGen Game Development 24
  25. 25. Performance Principles • Keep your db’s on their own disks • Proxy heavy sync users such as build farms or remote locations • Best place to improve hardware performance is memory and disk I/O for db’s • There is a big difference in filesystem speed Scaling Source Control for NextGen Game Development 25
  26. 26. Success Stories • New standard has had a huge impact on our local team productivity and P4 performance across EA. • New platform has scaled well for 1,000 user team and could scale to larger teams. • Stability is great! • No more memory or file descriptor limits • Can identify and kill individual threads. • $5 million saved per year in productivity across EA. Scaling Source Control for NextGen Game Development 26
  27. 27. NextGen EA P4 Architecture • P4 Server setup • Distributed Development • Proxy Build Farm (30% gain) • Case-insensitivity • Disk to disk mirroring Scaling Source Control for NextGen Game Development 27
  28. 28. The Future of Perforce at EA • RAID 50 (2x faster locking than RAID 10) • SAS SFF drives • AD2P4 • Multi-Node Cluster • Perforce + binary delta copy technology • Workspace mirroring • Single View of Perforce Servers Scaling Source Control for NextGen Game Development 28
  29. 29. Thanks • Michael Shields • Brett Taylor • Kenny Wu Scaling Source Control for NextGen Game Development 29
  30. 30. Trivia for SchwagQuestions? Scaling Source Control for NextGen Game Development 30

×