The document discusses global software development at Electronic Arts, highlighting the complexity and scale with over 8,400 employees and various services from game development to QA. It emphasizes the use of Perforce for version control, including features like proxy and edge servers to enhance remote experiences and address challenges like network latency and risk management. Additionally, it outlines security measures under a zero trust model and the implementation of Perforce Helix threat detection to manage privacy issues and audit logs.

1. Global Software
Development
Kevin Loken
Group Technical Director, EA

2. 2
Electronic Arts
20+
Development
studios
worldwide
#1 publisher on
PlayStation® 4
and Xbox One
8,400+
employees
http://ir.ea.com

3. 3
Electronic Arts & Perforce
5,001
EA
Licenses
15+
Years
Usage
10,000+
check-ins
In a day
160
Perforce
Servers
350+
TB
Data
180
Proxy
Servers

4. Global Software
Development

5. 5
Global Software Development
 It’s not 2 guys in a garage anymore
 Hundreds of staff
• Art & Animation
• Audio
• Code
• Online services & infrastructure
• QA, Certification, Manufacturing
 Not in one geographic location anymore

6. 6
Global Software Development
 Fun is proportional to productivity
 Productivity is about making the remote experience the
same as the local experience
 Perforce offers a number of features that help:
• Proxy servers
• Edge servers
• DVCS

7. 7
Global Software Development
 Perforce Proxy server
• Easy to setup
• Essentially zero maintenance
- Cull old files on a regular basis
• Extremely robust
• Great if all you do is sync,
edit & submit
• Terrible if you diff, revert,
switch streams

8. 8
Global Software Development
 Perforce Edge server
• Complicated to setup – too complicated to show a movie 
• Needs backup, checkpoints
- There is data that is specific and unique to the edge server
• p4 submit is longer than local
• All other operations just like local

9. 9
Global Software Development
 Still limited by TCP
protocol and network
latency
 But I want it to go faster!
• Enable parallel sync
• WAN acceleration

10. 10
Global Software Development
• Art content
• Engine development
• Game development
• Online services
• Sensitive data
• Localization
• Binary data stores
• Tools & CI
 Proxy and Edge servers are great
 Assumes a single Perforce Helix server
 Lots of Perforce Helix servers for a game:

11. 11
Global Software Development
 Risk of moving to a single repository is too high
 Human error is still a factor in our work
• Bad triggers
• Bad upgrades
 Time to execute commands increases
• Checkpoints
• Restoration
 Finding a window to upgrade would be impossible
• World-wide operations, games on different cycles, never a good time

12. 12
Global Software Development
 It would be nice to share file history between servers
 Especially between game engine and game team
 Over the years we have developed a ton of scripts to do this
 But there is something new …

13. 13
Global Software Development
 Perforce Helix DVCS Technology
• All documentation points to individual usage
 Also works server-server!
• This allows us to preserve history between servers
• It simplifies integration work flows for harvesting changes
• Custom scripts are gone, replaced by DVCS features

14. 14
Perforce Helix DVCS

15. 15
Perforce Helix DVCS

16. 16
Perforce Helix DVCS

17. 17
Perforce Helix DVCS

18. 18
Perforce Helix DVCS

19. 19
Perforce Helix DVCS

20. 20
Perforce Helix DVCS

21. 21
Perforce Helix DVCS
 Enables sharing of code across servers
 First-class feature in Perforce … no more custom scripts
 Works through proxy and edge servers
 A way of mitigating risk if a huge mono-repo is not for you

22. Security

23. 23
EA is a target

24. 24
EA is a target

25. 25
Security
 EA Security model
• Old days – ‘outside the network’ - bad, ‘inside the network’ - safe
• Today – zero trust model

26. 26
Security
 Perforce security models
• Protect table
• User management
• AD Security
- Starting to directly embed AD security groups into Perforce groups
- Simplifies management – “one source of truth”
- Potential downside: Not everyone is aware that adding a user to ‘this group’,
means they get Perforce access

27. 27
Security
 Perforce Helix Threat Detection
• Relies on audit logs
• Take a lot of space!
• Intersects with our Employee Monitoring & Privacy Policy
• Here come the lawyers

28. 28
Perforce Helix Threat Detection
 Understand privacy issues of your jurisdiction
• Switzerland: No, can’t monitor employees.
• Germany: Need approval of work council
• France: Need approval of individual employees
• Canada: Need to notify employees
• US: Don’t need to notify employees, but do anyway
• Contractors: Notify contracting agency in addition to country specific rules

29. 29
Perforce Helix Threat Detection
 Handling the privacy issues
• Build a system that intercepts and cleans all audit logs
• Integrated with HR systems
• Custom rule sets that allow filtering based on local rules
• Only pass the cleaned audit logs on to Threat Detection

30. 30
Perforce Helix Threat Detection
 Proof of Concept
• 3 months of audit logs from 2 servers
• Post processing
• Identified 10 unusual or suspicious users
• Individually handle each case, most of them explainable
• Due to post-process nature some instances couldn’t be fully investigated
• Results made the investment in Threat Detection a no-brainer for us

31. 31
Perforce Helix Threat Detection
 Going live
• Near real-time processing of audit logs from all servers
• Intrusion detection vs. Incident response
• Present case to legal department to get investigation authorization
• Investigate once authorization is achieved
 Future
• Integrate with other data sources like QRadar

32. 32
Perforce Helix Threat Detection
 Results
• Proof of Concept results made investment in Threat Detection a no-brainer
for EA
• Wide ranging legal issues involved, especially for multi-national companies
• Becomes exponentially more powerful as you integrate with other data
sources

33. Thank you!
Contact:
Kevin Loken
kloken@ea.com