Coordinating Change and Release Management in Games Development<br />Gordon Milligan<br />QA and Release Manager<br />Real...
Intro<br />Change control from games perspective<br />Case study discussing our initial forays into change control for a l...
Who are we?<br />Software technology company in entertainment sector<br />Based in Dundee, Scotland<br />Largest independe...
Change Control<br />“process of managing, and controlling, how a product is changed”<br />Michael E. Bays<br />
Change Control Considerations<br />Build changes <br />Test changes<br />Review changes<br />Quality<br />Risk<br />
Maintaining Quality: Stability<br />Continuous Integration<br />Daily smoke test at a minimum<br />Full regression passes<...
Maintaining Quality: Overall<br />Continuous Review<br />Designers<br />Artists<br />Producers<br />Creative directors<br />
Risk<br />Analyse risk of changes and likely impact<br />Highlight concerns to developers, producers, etc.<br />Attempt to...
Evolution of Change Processes<br />Initially we were naive<br />Lack of understanding of the issue<br />Size of team was b...
Development Team Make Up<br />~55 % Software Engineers<br />~20 % Artists<br />~20 % Level/Script Designers<br />~5% Produ...
Daily Build is Life Blood<br />...for non-programmers<br />Relied on to allow artists/designers to receive new code, desig...
Crackdown Approach <br />
Compunding Problems<br /><ul><li>Lack of due diligence on check-ins</li></ul>Long build verification turnaround<br />70 pe...
Process Changes<br />Introduced stringent check-in process<br />Could take 2-3 hours all in (or much longer on occasion)<b...
Example Check-in Process<br />Build data & code<br />Blow up car<br />Drive car <br />Kill NPC<br />Complete mission X<br ...
Process Changes<br />Introduced stringent check-in process<br />Could take 2-3 hours all in (or much longer on occasion)<b...
Reaction to Changes<br />Less check-ins so code would change underneath feature development<br />Also code/data sat on dev...
Further Process Changes<br />Restricted times when check-ins could happen<br />Created check-in queue (FIFO)<br />
Problems<br />Change submission bottleneck<br />Whose next in queue?<br />Coders not checking in regularly enough so alway...
Crackdown: Lessons Learned<br />Say NO to...<br />Large # devs submitting change to same location<br />Long build verifica...
Main thing I learned...<br />Trade Off<br />Make check-ins easy and less restrictive<br />V<br />Maintain a stable product...
Project X<br />Consists of multiple SCRUM teams working on separate areas<br />Mixture of industry veterans through to new...
Development Process Alterations<br />Introduced buddy check-in process to share knowledge/responsibility & reduce risk<br ...
Development Process Alterations<br />Mainline is no longer a development codelineand is owned by QA<br />Used to consolida...
Development Process Challenges<br />Educate developers on:<br /> Benefits of branching<br />In particular task & prototype...
Benefits of Current Setup<br />Lightweight check-in process for team branches<br />Allows quick change submission<br />Cha...
Benefits of Current Setup<br />Strict policy for mainline integration helps to ensure its quality<br />Rarely is it broken...
Hurdles crossed during changes<br />Resistance to methodology change<br />Stakeholder buy in<br />Educating developers on:...
Summary<br />Discussed types of changes & dependencies involved in game builds<br />Talked in detail about the problems of...
Tools<br />Perforce <br />CI System<br />CruiseControl.net<br />Moving to Team City<br />Jira: Feature tracking (with help...
Resources<br />Software Configuration Management Patterns<br />Berczuk and Appleton<br />Software Release Methodology <br ...
Upcoming SlideShare
Loading in …5
×

Coordinating Change And Release Management In Games Development

1,599 views

Published on

Slide deck from a talk I gave to the BCS Configuration Management group in London.

In this talk I discussed the issues of large scale software development and how you can control change in these environments. I highlight methods we have tried to improve our change control processes and where these have been a success or failure.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,599
On SlideShare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
37
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Coordinating Change And Release Management In Games Development

  1. 1. Coordinating Change and Release Management in Games Development<br />Gordon Milligan<br />QA and Release Manager<br />Realtime Worlds<br />
  2. 2. Intro<br />Change control from games perspective<br />Case study discussing our initial forays into change control for a large team<br />Discuss what we learned<br />A further case study on how we work now<br />
  3. 3. Who are we?<br />Software technology company in entertainment sector<br />Based in Dundee, Scotland<br />Largest independent games company in Scotland<br />Approximately 300 staff<br />
  4. 4.
  5. 5. Change Control<br />“process of managing, and controlling, how a product is changed”<br />Michael E. Bays<br />
  6. 6. Change Control Considerations<br />Build changes <br />Test changes<br />Review changes<br />Quality<br />Risk<br />
  7. 7. Maintaining Quality: Stability<br />Continuous Integration<br />Daily smoke test at a minimum<br />Full regression passes<br />Targeted feature testing<br />Automated Testing<br />
  8. 8. Maintaining Quality: Overall<br />Continuous Review<br />Designers<br />Artists<br />Producers<br />Creative directors<br />
  9. 9. Risk<br />Analyse risk of changes and likely impact<br />Highlight concerns to developers, producers, etc.<br />Attempt to make developers aware of the risk of their changes<br />
  10. 10. Evolution of Change Processes<br />Initially we were naive<br />Lack of understanding of the issue<br />Size of team was bigger than any of us had worked on<br />Hit many problems: no real control<br />Educated ourselves and continued to review and evolve or change processes<br />
  11. 11. Development Team Make Up<br />~55 % Software Engineers<br />~20 % Artists<br />~20 % Level/Script Designers<br />~5% Production<br />
  12. 12.
  13. 13.
  14. 14.
  15. 15. Daily Build is Life Blood<br />...for non-programmers<br />Relied on to allow artists/designers to receive new code, design & art content<br />Allows non-coders to develop their content against recent changes<br />Provides buffer against bleeding edge<br />
  16. 16. Crackdown Approach <br />
  17. 17. Compunding Problems<br /><ul><li>Lack of due diligence on check-ins</li></ul>Long build verification turnaround<br />70 people working on same codeline == continuous churn<br />Constant changes to common files<br />Productivity loss through broken builds<br />
  18. 18. Process Changes<br />Introduced stringent check-in process<br />Could take 2-3 hours all in (or much longer on occasion)<br />
  19. 19. Example Check-in Process<br />Build data & code<br />Blow up car<br />Drive car <br />Kill NPC<br />Complete mission X<br />Etc.<br />
  20. 20. Process Changes<br />Introduced stringent check-in process<br />Could take 2-3 hours all in (or much longer on occasion)<br />Analysed and improved build verification speed on build servers<br />
  21. 21. Reaction to Changes<br />Less check-ins so code would change underneath feature development<br />Also code/data sat on dev machines for longer so issues hidden<br />Friday afternoon became favoured check-in time<br />Coders loathed submitting changes<br />
  22. 22. Further Process Changes<br />Restricted times when check-ins could happen<br />Created check-in queue (FIFO)<br />
  23. 23. Problems<br />Change submission bottleneck<br />Whose next in queue?<br />Coders not checking in regularly enough so always seeing BIG CUMULATIVE changes<br />Major breakages of build and resultant loss of prodctivity<br />
  24. 24. Crackdown: Lessons Learned<br />Say NO to...<br />Large # devs submitting change to same location<br />Long build verification cycles (if possible)<br />Allowing developers to check-in code without review<br />
  25. 25. Main thing I learned...<br />Trade Off<br />Make check-ins easy and less restrictive<br />V<br />Maintain a stable product at all times<br />
  26. 26. Project X<br />Consists of multiple SCRUM teams working on separate areas<br />Mixture of industry veterans through to new graduates<br />Have a measured approach to quality<br />Similar team make-up to previous project<br />
  27. 27. Development Process Alterations<br />Introduced buddy check-in process to share knowledge/responsibility & reduce risk<br />Use unit, integration and higher level automated testing<br />In addition to black box testing<br />Code leads (& QA) meet most mornings<br />
  28. 28. Development Process Alterations<br />Mainline is no longer a development codelineand is owned by QA<br />Used to consolidate features<br />SCRUM teams each have own development codeline which they own<br />Allows bugs to be found early and code to mature<br />
  29. 29.
  30. 30. Development Process Challenges<br />Educate developers on:<br /> Benefits of branching<br />In particular task & prototype branches to reduce risk<br />How to use version control feature set<br />Encourage regular short check-ins<br />
  31. 31. Benefits of Current Setup<br />Lightweight check-in process for team branches<br />Allows quick change submission<br />Changes submitted earlier and so have time to mature and for issues to be found<br />SCRUM Masters own team branches<br />
  32. 32. Benefits of Current Setup<br />Strict policy for mainline integration helps to ensure its quality<br />Rarely is it broken<br />I have closer control of changes hitting mainline<br />More time to review these and assess risk<br />
  33. 33. Hurdles crossed during changes<br />Resistance to methodology change<br />Stakeholder buy in<br />Educating developers on:<br />Branching: when to, how to<br />Using version control system features<br />Communication<br />
  34. 34. Summary<br />Discussed types of changes & dependencies involved in game builds<br />Talked in detail about the problems of trying to over control change<br />Put forward a new strategy we have developed to control change<br />
  35. 35. Tools<br />Perforce <br />CI System<br />CruiseControl.net<br />Moving to Team City<br />Jira: Feature tracking (with help of p4 jobs)<br />
  36. 36.
  37. 37. Resources<br />Software Configuration Management Patterns<br />Berczuk and Appleton<br />Software Release Methodology <br />Michael E. Bays<br />Practical Perforce<br />Laura Wingerd<br />

×