Splitting a large depot can be a daunting task, and a hidden difficulty is the impact to the bug tracking system. This session will explain how Job Content Synchronization allows multiple depots to share a single link with the bug tracking system, eliminating needed changes and significantly reducing split preparation time. Adding new depots and supporting further splits is dramatically simplified.

1. Extending Perforce
Scalability Using Job
Content
Synchronization
Shannon Mann
Software Developer
Research In Motion

2. Outline
• Problem Definition
• Brief History
• Investigation
• Solution: Job Content Synchronization
• JCS Concept
• JCS Details
• Launch
• Implementation
• Benefits
• Concerns
• Wrap-up
• Conclusions
• Questions

3. Problem Definition

4. Brief History
• Early 2000 – first Perforce use
• Mid 2005 – large scale automation introduced
• Automated parallel builds
• Dramatic growth of continuous integration
• Performance issues began
• Late 2006 – performance severe
• Decision to split
• Mid 2007 – split not possible at this time
• Code inter-dependencies too high

5. Brief History
• Mid 2007 - Alternatives sought
• Monitoring introduced
• Hardware replaced - Sparc to Opteron
• OS replaced – Solaris to Linux
• Performance improved 10X
• Continued accumulated improvements by 100X
• Initial 2007 samples at 22 seconds, now .2 seconds
• Userbase activities increased by 6X
• Early 2010 - Decision to split revisited

6. Investigation
• Decision to split by duplicating db and depot data
• separate each depot’s trees with protections
• best done with explicit protections
• Reduce cost with post-split cleanup
• clientspec cleanup reduces db.have size
• Offline obliterates completes split
• leaves depot files as-is – expensive
• Depot cleanup needed – future research topic

7. Investigation
• Split Challenges
• large number of tool integrations
• multiple groups with conflicting needs
• moving product delivery dates
• Bug-tracking integration complex – show-stopper
• Bug-tracker itself integrated with many tools
• Integration complexity dramatically increased
• Alternative desperately needed
• Goal - avoid changing bug-tracker integration
• Job Content Synchronization concept saves project

8. Solution: Job Content
Synchronization

9. JCS Concept
• Middleman depot allows existing bug-tracking
integration unchanged
• JCS copies job data from middleman to production
• Perforce Support gave enthusiastic support
• Produced reference solution in two weeks

10. JCS Concept
P4DTG
Original Bug
Depot Tracker
JCS P4DTG
New P4Jobs Bug
Depot Depot Tracker
New
Depot

11. JCS Details
• Stand-alone depot has P4DTG integration with bug
-tracker
• Needs P4Broker to redirect ‘p4 jobs’
• Needs P4Auth to make redirect transparent
• Needs P4Change to make connections unique
• Needs triggers to request initial job data
• Uses P4DTG in a depot-to-depot mirror of job data

12. JCS Details
P4DTG
P4Change P4Auth P4Jobs Bug
Depot Tracker
JCS
New
Depot
P4Broker
Users

13. Launch

14. Implementation
• Tested using full test implementation
• All parts must be present
• Users must test their tool integrations
• Staged production release
• P4Broker, P4Auth and P4Change implemented first
• Split done at later date – reduces risk
• Additional depots added as needed

15. Benefits
• Greatly increased scalability
• Depot-side and bug-tracker side connections increase
linearly and independently
• Near-trivial low-impact scalability
• Non-split depots benefit – bug-tracking connect for free

16. Concerns
• Significant increase in deployed complexity
• Expert knowledge of subsystems now required
• Complexity extends investigations and corrections

17. Concerns
Original Bug
Depot Tracker
Users Original Setup

18. Concerns
P4Change P4Auth P4Jobs Bug
Depot Tracker
New
Depot
Users One Depot w/JCS

19. Concerns
P4Change P4Auth P4Jobs Bug
Depot Tracker
New New
Depot Depot
Users Two Depots w/JCS

20. Wrap-up

21. Conclusions
• JCS allows near-linear scaling of bug-tracking
interconnects to multiple depots
• JCS allows a reduction in performance impact
• JCS eliminates split impact to bug-tracking system
integrations

22. Conclusions (cont.)
• JCS also allows multiple non-split depots to share bug
-tracking integration
• JCS can be used just for bug-tracking integration sharing
• Splits aren’t required to consider using
• P4Auth, P4Change and P4Broker can be used alone
• Perforce Support are your friends. Seek their advice

23. Questions?