How many builds do you do per day? How many projects are part of each build? How many remote workers do you have? If your answer to any of the above is hundreds or thousands, then you may be aware of some of the challenges faced when attempting to provide a highly scalable CI/CD pipeline around Git. Our latest solution for Git in the enterprise, Helix4Git, enables enterprise organizations to accelerate their CI/CD pipeline by: - Reducing the time for delivering content for builds - Increases the concurrency of content delivery - Automatically mirror Git repositories for remote users

1. Git at Scale
with Helix4Git

2. 2© Perforce Software Inc. All Rights Reserved.
Introducing Helix4Git
• Helix4Git is a new
feature of 2017.1
Hierarchical Depot
Helix Versioning Engine

3. 3© Perforce Software Inc. All Rights Reserved.
Introducing Helix4Git
Hierarchical Depot
Helix Versioning Engine
• We’ve added the same data model used
for managing Git to the Helix database

4. 4© Perforce Software Inc. All Rights Reserved.
Introducing Helix4Git
Graph Depot Hierarchical Depot
Helix Versioning Engine
• The end result is that the Helix Versioning
Engine can manage Git data directly

5. 5© Perforce Software Inc. All Rights Reserved.
• A comparison of “Classic” Helix vs Git
What does that mean?
A comparison of “Classic” Helix vs Git

6. 6© Perforce Software Inc. All Rights Reserved.
The Hierarchical Helix Data Model
• Independent
file revisions
• Independent
namespaces
• InterFile
branching history

7. 7© Perforce Software Inc. All Rights Reserved.
The Git Data Model (aka Graph)
• Repository
snapshot history
• Singular namespace
• Commit Merge
history
• References are name
markers to particular
snapshots
dev
r1.0 r1.1 r2.0
master

8. 8© Perforce Software Inc. All Rights Reserved.
• New Depot type = Graph
• New object = Repo
• Helix Workspace can Sync:
• One Repo
• Subsection of a Repo
• Multiple Repos
• Multiple Repos and Hierarchical
Helix4Git allows the use of both data models
Graph Depot Hierarchical Depot
Helix Versioning Engine
DevOps Engineer
Build

9. 9© Perforce Software Inc. All Rights Reserved.
Why does this matter?
• Highly scalable CI/CD pipeline
with Git can be challenging
• Git clone/fetch and Git push are
expensive in terms of resources
• Packfile generation
• Can suffer from concurrency issues
• Can suffer from repository bloat

10. 10© Perforce Software Inc. All Rights Reserved.
• Clone of a Clone of a Clone –
Creates a cascade of git servers
• Reduces the load on a primary git server
by spreading amongst many
• This dramatically increases complexity of
infrastructure
• This can result in “consistency” problems
between repos
• “A” depends upon “B” but “B” is stale.
• Still doesn’t solve the issues of file
delivery at scale
Traditional solutions for scaling Git

11. 11© Perforce Software Inc. All Rights Reserved.
Highly-scalable CI/CD with Helix4Git
• Use any Git
Management Solution
• Mirror Repos to Helix
• Use Helix to mirror
repos to remote users
• User Helix
Workspace for builds
3rd party Git Management
Git Connector
Graph Depot
Helix Versioning Engine
Build
DevOps Engineer
Developer Git Developer (UK)
Git Connector

12. 12© Perforce Software Inc. All Rights Reserved.
Why does this matter?
The Helix Versioning Engine can
deliver file content faster than Git.

13. 13© Perforce Software Inc. All Rights Reserved.
CI @ Scale: 40-80% faster builds, 18% less storage
All tests performed with shallow clone of Linux kernel on a 1 Gbps link, and four parallel threads of p4 sync.
For WAN test, a 200 ms round trip latency was added between client and server.
Sync Linux kernel head
revision over WAN
Sync Linux kernel head
revision over LAN
Resulting sync sizes
0
5
10
15
20
25
30
35
40
45
50
Git Clone P4 Sync
796
MB
650
MB
43
min.
13
min.
47
min.
25
min.

14. 14© Perforce Software Inc. All Rights Reserved.
Why does this matter?
Helix4Git allows workspaces
beyond the scope of a single repo

15. 15© Perforce Software Inc. All Rights Reserved.
CI for Android using Helix:
18-36% faster builds, 25% less storage
All tests performed with shallow clone of full Android build on a 1 Gbps link, and four parallel threads for repo sync
and p4 sync. For WAN test, a 200 ms round trip latency was added between client and server
Sync Android head
revision over WAN (1100
repos, 200 ms latency)
Sync Android head
revision over LAN
(1100 repos)
Resulting sync sizes
0
20
40
60
80
100
120
140
160
Android Repo Sync P4 Sync
66
MB
28
min. 17
min.
150
min.
23
min.
Git Clone
14
min.
36
min.
67
MB
50
MB

16. 16© Perforce Software Inc. All Rights Reserved.
The Helix Versioning Engine can deliver file content faster than Git LFS.
Why does this matter?
The Helix Versioning Engine can
deliver file content faster than Git LFS.

17. 17© Perforce Software Inc. All Rights Reserved.
30% Faster than Git LFS
All tests performed on a 1 Gbps link, and four parallel threads of p4 sync.
Git Ifs Clone P4 SyncGit Clone
Sync 50 1GB binary
files over LAN
0
2
4
6
8
10
12
14
11.5
min. 10.5
min.
7.6
min.

18. 18© Perforce Software Inc. All Rights Reserved.
Helix4Git provides automated mirroring for remote locations
Why does this matter?
Helix4Git provides automated
mirroring for remote locations

19. 19© Perforce Software Inc. All Rights Reserved.
Automated mirroring
• Lightweight and
automatic mirroring
3rd party Git Management
Git Connector
Graph Depot
Helix Versioning Engine
Build
DevOps Engineer
Developer Git Developer (UK)
Git Connector

20. 20© Perforce Software Inc. All Rights Reserved.
Git at Scale
• Support large-scale, Git-based
development projects with continuous
build, integrate and test processes,
built-in mirroring, and a single source
of truth across multiple Git repos
• Up to 80% faster builds
• Organize multi-repo projects
• Replicates content automatically
around the globe
• Multiple Git Tool Support
• Integrated for Streamlined Release
• Standalone or with Helix Core

21. Thank You