Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Code Hosting: The Key to Autonomous, Self-Service Development
Similar to Code Hosting: The Key to Autonomous, Self-Service Development (20)
Recently uploaded
Recently uploaded (20)
Code Hosting: The Key to Autonomous, Self-Service Development
- 1. Code Hosting: The Key to Autonomous, Self-Service Development Managing the Flow of Change in Your Organization
- 2. 2© Perforce Software Inc. All Rights Reserved. • Director of Client Engineering • 7 years with Perforce Software • Clients, Integrations, and Cloud • Multiple teams, 8 countries • Many start-ups, investment banking • Lots of global consulting • Grew up in Sweden, in CA since 1997 • 4 kids, motorcycles, and kitesurfing • gostergren@perforce.com About Gunnar Östergren
- 3. 3© Perforce Software Inc. All Rights Reserved. Perforce Platform Enable connections for: • Software developers • Designers • Test engineers • DevOps engineers Increased efficiency for: • Project managers • Product managers • QA engineers • Business analysts Developer Collaboration Agile Project Management Version Control System & Repository Management DevOps Scaled Dev and DevOps for: • Software developers • Artists and designers • Project admins • DevOps engineers
- 4. 4© Perforce Software Inc. All Rights Reserved. Helix Core Before 2017 Hierarchical Depot Helix Versioning Engine • Introduced 1997 • Focused on enterprise & scale • Supported Git using GitFusion and GitSwarm
- 5. 5© Perforce Software Inc. All Rights Reserved. Introducing Helix4Git in 2017 Graph Depot Hierarchical Depot Helix Versioning Engine • Added the same data model used for managing Git to the Helix database • Now Helix Versioning Engine can manage Git data directly
- 6. 6© Perforce Software Inc. All Rights Reserved. • A comparison of “Classic” Helix vs Git What does that mean? “Classic” Helix vs Git
- 7. 7© Perforce Software Inc. All Rights Reserved. The Git Data Model (aka Graph) • Repository snapshot history • Commit Merge history • References are name markers to particular snapshots dev r1.0 r1.1 r2.0 master
- 8. 8© Perforce Software Inc. All Rights Reserved. The Hierarchical Helix Data Model • Independent file revisions • InterFile branching history
- 9. 9© Perforce Software Inc. All Rights Reserved. • New Depot type = Graph • New object = Repo • Helix Workspace can Sync: • One Repo • Subsection of a Repo • Multiple Repos • Multiple Repos and Hierarchical Helix4Git Allows Both Data Models Graph Depot Hierarchical Depot Helix Versioning Engine DevOps Engineer Build
- 10. 10© Perforce Software Inc. All Rights Reserved. Why Does This Matter? • Highly scalable CI/CD pipeline with Git can be challenging • Git clone/fetch and Git push are expensive in terms of resources • Packfile generation • Can suffer from concurrency issues • Can suffer from repository bloat • We can add unique features
- 11. 11© Perforce Software Inc. All Rights Reserved. • Clone of a Clone of a Clone • Creates cascade of git servers • Reduces load on primary git server by spreading among many • Increases complexity of infrastructure • Can result in “consistency” problems between repos • “A” depends on “B” but “B” is stale. • Doesn’t solve issues of file delivery at scale Traditional Solutions for Scaling Git
- 12. 12© Perforce Software Inc. All Rights Reserved. Highly Scalable CI/CD With Helix4Git • Use any Git management solution • Mirror Repos to Helix • Use Helix to mirror repos to remote users • User Helix Workspace for builds 3rd party Git Management Git Connector Graph Depot Helix Versioning Engine Build DevOps Engineer Developer Git Developer (UK) Git Connector
- 13. 13© Perforce Software Inc. All Rights Reserved. DESIGNER DEVELOPER DESIGNER/ DEVELOPER DESIGNER/ DEVELOPER DESIGNER/ DEVELOPER Helix Core Additions • Support for 3rd party software WAN accelerator (i.e. Aspera) • Faster submits and server- to-server replication • Benefits: • Improved experience/ productivity for remote users with Edge/Commit • Faster backup/DR replication • Faster build/CI with remote build farm
- 14. 14© Perforce Software Inc. All Rights Reserved. Why does this matter? The Helix Versioning Engine Delivers File Content Faster Than Git
- 15. 15© Perforce Software Inc. All Rights Reserved. CI @ Scale: 40-80% Faster Builds, 18% Less Storage All tests performed with shallow clone of Linux kernel on a 1 Gbps link, and four parallel threads of p4 sync. For WAN test, a 200 ms round trip latency was added between client and server. Sync Linux kernel head revision over WAN Sync Linux kernel head revision over LAN Resulting sync sizes 0 5 10 15 20 25 30 35 40 45 50 Git Clone P4 Sync 796 MB 650 MB 43 min. 13 min. 47 min. 25 min.
- 16. 16© Perforce Software Inc. All Rights Reserved. Why does this matter? Helix4Git Allows Workspaces Beyond the Scope of a Single Repo
- 17. 17© Perforce Software Inc. All Rights Reserved. CI for Android Using Helix: 18-36% Faster Builds, 25% Less Storage All tests performed with shallow clone of full Android build on a 1 Gbps link, and four parallel threads for repo sync and p4 sync. For WAN test, a 200 ms round trip latency was added between client and server Sync Android head revision over WAN (1100 repos, 200 ms latency) Sync Android head revision over LAN (1100 repos) Resulting sync sizes 0 20 40 60 80 100 120 140 160 Android Repo Sync P4 Sync 66 MB 28 min. 17 min. 150 min. 23 min. Git Clone 14 min. 36 min. 67 MB 50 MB
- 18. 18© Perforce Software Inc. All Rights Reserved. The Helix Versioning Engine can deliver file content faster than Git LFS. Why does this matter? The Helix Versioning Engine Delivers File Content Faster Than Git LFS
- 19. 19© Perforce Software Inc. All Rights Reserved. All tests performed on a 1 Gbps link, and four parallel threads of p4 sync. 30% Faster Than Git LFS Git Ifs Clone P4 SyncGit Clone Sync 50 1GB binary files over LAN 0 2 4 6 8 10 12 14 11.5 min. 10.5 min. 7.6 min.
- 20. 20© Perforce Software Inc. All Rights Reserved. Helix4Git provides automated mirroring for remote locations Why does this matter? Helix4Git Provides Automated Mirroring for Remote Locations
- 21. 21© Perforce Software Inc. All Rights Reserved. Automated Mirroring • Lightweight and automatic mirroring 3rd party Git Management Git Connector Graph Depot Helix Versioning Engine Build DevOps Engineer Developer Git Developer (UK) Git Connector
- 22. 22© Perforce Software Inc. All Rights Reserved. Git at Scale • Support large-scale, Git-based development projects with continuous build, integrate and test processes, built-in mirroring, and a single source of truth across multiple Git repos • Up to 80% faster builds • Organize multi-repo projects • Replicates content automatically around the globe • Multiple Git tool support • Integrated for streamlined release • Standalone or with Helix Core
- 23. 23© Perforce Software Inc. All Rights Reserved. Helix4Git provides automated mirroring for remote locations Why does this matter? But … Something Is Missing!
- 24. 24© Perforce Software Inc. All Rights Reserved. Helix4Git - 2017 3rd party Git Management Git Connector Graph Depot Helix Versioning Engine Build DevOps Engineer Developer Git Developer (UK) Git Connector • 3rd party solutions do not support our full capabilities
- 25. 25© Perforce Software Inc. All Rights Reserved. Helix4Git - 2018 Graph Depot Helix Versioning Engine Build DevOps Engineer Developer Git Developer (UK) Git Connector Helix TeamHub • Helix TeamHub Enterprise will move forward along with, and expand on, our core offering
- 26. 26© Perforce Software Inc. All Rights Reserved. Helix4Git provides automated mirroring for remote locations Why does this matter? Helix TeamHub Enterprise 2018
- 27. 27© Perforce Software Inc. All Rights Reserved. Helix TeamHub for Git at Scale • Flexible developer workflows • Manage build artifacts and graphics files • Accelerated DevOps build performance for CI/CD • Side-by-side diffs, commit history, code browsing, code comparison, code search, highlighting • Integrates with more than 75 of the most popular tools, including Jira, Slack, and Jenkins • Available as a cloud service and an on-premises version called Helix TeamHub Enterprise
- 28. 28© Perforce Software Inc. All Rights Reserved. Pain Points for Developers @ Scale • Project administration and the role of IT • Tracking changes across multiple repos • Team collaboration and code reviews • Continuous integration and feedback loops • Serving developer and organization needs
- 29. 29© Perforce Software Inc. All Rights Reserved. Project Administration and the Role of IT • Self-service project administration
- 30. 30© Perforce Software Inc. All Rights Reserved. Multiple Repos, Multiple Repo Types • Projects that span across multiple repositories
- 31. 31© Perforce Software Inc. All Rights Reserved. Easy Team Collaboration and Code Reviews • Simplifies the Git environment for developers
- 32. 32© Perforce Software Inc. All Rights Reserved. Continuous Integration and Feedback Loops • Accelerates DevOps build performance for CI/CD
- 33. 33© Perforce Software Inc. All Rights Reserved. Serving Developer and Organization Needs • Why Helix TeamHub?
- 34. 34© Perforce Software Inc. All Rights Reserved. Helix4Git provides automated mirroring for remote locations Why does this matter? Now Helix TeamHub Solves Developer Challenges
- 35. 35© Perforce Software Inc. All Rights Reserved. Why Developers Love Helix TeamHub • Modern UI • Delegated access management • Code reviews • Streamlined communications • Multi-repo projects • Code & build artifacts • Integrations
- 36. 36© Perforce Software Inc. All Rights Reserved. Helix4Git provides automated mirroring for remote locations Why does this matter? Helix TeamHub Features and Benefits
- 37. 37© Perforce Software Inc. All Rights Reserved. High Performance CI/CD with Helix4Git Code Browser Issue Tracking and Wiki Code Review Email Support SSO (SAML 2.0) Multi-Repo Support Repository-Level Authorization Code Search Collaborator Accounts Self-Service Project Administration HA and DR Configurations Helix TeamHub - Major Features FeaturesEnterprisePremiumStandardFree
- 38. 38© Perforce Software Inc. All Rights Reserved. Helix4Git provides automated mirroring for remote locations Why does this matter? Questions? Thank you very much for your attendance!
- 39. 39© Perforce Software Inc. All Rights Reserved. Helix4Git provides automated mirroring for remote locations Why does this matter? Thank You!
- 40. Follow us for news and insights! Visit www.perforce.com
Editor's Notes
- Title perhaps a bit misleading … but try to keep it within that frame if possible.
- Keep this short, maybe just give people a glance, it’s just to provide a quick idea of who’s talking.
- Project Management & Issue Tracking Hansoft (complex project with lots of dependencies and tight deadlines, particularly popular in the media/gaming world) HTH Issues (lightweight agile, tightly integrated with workflow) Helix ALM (Requirements focus, particularly useful for regulated efforts) Integrations (JIRA, Bugzilla, Redmine, etc) Developer Collaboration Swarm (for Classic, and Streams) Helix TeamHub (for Git, Git @ Scale) Version Control & Repository Management Helix Core (Classic, Streams, Graph Depot can be used without TeamHub) Helix TeamHub (Graph Depot)
- 20 years is a long time to work on scale and performance issues, we have used the time well It’s an honor to serve our extensive list of high profile customers include Samsung, Qualcom, Nvidia, Pixar, New York Stock Exchange, EA (and most other gami developers), etc. However, so far we haven’t had a great story for small teams. Also, our first wave of Git support was a great engineering effort and used by many clients to manage some of their challenges. But it was a bit cumbersome and didn’t scale as far as we’d like. GitFusion: A translation layer allowing users to work both in Git and Helix against the same repository. GitSwarm: A modified GitLab version with support for GitFusion. It worked fairly well, even amazingly well in some respects, added more complexity then we like. Translation is not free, obviously, and teams generally work only in one VCS format (at least per component/service/product). Thus, we are moving away from both and I’m just about to tell you what we have done and are doing instead.
- Now why would we want to do that? Let’s compare the two and get into what this can bring to the table!
- Git is very popular and we use it for some projects too to make sure we have expertise in the area. It isn’t really a version control system though, it’s more of a snapshot tracker.
- Helilx Classic Data Model provides a lot more detail about what’s going on then Git’s repository snapshots. Helix core tenant is to 1: never loose or destroy data and 2: operate quickly Some distinctive features: Work on only partial and/or multiple repositories File locking Know who’s working on what Almost crazy granular permissions Amazing auditability Massive scale (size, number of files, change rate, replication) Change identifiers are sequential, combined with above it makes it easier to manage complex products with hundreds of code lines and massive merges at the end.
- Perforce organizes data in Depots and Git, in this context, is a new type of Depot that can hold Git repositories. Many p4 commands can operate on both, even at the same time. For builds you can, for example, grab whatever you want from anywhere on the server. However, we are not going back to protocol/date translations so users will submit/push to one model at a time.
- We'll touch on some unique features later ...
- Normal Git clients and management solutions still work. Helix makes sure nothing is stale by checking and updating caches as needed.
- Helix4Git also leverages our “normal” replication and scalability features. We can cover more of this if you have interest and there's time.
- Generally, when scalability is an issue customers make sure they have the data accessible in their LAN so 80% is realistic in most real-world scenarios. But, if that isn't the case, 40% isn't bad either. 18% less data might seem unimportant to some but it really isn't when you're talking about high-end petabyte volumes.
- Perhaps skip this one?
- Perhaps skip this one?
- I've already touched on this but perhaps explain more what this means and why it is important? ...
- As an interesting data point, one of our customers who evaluated Helix4Git expects to reduce their build farm from 400 servers to under 150 when they have it fully in production.
- Anybody wants to venture a guess?
- We obviously need to do something about that, right?
- Enter Helix TeamHub ... We started this effort in 2016 as a new project and decided to accelerate it buy acquiring Deveo and integrate it with Helix Core. By doing so, our customers can leverage the full functionality of both Helix Core, Streams, and Graph Depots.
- These are some of the unique attributes of Helix TeamHub. It has features that work for the individual developer, as well as small and large teams. Acceleration of builds provides 40-80% faster builds One of our customers reported they could go from 400 build servers to 150 Integrations are easy to set up and based on normal web hooks.
- Let’s start by listing the pain points that we are trying to address. Perforce is all about scale and for us that means: @ Scale = Number of developers, number of locations, number of repos, number of assets, number of commits, number of technologies involved, and rate of change. As these numbers grow, so does the problems: - Role of IT, how does IT fit to the picture and how can they help - Tracking changes on features or releases that span across multiple repositories - Making the development workflow as smooth and automated as possible - Continuous integration and other automated feedback loops And finally, Serving the developer needs as well as the organization needs
- All the way since the PCs, followed by the Web, Clouds, SaaS, PaaS, BYOD, and in general “Software Eating The World”, IT departments are struggling to respond quickly enough. New Projects are created more often and teams, particularly distributed ones, can not wait for IT to get around to it. Self service is a requirement, not a nice-to-have feature We have customers with team/department servers all-over-the-place and it’s causing massive issues with security (hacked cloud instances, data leaks, etc), availability, DR, and performance. Self service in code hosting and HTH in particular means anyone with proper credentials can create a project and become the owner and thus the administrator of that project Earlier there were tickets created to IT to accomplish this and all of the related tasks Now that person who created the project can invite others to the project Set permissions based on roles Create necessary repositories and the project structures Set up the rest of the tooling around it Everything happens with a couple of clicks It sounds very basic, but platforms that can deliver superior performance generally tends to be hard to configure and maintain or lack in flexibility. The role of the IT organization is becoming more and more To enable self-service within the development organization and Making sure the compliance and security requirements are fulfilled Self-service actually helps in this regard as the number of “self managed” resources diminishes and the self-service platform acts as a single source of truth
- Multi-repo in single team: Early days we were building Monoliths, software consisted of a component that was stored under a single repository -> divide development into components, good example being dividing Backend and frontend where we already typically divide the code to two different repositories - starting to use libraries and frameworks -> coming to today, where micro-services architectures are more than common - biggest percentage of code comes from 3rd party components -> in the future, we see trends such as serverless emerging - even smaller components The number of code lines per component is getting smaller The number of components is getting larger - Various build tools used, various artifacts being produced by those build tools, need a way to store and version those artifacts We rely overgrowing number of external dependencies, 3rd party libraries and tools Up to today, there’s no ability to manage multiple components seamlessly Managing both code and the dependencies under one platform is something that brides the gap between the development tools If we then think about these problems in a larger context, we face the same problems across the organization: We have various projects, some of them legacy, the projects have been done in various technologies and use various version control systems Companies do more and more acquisitions and obtain the IPR, which means the source code and the dependencies What we really need is a Single source of truth for managing the sprawl that happens. Having that single source of truth allows us to Protecting IP Protecting against claims Enable discovery and reuse of software, components or even source code
- Code Review benefits: - Fewer defects in code, better design - Improved communication and sharing of best practices - Education of junior programmers and peer learning Done right, code reviews can result in Shorter feedback loops, Shorter lifetime for bugs and More maintainable code And all this boils down to, is of course, more customer satisfaction and less support calls - For code reviews, we need proper tools to both conduct and store the review information - What we need from the code review tools is: - Easy workflow for developers to conduct the reviews - And here I see that a ”pull requests” in which I mean a contribution outside of the project, happens less often outside of open source community, code reviews are typically done within the same repo, across branches, not across repositories Typical workflow is doing a feature in isolation inside a branch, there can be numerous feature branches in parallel When feature is done, create a review which will also conduct the merge between the branches after the is done In order to make this code review workflow efficient, we want to have multiple layers of control, before we actually merge In HTH we can set number of approvals, the number of “other” team members who need to explicitly approve the changes We can also require that a continuous integration tool, such as Jenkins gives a successful Build status for the feature branch We can ensure that the feedback given during the code review gets addressed using Task comments that need to be resolved And finally, we can set Default reviewers, that are automatically assigned as reviewers to new code reviews, we can think default revierwers as owners or guardians to a specific component
- Code Hosting solutions makes setting up CI with a team much simpler Earlier it took custom scripts or a lot of manual work to test that features work well together and not just “on my machine” CI is the feedback loop that we build on top of the VCS. It ties to the development workflow giving early feedback often and constantly. It also acts as a gate keeper or quality gate during the code review process Now, as the projects get bigger and bigger, the CI can become the bottle neck. Especially when we are talking about projects that span across geographical locations The problem we usually see that the clone and pull operations take longer and longer. Usually this is tackled with multi-site replication and this is a good solution on some cases. However, when the projects span both across multiple geographical locations, and across tens, hundreds or even thousands of repositories, we start to experience the problems with clone and pull performance even if we use replication. Helix TeamHub Enterprise we have solved this by creating Helix4Git Helix4Git is a basically a reimplementation of the Git backend utilizing the proprietary capabilities in Helix version engine, but The difference is, you can both manage and interact with multiple Git repositories atomically. Currently this allows checkout changes 80% faster and in some cases, reduce the number of build servers required by 75% In the future, it will allow us to do even more
- We divide the reasoning to the developers as well as to the organization around them For the developer Developer: We have always wanted to build tools that Developers love to use, after all, we are developers ourselves One of our key values has been Simplicity, which means getting things done efficiently Another is extensibility, the ability to build your own tooling around the product, we accomplish this with the 100% API coverage, which means you can do everything via APIs that you can do via the web UI Additionally, we have a lot of hooks that can trigger actions in other tools and services according to actions taken in Helix TeamHub If we think about the organization as a whole: We strive to deliver our promise of a Single source of truth, where all the assets, both code, build artifacts and other digital assets can be found from one place This ensures that the Compliance and security requirements are met and that the IPR are safe.
- Three cloud versions:. Free is for up to 5 users, and includes 1gb of storage per user. Standard is for 6+ users for $19/users/year and adds email support. Premium is $29/user/year and adds SSO, Repo and Branch-level authorizations, Code Search and Collaborator accounts. Enterprise is $179/user/year and adds: HA/DR and Higher Performance Build with H4G. Unique features backed by GD (multi repo CR, replication, …) YOU CAN SEE THE FEATURE BREAKOUT and pricing on Perforce.com
- Facebook: https://www.facebook.com/Perforce/ LinkedIn: https://www.linkedin.com/company/perforce-software?trk=top_nav_home Twitter: https://twitter.com/perforce Blog: https://www.perforce.com/blog