[SAP] Perforce Administrative Self Services at SAP
MERGE 2013 THE PERFORCE CONFERENCE SAN FRANCISCO • APRIL 24−26 Perforce AdministrativeSelf- Services at SAPWolfram Kramer, SAP AGPerforce White PaperSAP changed the processes for user and groupadministration and for project management step bystep from a purely centralistic approach to a muchleaner one.
2 Perforce Administrative Self-Services at SAPBackgroundSAP AG has used Perforce as its version control system for almost 15 years. The use ofPerforce has grown over time; currently, SAP hosts one of the largest Perforce serverinstallations in the world in terms of the number of licensed users: 10,000. In terms of thenumber of productively used Perforce server instances, it is the largest one, with approximately100. Many of the developers accessing Perforce use it to work with more than one instance.Perforce implements a per-server user and group management for which instance-specificprotection tables are defined. It is evident that on the one hand, a synchronization of users,groups, and also protection tables is necessary to ensure that any user gets access to all theservers where he or she expects it. On the other hand, it is also clear that such asynchronization of user and group data cannot be done manually on all the servers.For this reason, early in the 21st century SAP implemented a central user and group storagebased on a relational database. The user and group data for each Perforce server instance aresynchronized from there. Manual user and group administration directly on a Perforce instance(via p4 user or p4 group) was not done any more. The leading paradigm for theimplementation of the central user and group storage was a centralistic approach of usermanagement and any user and group management activities were done by a central team.Tightly coupled with the scheme of permission groups and protections is the centralizedgoverned depot structure on any Perforce server at SAP. Versioned files are organized in so-called projects as folders on the second level on the Perforce server directly below the depots.The creation of projects and their code lines (organized in the third level below) is strictlystandardized and can only be done through a central infrastructure. Perforce triggers prohibitadding new files outside an already existing project and/or code line. The design of the projectand code line creation process followed the paradigm of central administration by a centralteam.Although central teams are good at providing services in a workflow-like manner forimplementing and empowerment of governance, they are a hindrance in the context of leanand agile principles because they are bottlenecks for the flow. They produce waiting time forthe requestors (project managers or developers) because the workflow still needs humaninteraction by the central team. This white paper describes how SAP changed the processesfor user and group administration and for project management step by step from a purelycentralistic approach to a much leaner one with—in the ideal case—no more waiting time forconsumers.Perforce Server Landscape and Centralized User StorageThe Perforce server landscape at SAP consists of:• Approximately 100 Perforce server instances• A relational database named “P4SAP” containing central metadata such as user, group,and project data• The P4 Management System P4MS as a framework of tools and services for datamanipulation on the P4SAP database
3 Perforce Administrative Self-‐Services at SAP • Tools and services for data synchronization from the P4SAP database to Perforceservers, also integrated into P4MS• Self-services and web services around P4MS to enable user interactionFigure 1 illustrates the architecture:Figure 1: Perforce server and user storage at SAPUsers and groups as parts of the metadata are stored and maintained on the P4SAP databasevia the P4 Management System (P4MS). Each group that is stored in the database has aspecific validity scope of Perforce servers to which it is synchronized. This is because thesources of certain larger products are distributed over more than one Perforce server. Thedefinitions of groups affecting accesses to the code base therefore have to be the same oneach Perforce server. This is ensured by the synchronization of the group from the P4SAPdatabase to all Perforce servers within the validity scope of the group.Synchronization of user data to Perforce servers happens together with group synchronization.The protection scheme at SAP assumes that any user gets permission through a group inwhich the user is a member. As soon as the first group that contains the user as a member issynchronized to a dedicated server, the user data is synchronized as well. Thissynchronization ensures that any member of a group that exists on one Perforce server is alsoa valid user defined on the Perforce server. From a technical point of view, P4MS uses p4 user–i to create a user on the relevant Perforce servers.Permission groups control the access to files within Perforce depots. At SAP, a standardPerforce depot is structured along projects. A project is a location in the Perforce depot treethat hosts the code for a subsystem of an SAP application. Because subsystems can havetheir own release cycles, the code line level is below the project level, which ends up in aPerforce depot structure (see Error! Reference source not found.).RubyDele
4 Perforce Administrative Self-Services at SAPPerforce creates folders as pure representations of the full path of versioned files, appearingwhenever a file having a new path is submitted. To prohibit wild creation of folder structures atSAP’s Perforce servers, pre-submit triggers prevent the creation of new project and code linefolders. Creation of new projects is therefore also a task for the central Perforce team thatoperates on request by the development teams. It has permission to circumvent the pre-submittrigger for the creation of a first, empty file in the path of the new project. The purpose of thisfirst “createdir.txt” file is to make the path visible in the Perforce navigation tree. By thisoperation, the pre-submit trigger is deactivated for future submits on the project path.Currently, there are more than 9,000 projects; each has a lot of code lines. New code lines of aproject are usually needed for release branching. A service offered in the P4MS frameworkautomatically creates the needed code line as a folder in Perforce. Nevertheless, the creationof new projects is still a manual, central activity.Figure 2: At SAP, a standard Perforce depot is structured along projectsDesign of Perforce Access Groups and PermissionsIn the past, development teams followed a lot of different processes that slightly differed inrequirements concerning code line patterns, their naming, and their permission groups. Butone pattern applied to almost all variants contained two basic groups:• A dev group that has permissions for writing to the dev code line where all new changesare created; developers are typically members of this group.• A patcher group that has permissions for writing on the qual code line that contains thequalified, stable code; permissions to this code line usually have build operators andquality managers.
5 Perforce Administrative Self-‐Services at SAP //product_Xproject_Adevqualdev groupproject_A-‐devpatcher groupproject_A-‐patcherwritereadwritewriteFigure 3: Permissions are simplified for project admins at SAPAlthough it is possible to apply this pattern on a per-project level, in most cases, this approachis too finely granular. In general, a product consists of several projects that are contained inone depot. The same people are working on any of the projects within the depot. Therefore,the group pattern is often applied on a depot level: instead of groups like project_A-dev andproject B-dev, product_X-dev is used. This simplification means the project admin has tomaintain dev and patcher groups only once and not for each project separately (see Figure 3.)Also the central Perforce team that creates new projects does not have to create new groupsor make new entries into the protection table. The latter are covered by entries using wildcardssuch as:write group product_X-dev * //product_X/*/dev/...read group product_X-dev * //product_X/*/qual/...write group product_X-patcher * //product_X/*/dev/...write group product_X-patcher * //product_X/*/qual/...Centralized User ManagementShortly after the introduction of Perforce in 2001, SAP began to distribute projects to more thanone Perforce server instance. After first developing intermediate solutions using script-basedcopy of users from one server to another one, the central user and group storage describedabove was introduced (see Figure 4). After that, it was possible for the central Perforceoperating team to use a web front end to maintain users and groups in the central database.Groups were assigned to one or more Perforce instances. Publishing users and groups to thePerforce server was done instantaneously by pressing the save button.RubyDele
6 Perforce Administrative Self-Services at SAPFigure 4: A web front end is used to maintain users and groups in the central databaseIt was a great step forward because per-instance maintenance of groups was no longerneeded. But still all requirements from developer teams were handled by processes based onworkflows. Typical scenarios handled by the central operating team were:• Adding a user to a permission group to grant read or write access to a project; keep inmind that the admin groups already mentioned did not yet exist.• Creation of a new project including its permission groups.There were of course tools that made the creation of a project as comfortable as possible forthe operation team; creation of the project and the groups was possible in one step. But thestrict centralization of any activity concerning user and project management turned out to be abottleneck: it was not scalable with the increasing number of projects. Instead, it createdwaiting time for the developer teams and it ate up the time of the central Perforce operatingteam, preventing it from investing in improvements to the infrastructure.Group Member Self-AdministrationThe consequence was the insight that user management had to be decentralized. In fact, thedecision whether permissions on a project shall be granted to a specific user was not done bythe central team but by the responsible development manager of the project. This was whenthe admin group for a project was introduced (see Figure 5).The admin group has the purpose to modify the dev group and patcher group for a project andfor itself. It exists only inside P4MS and is in general not synchronized to Perforce servers atall. Project managers are usually members of this group since they decide who should join asdeveloper or build operator of the project.
7 Perforce Administrative Self-‐Services at SAP //product_Xproject_Adevqualdev groupproject_A-‐devpatcher groupproject_A-‐patcheradmin groupproject_a-‐adminchangechangechangewritereadwritewriteFigure 5 An admin group was established to decentralize user managementP4MS has an own authorization concept that is able to assign P4MS-application specificpermissions to groups. Members of the admin groups of a project can modify the dev andpatcher groups in P4MS and thus in the P4SAP database. The synchronization mechanismdescribed above does the job to make these groups productive on the dedicated Perforceservers.By what has been described, the use case “I as a project manager would like to define thegroup members who have access to my project” is covered. For the use case “I as a developerwould like to get access for a dedicated project,” the mechanism is not sufficient. Theassumption is unfortunately wrong that the developer could just write an email to his or herproject manager to add him or her into the project dev group. Often the developer does notknow who the project manager is.Figure 6: Any employee is able to request access to a project in Perforce at SAPTo cover the second use case, P4MS offers a service by which any employee is able torequest access to a project in Perforce. This can be done in two ways: either the user knowsthe name of the access group of the project he/she wants to be join or—and this happens in
8 Perforce Administrative Self-Services at SAPmost cases because most people are not familiar with the group names—the user provides thePerforce path of the project he or she want to get access to. In the latter case, the system findsout by itself which project the path points to and what the name of the dev group is (see Figure6).Then the system determines the admin group that is able to maintain the dev group, and anemail is sent to members of the admin group. Via the link that is included in this email, any ofthe addressed administrators is able to open the maintenance dialog for the dev group therequesting user wants to join. After having pressed the button for approval, the user is addedto the group within the P4SAP database. The synchronization of the group to the affectedPerforce servers is scheduled and takes place within a few minutes asynchronously.What makes this dialog so powerful is that a new Perforce user is created automatically on anyaffected Perforce server where the user did not exist before. In this case, a strong randompassword is created, and the user receives a generated email to set a password of his or herchoice. Hence the outlined workflow applies to already existing Perforce users who get accessto an additional project as well as for people who did not have a Perforce user before.At this point, it is worthwhile to outline the procedure of setting passwords on SAP’s Perforceservers. As already mentioned, many of the users have to connect to more than one Perforceserver. Letting them maintain and synchronize their passwords by themselves by applying thep4 passwd command on each server would create much frustration. Therefore, the passwordsetting and distribution occur via a central service in P4MS that sets the passwordautomatically on each Perforce server on which the user is synchronized. For security reasons,the passwords are not stored (not even encrypted) in the P4SAP database. Hence wheneverusers get access to an additional server, they have to go through this service once again (butwith the advantage that, afterward, the password is the same on all Perforce servers).Limited Local Perforce Project ControlGroup administration has been successfully decentralized, but the creation of new projects andcode lines until recently was completely a central responsibility. But agility for the developmentteams also means being able to create very quickly the prerequisites to start a new prototypewithout any unnecessary loss of time. A project on a Perforce server belongs to the most basicinfrastructural pieces that enable developers to do their work. A central workflow to create aproject with all its waiting time and involvement of central people that does not directly addvalue to the product does not fit any more. Therefore, SAP introduced a self-service for projectcreation in a dedicated area where speed is crucial: projects hosting code for mobileapplications.RubyDele
9 Perforce Administrative Self-‐Services at SAP Figure 7: Self-service project creation at SAPThe solution enables project managers to create a project by themselves without having torequest project creation from a central team. The self-service asks the project manager for aproject name; provided that the name is not yet in use, the project is created (i.e., thecorresponding path on the Perforce server is created) (see Figure 7).So far, this self-service is only able to create the project in one single depot on one singleserver hosting all SAP mobile applications. Also the permission concept is so far ratherpragmatically solved: for all the projects within this depot, there is only one dev group, onepatcher group, and one admin group. As long as no special, more granular permissions on aper-project level are needed, this is sufficient. Otherwise, in rare cases, interaction by thecentral operating team is still required.The Future: Full Local Perforce Project ControlFor the future, SAP plans to extend this offering to any kind of projects in any depot and onany Perforce server such that any project manager at SAP could get rid of lengthy workflowsfor project creation.But this will not be possible without solving a structural problem: the right choice of the hostingdepot and Perforce instance for a completely new project still needs some central governance.Often the creation of a completely new depot is required; in the future, this will be a task ofcentral administration. But luckily, that only happens when completely new products areinvented. The much more frequent use case for a project manager is: “I need a new project inthe same depot where all my other projects are already located.” Specifying an alreadyexisting project in a dialog or choosing it from a drop-down box is then an easy task for theproject manager and a feasible approach of a generic self-service for project creation.
10Perforce Administrative Self-Services at SAPAnother challenge comes with the creation of protections on the Perforce server.Unfortunately, there is no common pattern for code lines and groups that have read or writeaccess to them that fits for all development teams. Much worse, development teams need theability to create new code lines (e.g., for prototyping that does not follow any name pattern). Toavoid involving central manual interaction again, the protections on the Perforce servers haveto be adapted as well: the self-service must be able to add lines into the protection table togrant permissions on a code line defined by the project administrator.