[AMD] Novel Use of Perforce for Software Auto-updates and File Transfer
MERGE 2013 THE PERFORCE CONFERENCE SAN FRANCISCO • APRIL 24−26AbstractUsers can leverage the attributes of Perforce to createa mechanism that allows seamless file transfer amongthe multiple OSes that Perforce supports, without theoverhead of setting up different permissions andprotocols that may be proprietary to each OS. Such asolution can be written in a single codebase, effectivelyreducing maintenance overhead. The file-transfermechanism can then be extended to support auto-updating client software, so users always will beguaranteed that the locally installed applications will bethe latest version executed at run time.Novel Use of Perforce for SoftwareAuto-Updates and File Transfer ina Multi-OS EnvironmentXavier Galvez, AMD
2 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment This is text for annotations in footer. Similar to footnotes treatment.Background and MotivationThe tools team in the graphics division at AMD created an in-house, end-to-end solution forpre-submit developer builds (PSDB). This PSDB mechanism:1) Takes a code change before it is submitted into version control,2) Overlays the changes on the last-known-good version of a successful build,3) Compiles these using an accelerated build farm to ensure compilation does not break, and4) Deploys tests on the resulting build to ensure the changes are safe.Once the changes pass the tests, the developer can submit the changes into version control.However, two problems arise from Step 1:Problem #1: How will the files be uploaded to the servers that run the mechanism?Developers at AMD work on Windows®and Linux operating systems. Different permissions andprivileges have to be set up for file transfer between these systems. Ideally, if a software clientperforming the file transfer is installed locally on the user’s computer, then the source code forthis client should be compilable for all OSes (i.e., a single codebase should run on multipleOSes) for ease of maintenance.Also, development teams work with different version-control systems; some teams submitcode into Perforce while others use Subversion (SVN). The client should work with anyversion-control system.Problem #2: Given that a client will be installed on the developer’s computer to uploadthe files, how can we ensure that the developer is running the latest version of theclient?Software is a living document that gets updated frequently, especially when bugs are foundand features are added. Client updates can be deployed proactively (i.e., pushed) to ensuredevelopers use the latest version.File TransferA crucial component of PSDB is a way to upload the code changes to the solution withoutsubmitting them into revision control. This component must be both OS- and version-controlsystem-agnostic. AMD developers work in Windows and Linux OSes, and with Perforce depotsand SVN repositories for revision control.First AttemptTo address these challenges, the first generation of the file-transfer mechanism had twoseparate codebases (one codebase determined modified files in Perforce, and another inSVN). The first attempt at writing this component used a remote file server with a shareddirectory. However, file transfer was done differently for each OS.In Windows, file transfer was done using Robocopy, a third-party tool used for synchronizingfiles in two different locations. After the modified files were listed, this tool took the modifiedfiles from the user’s local computer and copied them to a shared directory on the remote fileserver. With the file server hosted on a Windows backend, the permission setup wasstraightforward.This was not the case in Linux. The Robocopy tool was not available for this OS, prompting adifferent process. A Linux user would go through the additional preliminary steps of mounting
3 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OSEnvironment This is text for annotations in footer. Similar to footnotes treatment.the remote directory as a superuser and ensuring the mount happened on login. Additionalpackages needed to be installed (e.g., application wmctrl was needed for manipulatingwindows management). Permissions also had to be set up correctly. To simplify the setup, ascript and installation recipe manual was written for Linux users. However, support callsincreased due to the complexity of the instructions, affecting quality of service.The first generation of the file-transfer mechanism exposed the need for a simple workflow toprovide a better user experience: for instance, simple installation and a one-click upload. Astatus bar could provide visual validation to display progress. Also, to reduce developmenttime, maintenance, and support, the succeeding version could be written in a single codebaseand employ the same mechanics for performing file transfer on different OSes.Enter Perforce.Current VersionThe idea behind using Perforce as a back-end solution is to set up an intermediary “pre-submit” Perforce depot that would house the files that users have modified. Once the files havebeen “uploaded” to this Perforce depot, the PSDB solution can then “download” the files fromthe depot and overlay them on the last-known-good version of a successful build. In Perforceparlance, the modified files will be submitted to the pre-submit Perforce depot, andsynchronized by the PSDB for retrieval.No changes are submitted to the main Perforce depot or SVN repository—all writeable actionsare done on the intermediary pre-submit Perforce depot. The intermediary depot exists only toperform the file transfer. Use of this intermediary depot must be transparent to the user, as ifthe depot didn’t exist.Using PerforceThe following steps describe how AMD used Perforce to create the second generation of thefile-transfer mechanism.1) Create a new workspace (clientspec) on the pre-submit depot.A developer using Perforce as a version-control system already has an activeworkspace for viewing and editing files. A new workspace is created on the pre-submitdepot with a View that is an exact duplicate of this active workspace’s View on the maindepot. Mimicking the active workspace guarantees that the files will be checked into thecorrect relative locations in the pre-submit depot, no matter how convoluted the View is.SVN does not adhere to the notion of a workspace, so files in a directory appear “as is”in the repository (i.e., the new workspace View is wide, as in “//depot/…”). The dangerof a convoluted workspace is non-existent, and modified files in SVN can be uploadedto the pre-submit Perforce depot as-is.When creating the workspace on the pre-submit depot, ensure that the allwrite option isenabled (i.e., the noallwrite option is disabled). Because this workspace is used tosubmit the modified files into the pre-submit depot, these files should remain untouchedon the local computer from the main depot’s perspective. Enabling the allwrite option iscrucial because this ensures that the files are not switched to read-only after beingsubmitted to the pre-submit depot.
4 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment This is text for annotations in footer. Similar to footnotes treatment.2) Create a pending changelist on the pre-submit depot using the new workspace;populate this with the modified files.This pending changelist will contain the modified files. The output of a p4 describe onthe description of the changelist on the main depot can be parsed to enumerate all filesthat have been modified. These modified files on the main depot are then added to thispending changelist, which will then be submitted to the pre-submit depot.Modified files in SVN can be parsed from svn info. Other version-control systems wouldhave a similar command for listing modified files.Files that have been deleted or removed (due to a rename or integrate/move) cannot beadded to this pending changelist. Instead, the description of the pending changelist willenumerate these files. The PSDB mechanism will then remove these files from its copyof the last-known-good in the next step, thereby accurately mimicking the modified stateon the user’s local computer.3) Submit the pending changelist and trigger the pre-submit build.Once the pending changelist has been submitted, the PSDB mechanism takes over.PSDB takes a copy of the last-known-good version of the latest successful build,overlays the added/edited files, removes the deleted/moved files as described in theprevious step, and initiates the accelerated build.BenefitsBy using Perforce as the main mechanism for transferring files from the user’s computer to thePSDB server, a single codebase can be written that treats the Perforce command-line calls asan API. With the codebase written in Perl, a write-once/run-on-all-OSes model is achieved.The only per-OS task involved is packaging the Perl script into an executable for each OS.Having Perforce as the back end means that the changelist submitted into the pre-submitdepot and its accompanying changelist description can be used as a rudimentary database forkeeping track of metadata and storage of modified files. Having a file server and separate SQLdatabase is not necessary.AMD leveraged a Perforce installation that already existed at the company by simply creating anew depot. We take advantage of the robustness inherent with Perforce, and merely extend itsapplication by piggybacking on an existing framework.Another benefit is eliminating race conditions; when accessing the back end for uploading files,each transaction—the Perforce submit—is atomic. Unlike FTP and rsync, this also makeshandling error conditions in the script simpler and more reliable than using an alternate datastore.ShortcomingsBy design, uploading modified files into the PSDB system can be done only through anumbered pending changelist. This can be modified to use a default pending changelist (i.e.,unnumbered), but numbered changelists force users to adhere to best practices.The pre-submit depot can be filled quickly with PSDB requests. On a normal file server,cleanups can be done easily by deleting older files. When using Perforce as an intermediarydepot, a p4 obliterate would be necessary to conserve space on the server. Maintenanceoverhead can be reduced by having the obliterate command called when the server is not busy
5 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OSEnvironment This is text for annotations in footer. Similar to footnotes treatment.and automated on a schedule.Doing a PSDB request on a virtual integration is not possible because the modified files willexist only on the main server, and not locally on the user’s computer. If a PSDB request isnecessary in this scenario, users are asked to perform the integration locally. (The actualsubmission, if the PSDB tests pass, can be performed virtually.)Given this, using a version control system as a file storage solution is overkill becausePerforce is used as an intelligent file server. However, the need to maintain a separatedatabase and dumb file server is removed, making this solution practical and efficient.To implement this solution, an executable client that initiates this mechanism (i.e., the fileupload action) must be installed on the user’s local computer. Ensuring that the user has thelatest version of this client may be an issue. This is addressed in the next section.Self Auto-UpdateSoftware is a living document that gets updated frequently as bugs are fixed and featuresadded. Ideally, all users would be running the same latest version to have the best experience;users benefit by having access to the newest features and fixes, and this also reducesmaintenance work because developers do not spend time debugging legacy versions.However, deployment may be a necessary overhead because this requires the developer toproperly package the software (e.g., create the installer), make this available to users (i.e.,announce and publish), and enforce (i.e., nag users) to ensure that users install the latestversion.Web apps are not susceptible to this shortcoming. However, web apps have limitations if adesired function can be executed only as a binary running on a local user’s computer (such asuploading multiple files in the background — modern web browsers do not allow this becauseit presents a security hole). In the case of the file-upload client, creating this client as a webapp was not an option.The rationale behind a self auto-updating mechanism is to provide a seamless experience inwhich the user is not required to perform any manual actions and is assured that the latestversion of the binary always is executed at run time.ComponentsThe self auto-updating mechanism consists of three entities: (1) a centralized version controlsystem, (2) the “caller” program installed on the user’s local computer, and (3) the “client” filesthat are updated, also installed on the user’s local computer.The centralized version control system is a file repository that tracks users’ files and theirrevisions. Typically, the file repository is used for revision control: users check out files forediting and check in files with the desired modifications. The file repository keeps track ofwhich files the users have on their computers, and at which revision. The file repository runson a server accessible to users running different OSes across the network. In our case, weleveraged an existing Perforce implementation.The caller is one part of the software application installed by the user manually. The caller isnot automatically updated. The caller consists of a Perforce command-line client and anexecutable that calls the Perforce client with the necessary commands to perform the auto-
6 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment This is text for annotations in footer. Similar to footnotes treatment.update.The client files make up the rest of the software application installed by the user. These filesare the core of the installed software and handle the file upload functionality. This portion of thesoftware application can be updated on the user’s local computer when necessary.WorkflowThis section describes the workflow that Figure 1 illustrates.1) Initially, the user installs the deployed software package from the author. On the localuser’s computer, the caller and client files are installed. When the user executes thesoftware application, the caller executable is first run in the background.2) In the background, unknown to the user, the caller executable connects to Perforce(using the Perforce command-line client) and creates a workspace for the user.3) With this workspace, the caller synchronizes the client files from the Perforce server.This action effectively overwrites the user’s local copy, thus assuring that the user’slocal copy is the latest version. When the sync is complete, the caller runs the client.The client is essentially the core of the software, and is the portion that is updated andexecuted.Every time the user runs the software, the caller synchronizes with the Perforce server.If all files are up to date, no action is necessary and the installed application (i.e., theclient) runs as usual.4) When a new revision of the software is ready, the author checks in the modified clientfiles to the file repository. No further action is required from the author.5) If the user runs the installed software after the update, the caller connects to Perforceand detects that the client files on the user’s local computer are not up to date throughp4 sync.6) The same p4 sync call grabs the latest version of the client files. After the sync, theupdated version of the client files—now the latest—is executed.
7 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OSEnvironment This is text for annotations in footer. Similar to footnotes treatment.Figure 1: The workflow for the self auto-updating mechanismBenefitsThis method leverages the existing Perforce set up. A separate database or file server is notnecessary.Whenever users initiate the file upload from their local computers (regardless of using Perforceor SVN as their main repository), the caller ensures that the latest client is alwayssynchronized locally and executed. This happens entirely in the background, silently, andusers are not required to perform any explicit actions to update their local copy of the client.The version number is logged on the submit changelist in the intermediary Perforce depoteach time the file upload is performed. Auditing these version numbers can verify that the auto-update mechanism is working and that the user is running the latest version.To verify initially that the auto-update mechanism is working, the package used for the firstinstallation contains a client that is one version behind. When the user runs the installer for thefirst time, the auto-update mechanism engages and retrieves the latest version from the
8 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment This is text for annotations in footer. Similar to footnotes treatment.Perforce server. The version number is then logged and inspected. If the version number is notthe latest, contact is initiated with the user to debug the issue. This proactive measureimproves the quality of service provided to users of the application.ShortcomingsThe full package initially installed by the user is not updated automatically; only the client filesare updated silently. This is sufficient for deploying features and bug fixes that affect file-upload functionality.If a full application upgrade is required (i.e., the caller needs to be updated), then the originalmanner of deployment (i.e., packaging/publishing/enforcing) is pursued. However, this isperformed rarely because the caller portion of the software application has matured.The caller currently consists of a separate Perforce command-line client and the executablethat calls the Perforce client. The executable calls the Perforce commands through the shell.The executable can be rewritten to take advantage of existing Perforce APIs insteadWalk-throughThe caller and client scripts are written in Perl and converted into executable binaries usingActivePerl for Windows (32-bit) and Linux (Ubuntu 32-bit and 64-bit). The Windows 32-bitexecutable can be used on 64-bit versions of Windows. These binaries are then packagedusing InstallShield (for Windows) or tarballed (Linux) and uploaded to a server, ready fordownload.The user downloads the package and runs the installer. The installer takes care of adding thecaller to the list of Custom Tools (see Figure 2). An install script written for Linux usersperforms this task.Figure 2: The caller added to the list of Custom Tools in P4VThe file-upload mechanism can be initiated by right-clicking a numbered pending changelist inP4V for Perforce users (see Figure 3). In SVN, right-clicking a folder in Windows Explorerpresents this option. The file-upload mechanism also can be initiated from the command line.
9 Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OSEnvironment This is text for annotations in footer. Similar to footnotes treatment.Figure 3: Calling the file uploader mechanism from P4VThe caller validates all parameters and synchronizes the client. Figure 4 shows the client as upto date, ensuring that the user is running the latest version. The caller then runs the client toperform the file upload.Figure 4: The log shows the caller running p4 sync; the client is at the latest versionThe client parses the pending changelist using p4 describe and lists the files. A workspacemimicking the user’s local workspace is created on the intermediary Perforce depot, and apending changelist is created. Added/modified files are included in this pending changelist.Deleted/renamed/moved files are noted in the changelist description and will be removed fromthe PSDB copy. Figure 5 shows these steps.
10Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OS Environment This is text for annotations in footer. Similar to footnotes treatment.Figure 5: The log shows the client preparing the files for uploadThe pending changelist is then submitted to the intermediary Perforce depot. Once this hasbeen verified, a SOAP call is made to the PSDB mechanism to initiate the build. A webbrowser is launched to display the progress of the build and the eventual test results. Figure 6presents these steps.Figure 6: the log shows the client uploading the files and making the SOAP call to initiate the build
11Novel Use of Perforce for Software Auto-Updates and File Transfer in a Multi-OSEnvironment This is text for annotations in footer. Similar to footnotes treatment.For simplicity, the user sees only a progress bar and status to demonstrate the stages of thefile upload during this time (see Figure 7). The user has the option to view the log if desired;the log also appears when an error occurs. The user can then send the log to the PSDB teamfor debugging.Figure 7: Progress bars provide visual validationConclusionThis white paper describes an unintended use of Perforce to perform file transfers byleveraging Perforce as an intelligent file storage server. Although this may be overkill for aversion-control system and other file transfer protocols such as FTP or SCP could have beenused, this novel method removes the overhead of setting up and maintaining a separatedatabase to manage the files that have to be transferred.By taking advantage of Perforce’s robustness and its consistency of command calls beingsupported on different operating systems, the described method can be applied without havingto maintain different codebases and set-up procedures for each OS.The file transfer method is then extended into a self auto-updating mechanism to ensure thatusers have the latest version of software installed on their local computers.