Merge2013 mwarren(pv5)


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Merge2013 mwarren(pv5)

  1. 1. MERGE 2013THE PERFORCE CONFERENCESAN FRANCISCO APRIL 24 26Perforce White PaperTo provide a solid foundation for software developmentexcellence in today’s demanding economy, it’s criticalto have a software version management solution thatcan meet your demands.Extracting depot paths into newinstances of their ownMark Warren
  2. 2. 2 DEPOT SPLITTINGINTRODUCTIONAs instances are used over time they naturally grow in file and metadata size. New files aresubmitted and metadata increases in size and the instance become unwieldy. At some pointnormal operations take long enough table locks that affect all users. To help mitigate this wecan increase hardware performance but there is a hard limit to what hardware you can replace.A more practical method to release the building metadata pressure to move select datasets totheir own instance/depot.Perfsplit 1is a tooldeveloped by perforce that extracts a section of a depot from an existingPerforce database. It will access a Perforce Servers database directly, and is platform andrelease specific. Perfsplit does not remove data from the source Perforce server but does copyarchive files from it. Perfsplit is a good tool for this operation but does not resolve someproblems;The need for zero downtime. Most instances that are in need of splitting have a verylarge user-base. The need to keep instances up and running are compounded by thenumber of users unable to access their instance.Perfsplit does not rename the new instance depot. This is undesirable since it can beconfusing to users having the same depot name across multiple instances.The need to use „p4 snap‟ to copy lazy integrated files to their physical lbr location.p4snapcan considerably increase the size of the original depot depending on the sizearea we are splitting off.This document is intended to give guidelines on a method to resolve all these issues.PREPARATIONTo make sure we gather a complete dataset for migration from a live instance, it‟s necessary toprevent users from making changes to the path(s) we are splitting. With super access rights,this can be done by simply restricting ReadWrite access to this path and only allowingReadOnly. This is to ensure the metadata structure we are splitting off will be up-to-date. Oncethis is done we need to create a checkpoint of the instanceto gather lbr records and we needto have a running instance of this checkpoint for perfsplit use.Despite the inadequacies perfsplit has, this process makes use of it;Perfsplit is necessary tobuild the foundation of the new instance. The key function of perfsplit is using a map file todirect it to the select path(s) to extract. Since we are splitting not only the initial path(s) but alsothe integration history we will need to append this dataset to the splitmap. To get this, we needto grep from the newly created original instances checkpoint the lbrFilerecord defined indb.rev2of all files associated with the depot path we are splitting.For examplegrep @db.rev@ /checkpoint.XXX | grep //targeted/path/to/split/1
  3. 3. 3 DEPOT SPLITTINGThis will give you the db.rev entries for the path you want to split. From these entries you pullthe lbrfile column and remove all entries referring to the original path. This will give you thelocation of all lazy integrated files.We will add these paths to the splitmap(mapping) files already containing the path(s) we aresplitting from the original depot. This is a necessary step because we are not making use ofthe p4 snap feature of perfsplit.TRANSITIONOnce we have thismapping we can begin our split using perfsplit using the minimum options,source, output, splitmap file, but we also need an additional (undocumented) option „–a‟ to skipthe perfsplit archive file copy step. This will build a duplicate instance of the original metadatafor all depots associated with the originals split path in the output path. Since we don‟t wanttwo instances with depots of the same name, we need to take another checkpoint of this newinstance.CONVERSIONWith this new checkpoint, we can shape the metadata into a new data structures.To do this webuild another instance from the newly created checkpoint, but during creation (replay) wemake some substitutions to point the current data structure to what we want.For example, to do this we would use the following commands:p4d -r $p4root -f -jr – | sed –e s#//foo/path/#//bar/path/#Now we have a new instance with the correct metadata.CONNECTIONThe conversion has now pointed the original metadata to a new depot area. We will need tocreate this new depot „bar‟ to access this area. The files for this depot need to be copied fromthe original location or if space is a factor it can be left in the original locationor moved to a newone and the symlinked(depending on an installations particular circumstance).Once this is done, you will have a new instance with a different name containing a completedata structure of split files.VERIFICATIONVerification of the new instance should be run to test the success of the transfer. There areonly two problems that can occur.Verification returns a "BAD" error. This is reported when the MD5 digest of a filerevision stored in the Perforce database differs from the one calculated by the p4 verifycommand. This error indicates that the file revision might be corrupted. This is mostlikely due to changes to the physical files during transfer. Otherwise files should beconfirmed by someone familiar with them or by diffing them from the original.33
  4. 4. 4 DEPOT SPLITTINGVerification returns a “MISSING” error. This indicates that Perforce cannot find thespecified revisions within the versioned file tree. Most likely this means the archive fileis missing from the versioned file tree. Check the lbrfile record of this file and makesure this file is in it correct location, make sure the new instance can access thislocation, and make sure this files location was part of the splitmap4.CLEANUPYou will notice perfsplit carried over all the depots from the original instance regardless if theywere part of the splitmap or not. These can be removed except for any depot containing datafrom integrated files located on your original split path. These extra depots can be easilyhidden by restricting their view in the protection table making the new instance looks like it onlyhas the single depot.COMPLETIONBy implementing these steps using the perforce tool perfsplit, the issues regarding the zerodowntime, duplicate naming, and integration history are addressed. Resolution of these issuesmakes Perfsplit a more desirable tool in a large installation environment ….4