The Great Migration: from TeamWare to Mercurial

The Great Migration: from TeamWare to Mercurial James C. McPherson Senior Kernel Engineer Sun Microsystems The Great Migration: from TeamWare to Mercurial, what we learned along the way James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Introduction Nomenclature and History TeamWare Replacement SCM Candidates Mercurial What we have learned References The Great Migration: from TeamWare to Mercurial, what we learned along the way 2 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Introduction January 2005 Sun releases DTrace under CDDL 14 June 2005 Sun releases the bulk of the core Solaris kernel and basic userland under CDDL 30000+ files, several million LoC Source Code Management system? SCCS wrapped in TeamWare It was Time For A Change(tm) The Great Migration: from TeamWare to Mercurial, what we learned along the way 3 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Some nomenclature...... “Solaris” and “Solaris Express” are constructed from several Consolidations: OS and Networking, aka OS/Net, aka “ON” Admin and Install, aka “AI” Java JDS (Desktop) and X SunCluster NetWorkStorage (now part of ON), and xVM The Great Migration: from TeamWare to Mercurial, what we learned along the way 4 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

A bit more nomenclature Under TeamWare, each Consolidation is housed in The Gate, which is updated to The Clone every night (11pm US/Pacific is the convention) You putback to The Gate You bringover from The Clone The Gate and Clone filesystems are exported read-only, but TeamWare's metadata directories are exported read- write The Great Migration: from TeamWare to Mercurial, what we learned along the way 5 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

A picture, please? The Gate exported r/o TeamWare metadata putback exported r/w Updated at 11pm US/Pacific every night Development process The Clone exported r/o TeamWare metadata bringover exported r/w The Great Migration: from TeamWare to Mercurial, what we learned along the way 6 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Why not stick with TeamWare? TeamWare's code was not Open TeamWare had been EOL'd (End Of Life'd) Getting patches for bugfixes was hard Getting patches for new features was really hard TeamWare was ill suited to globally distributed development, and The operational model did not allow moving Gates outside Sun The Great Migration: from TeamWare to Mercurial, what we learned along the way 7 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Development with TeamWare TeamWare is not just a collection of wrappers around SCCS, it also provides application/scripting triggers, tracks name changes, aggregates operations and even provides checkpointing. Heavy use of NFS – fine when all the developers are in one building Over time, engineers wrote their own wrappers around TeamWare, to workaround its shortcomings The Great Migration: from TeamWare to Mercurial, what we learned along the way 8 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Other development infrastructure Developer scripts include nightly – to build your workspace wx (Workspace Extensions) – to manage your workspace bfu (Bonwick-Faulkner Upgrade) webrev – generates webpages for code review The Great Migration: from TeamWare to Mercurial, what we learned along the way 9 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

What do these scripts do? (1) nightly – general purpose gate and workspace build script. So-called because it's run every night on the gate source, and used to take all night to run. nightly is generally maintained by ON engineers, and used by several other consolidations as well nightly has a number of options, including package generation, lint checking, C and Java style checking The Great Migration: from TeamWare to Mercurial, what we learned along the way 10 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

What do these scripts do? (2) wx sits atop TeamWare wx helps track checked out files, bug and ARC comments and sanity checks your workspace on request wx shields the user from most raw SCCS operations The Great Migration: from TeamWare to Mercurial, what we learned along the way 11 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

What do these scripts do? (3) bfu, the Bonwick-Faulkner Upgrade, takes cpio archives produced by 'nightly' and blats them onto your system under test Currently the fastest way of getting a new kernel and consistent minimal userland (commands + libraries) on your system. Gate machines are bfu'd every morning – we “eat our own dogfood” The Great Migration: from TeamWare to Mercurial, what we learned along the way 12 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

What do these scripts do? (4) Webrev generates web pages of changes for easy code review The Great Migration: from TeamWare to Mercurial, what we learned along the way 13 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Gate infrastructure Scripts written by the gate staff (yes, called gatekeepers... no, there is no keymaster) nightly buglist backout – used for reverting bad changes lock-gate / unlock-gate – used when large changes are integrated update-flagdays The Great Migration: from TeamWare to Mercurial, what we learned along the way 15 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Time For A Change Both Sun and the OpenSolaris Community agreed that updating the SCM system was required OpenSolaris project created to oversee this change: scm-migration Requirements list generated by the Community and tuned by the project Candidate SCM systems evaluated against requirements, evaluation reports published on opensolaris.org The Great Migration: from TeamWare to Mercurial, what we learned along the way 16 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Major Requirements Must be Open Source Unbiased and disconnected distribution Networked operation Interface stability and completeness Standard operations and transactions Per-changeset metadata For the full list, visit http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/ The Great Migration: from TeamWare to Mercurial, what we learned along the way 17 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Candidate SCM systems SubVersion Mercurial Bazaar / Bazaar-NG Git Monotone SVK TeamWare The Great Migration: from TeamWare to Mercurial, what we learned along the way 18 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Evaluations SubVersion, SVK, Monotone and TeamWare were winnowed out via mailing list discussion (tools-discuss) in the preliminary phase Bazaar, Git and Mercurial all continued to the next phase Evaluation reports are available at http://www.opensolaris.org The Great Migration: from TeamWare to Mercurial, what we learned along the way 19 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Once Mercurial was chosen, the real work could get started .... and boy, was there a lot of it! The Great Migration: from TeamWare to Mercurial, what we learned along the way 21 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Migration tasks Work divided into four major areas: Developer-focused tools Changing source in the gate Gate infrastructure tools Education The Great Migration: from TeamWare to Mercurial, what we learned along the way 22 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Developer-focused tools Three major scripts needed updating: nightly wx webrev Many wx features were ported to Mercurial extensions (thus written in Python) and called “Cadmium” nightly and webrev were enhanced to understand Mercurial as well as TeamWare The Great Migration: from TeamWare to Mercurial, what we learned along the way 23 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Why call it Cadmium? http://www.csudh.edu/oliver/chemdata/periodic/periodic-1.htm The Great Migration: from TeamWare to Mercurial, what we learned along the way 24 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Gate source changes While managed by TeamWare, all files had embedded SCCS ident strings: %Z%%M% %I% %E% SMI Since these are an anachronism with Mercurial, they were removed in files that were under active development at the migration point The Great Migration: from TeamWare to Mercurial, what we learned along the way 25 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Gate infrastructure changes The infrastructure saw a lot of the heavy lifting for the migration, leveraging the changes for the wx to Cdm work Scripts (ksh88) were re-written in Python as Mercurial hooks These are still being tweaked, but the majority of changes are bedded down The Great Migration: from TeamWare to Mercurial, what we learned along the way 26 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Gate back-end model Images by David Marker, used with permission The Great Migration: from TeamWare to Mercurial, what we learned along the way 27 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

New model: general The Gate hg push Development process The Clone Updated after every successful hg push (almost immediately) hg pull The Great Migration: from TeamWare to Mercurial, what we learned along the way 28 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

New model: Open and Closed Gate (Closed) hg push Gate (Open) Separate hg push and hg pull/clone required for Closed and Open parts Development process Clone (Open) Clone (Closed) hg pull Updated after every successful hg push (ie, almost immediately) The Great Migration: from TeamWare to Mercurial, what we learned along the way 29 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

ON Go-live ON went live with Mercurial on 6 August 2008, when build snv_97 opened: 6733918 Teamware has retired, please welcome your new manager, Mercurial The Great Migration: from TeamWare to Mercurial, what we learned along the way 30 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

What did we learn? (1) The OpenSolaris community now knows a lot more about Sun's internal processes Switching to a new SCM is like a sausage (it's really ugly while it's happening) Corporate firewalls are necessary, but get in the way of migrating The longer you've been using the old system, the harder it is to change The longer you've been using the old system, the more obscure its foibles The Great Migration: from TeamWare to Mercurial, what we learned along the way 32 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

What did we learn? (2) Your time estimates are too optimistic Push up to date tools to the community as soon as possible We had (and have!) essential developer tools that very few people understood We have developer tools so old that the last people to understand them properly have retired The better you know SCMs in general, the easier your migration will be The Great Migration: from TeamWare to Mercurial, what we learned along the way 33 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

What did we learn? (3) The existing SCM dictates workflow implementation, which influences what developers see as gaps needing plugs Engineers don't like change. At least, not too much change A consistent developer tool experience requires SCM-agnosticism Migrating to a modern, sophisticated SCM is a great opportunity to change lots of niggles that require coordination The Great Migration: from TeamWare to Mercurial, what we learned along the way 34 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

What did we learn? (4) The tools in use must support both the old and new SCMs developers need to migrate existing workspaces Non-integrated projects need to change and sync up Sustaining organisations need to work with both SCMs Productivity will dip while people work their way up the learning curve The Great Migration: from TeamWare to Mercurial, what we learned along the way 35 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Finally.... SCM migration is an all-or-nothing change Your bags must be packed for the journey The Great Migration: from TeamWare to Mercurial, what we learned along the way 36 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

References http://www.opensolaris.org/os/community/tools/scm SCM Migration project page http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/ SCM Migration requirements list http://www.opensolaris.org/os/community/tools/scm/scm_milestone SCM Migration milestones http://www.opensolaris.org/os/community/tools/scm/on-scm-tools ON consolidation tools The Great Migration: from TeamWare to Mercurial, what we learned along the way 37 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

Further reading http://opensolaris.org/os/community/on/os_dev_process/ OpenSolaris development process http://opensolaris.org/os/community/on/devref_toc OpenSolaris ON developer's reference guide An extended version of this presentation will appear at http://blogs.sun.com/jmcp The Great Migration: from TeamWare to Mercurial, what we learned along the way 38 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

What Questions Do You Have? What Questions Do You Have? What Questions Do You Have? What Questions Do You Have? What Questions Do You Have? What What Questions Do You Have? Questions What What Questions Questions Do Do You You Have? Have? Do You Have? What Questions Do You Have? What Questions Do You Have? What Questions Do You Have? The Great Migration: from TeamWare to Mercurial, what we learned along the way 39 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

The Great Migration: from TeamWare to Mercurial James C. McPherson Senior Kernel Engineer Sun Microsystems jmcp@Sun.COM The Great Migration: from TeamWare to Mercurial, what we learned along the way James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved

The Great Migration: from TeamWare to Mercurial

0 comments

Notes on slide 1

1 Favorite & 1 Event