The Great
       Migration:
       from TeamWare to
       Mercurial

         James C. McPherson
         Senior Kernel E...
Introduction
Nomenclature and History
TeamWare
Replacement SCM Candidates
Mercurial
What we have learned
References



   ...
Introduction
  January 2005
        Sun releases DTrace under CDDL
  14 June 2005
        Sun releases the bulk of the cor...
Some nomenclature......
  “Solaris” and “Solaris Express” are
  constructed from several Consolidations:

  OS and Network...
A bit more nomenclature
  Under TeamWare, each Consolidation is
  housed in The Gate, which is updated to
  The Clone ever...
A picture, please?

    The Gate
    exported r/o
    TeamWare metadata                                                   ...
Why not stick with TeamWare?
  TeamWare's code was not Open
  TeamWare had been EOL'd (End Of Life'd)
  Getting patches fo...
Development with TeamWare
  TeamWare is not just a collection of
  wrappers around SCCS, it also provides
  application/sc...
Other development infrastructure
  Developer scripts include
       nightly – to build your workspace
       wx (Workspace...
What do these scripts do? (1)
  nightly – general purpose gate and
  workspace build script. So-called because
  it's run ...
What do these scripts do? (2)
  wx sits atop TeamWare
  wx helps track checked out files, bug and
  ARC comments and sanit...
What do these scripts do? (3)
  bfu, the Bonwick-Faulkner Upgrade, takes
  cpio archives produced by 'nightly' and
  blats...
What do these scripts do? (4)
  Webrev generates web pages of changes
  for easy code review




      The Great Migration...
The Great Migration: from TeamWare to Mercurial, what we learned along the way
                                           ...
Gate infrastructure
  Scripts written by the gate staff (yes,
  called gatekeepers... no, there is no
  keymaster)
       ...
Time For A Change
  Both Sun and the OpenSolaris Community
  agreed that updating the SCM system was
  required
  OpenSola...
Major Requirements
  Must be Open Source
  Unbiased and disconnected distribution
  Networked operation
  Interface stabil...
Candidate SCM systems
  SubVersion
  Mercurial
  Bazaar / Bazaar-NG
  Git
  Monotone
  SVK
  TeamWare



      The Great M...
Evaluations
  SubVersion, SVK, Monotone and
  TeamWare were winnowed out via mailing
  list discussion (tools-discuss) in ...
Successful SCM candidate




                   Mercurial



     The Great Migration: from TeamWare to Mercurial, what we...
Once Mercurial was chosen, the real work
could get started


.... and boy, was there a lot of it!




    The Great Migrat...
Migration tasks
  Work divided into four major areas:

        Developer-focused tools
        Changing source in the gate...
Developer-focused tools
  Three major scripts needed updating:
        nightly
        wx
        webrev
  Many wx feature...
Why call it Cadmium?




http://www.csudh.edu/oliver/chemdata/periodic/periodic-1.htm
           The Great Migration: from...
Gate source changes
  While managed by TeamWare, all files
  had embedded SCCS ident strings:

  %Z%%M%                   ...
Gate infrastructure changes
  The infrastructure saw a lot of the heavy
  lifting for the migration, leveraging the
  chan...
Gate back-end model




Images by David Marker, used with permission
              The Great Migration: from TeamWare to M...
New model: general

    The Gate
                                                                                         ...
New model: Open and Closed

 Gate (Closed)
                                                                               ...
ON Go-live
  ON went live with Mercurial on 6 August
  2008, when build snv_97 opened:


  6733918 Teamware has retired, p...
What did we learn?




  The Great Migration: from TeamWare to Mercurial, what we learned along the way
                  ...
What did we learn? (1)
  The OpenSolaris community now knows a
  lot more about Sun's internal processes
  Switching to a ...
What did we learn? (2)
  Your time estimates are too optimistic
  Push up to date tools to the community as
  soon as poss...
What did we learn? (3)
  The existing SCM dictates workflow
  implementation, which influences what
  developers see as ga...
What did we learn? (4)
  The tools in use must support both the old
  and new SCMs
        developers need to migrate exis...
Finally....




   SCM migration is an
   all-or-nothing change

   Your bags must be packed
   for the journey

       Th...
References

  http://www.opensolaris.org/os/community/tools/scm
  SCM Migration project page
  http://www.opensolaris.org/...
Further reading


  http://opensolaris.org/os/community/on/os_dev_process/
  OpenSolaris development process

  http://ope...
What   Questions                                      Do            You                 Have?
What   Questions            ...
The Great
       Migration:
       from TeamWare to
       Mercurial
         James C. McPherson
         Senior Kernel En...
Upcoming SlideShare
Loading in...5
×

The Great Migration: from TeamWare to Mercurial

3,402

Published on

The Great Migration: from TeamWare to Mercurial, what we learned along the way.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,402
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
19
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

The Great Migration: from TeamWare to Mercurial

  1. 1. The Great Migration: from TeamWare to Mercurial James C. McPherson Senior Kernel Engineer Sun Microsystems The Great Migration: from TeamWare to Mercurial, what we learned along the way James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  2. 2. Introduction Nomenclature and History TeamWare Replacement SCM Candidates Mercurial What we have learned References The Great Migration: from TeamWare to Mercurial, what we learned along the way 2 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  3. 3. Introduction January 2005 Sun releases DTrace under CDDL 14 June 2005 Sun releases the bulk of the core Solaris kernel and basic userland under CDDL 30000+ files, several million LoC Source Code Management system? SCCS wrapped in TeamWare It was Time For A Change(tm) The Great Migration: from TeamWare to Mercurial, what we learned along the way 3 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  4. 4. Some nomenclature...... “Solaris” and “Solaris Express” are constructed from several Consolidations: OS and Networking, aka OS/Net, aka “ON” Admin and Install, aka “AI” Java JDS (Desktop) and X SunCluster NetWorkStorage (now part of ON), and xVM The Great Migration: from TeamWare to Mercurial, what we learned along the way 4 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  5. 5. A bit more nomenclature Under TeamWare, each Consolidation is housed in The Gate, which is updated to The Clone every night (11pm US/Pacific is the convention) You putback to The Gate You bringover from The Clone The Gate and Clone filesystems are exported read-only, but TeamWare's metadata directories are exported read- write The Great Migration: from TeamWare to Mercurial, what we learned along the way 5 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  6. 6. A picture, please? The Gate exported r/o TeamWare metadata putback exported r/w Updated at 11pm US/Pacific every night Development process The Clone exported r/o TeamWare metadata bringover exported r/w The Great Migration: from TeamWare to Mercurial, what we learned along the way 6 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  7. 7. Why not stick with TeamWare? TeamWare's code was not Open TeamWare had been EOL'd (End Of Life'd) Getting patches for bugfixes was hard Getting patches for new features was really hard TeamWare was ill suited to globally distributed development, and The operational model did not allow moving Gates outside Sun The Great Migration: from TeamWare to Mercurial, what we learned along the way 7 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  8. 8. Development with TeamWare TeamWare is not just a collection of wrappers around SCCS, it also provides application/scripting triggers, tracks name changes, aggregates operations and even provides checkpointing. Heavy use of NFS – fine when all the developers are in one building Over time, engineers wrote their own wrappers around TeamWare, to workaround its shortcomings The Great Migration: from TeamWare to Mercurial, what we learned along the way 8 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  9. 9. Other development infrastructure Developer scripts include nightly – to build your workspace wx (Workspace Extensions) – to manage your workspace bfu (Bonwick-Faulkner Upgrade) webrev – generates webpages for code review The Great Migration: from TeamWare to Mercurial, what we learned along the way 9 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  10. 10. What do these scripts do? (1) nightly – general purpose gate and workspace build script. So-called because it's run every night on the gate source, and used to take all night to run. nightly is generally maintained by ON engineers, and used by several other consolidations as well nightly has a number of options, including package generation, lint checking, C and Java style checking The Great Migration: from TeamWare to Mercurial, what we learned along the way 10 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  11. 11. What do these scripts do? (2) wx sits atop TeamWare wx helps track checked out files, bug and ARC comments and sanity checks your workspace on request wx shields the user from most raw SCCS operations The Great Migration: from TeamWare to Mercurial, what we learned along the way 11 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  12. 12. What do these scripts do? (3) bfu, the Bonwick-Faulkner Upgrade, takes cpio archives produced by 'nightly' and blats them onto your system under test Currently the fastest way of getting a new kernel and consistent minimal userland (commands + libraries) on your system. Gate machines are bfu'd every morning – we “eat our own dogfood” The Great Migration: from TeamWare to Mercurial, what we learned along the way 12 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  13. 13. What do these scripts do? (4) Webrev generates web pages of changes for easy code review The Great Migration: from TeamWare to Mercurial, what we learned along the way 13 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  14. 14. The Great Migration: from TeamWare to Mercurial, what we learned along the way 14 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  15. 15. Gate infrastructure Scripts written by the gate staff (yes, called gatekeepers... no, there is no keymaster) nightly buglist backout – used for reverting bad changes lock-gate / unlock-gate – used when large changes are integrated update-flagdays The Great Migration: from TeamWare to Mercurial, what we learned along the way 15 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  16. 16. Time For A Change Both Sun and the OpenSolaris Community agreed that updating the SCM system was required OpenSolaris project created to oversee this change: scm-migration Requirements list generated by the Community and tuned by the project Candidate SCM systems evaluated against requirements, evaluation reports published on opensolaris.org The Great Migration: from TeamWare to Mercurial, what we learned along the way 16 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  17. 17. Major Requirements Must be Open Source Unbiased and disconnected distribution Networked operation Interface stability and completeness Standard operations and transactions Per-changeset metadata For the full list, visit http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/ The Great Migration: from TeamWare to Mercurial, what we learned along the way 17 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  18. 18. Candidate SCM systems SubVersion Mercurial Bazaar / Bazaar-NG Git Monotone SVK TeamWare The Great Migration: from TeamWare to Mercurial, what we learned along the way 18 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  19. 19. Evaluations SubVersion, SVK, Monotone and TeamWare were winnowed out via mailing list discussion (tools-discuss) in the preliminary phase Bazaar, Git and Mercurial all continued to the next phase Evaluation reports are available at http://www.opensolaris.org The Great Migration: from TeamWare to Mercurial, what we learned along the way 19 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  20. 20. Successful SCM candidate Mercurial The Great Migration: from TeamWare to Mercurial, what we learned along the way 20 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  21. 21. Once Mercurial was chosen, the real work could get started .... and boy, was there a lot of it! The Great Migration: from TeamWare to Mercurial, what we learned along the way 21 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  22. 22. Migration tasks Work divided into four major areas: Developer-focused tools Changing source in the gate Gate infrastructure tools Education The Great Migration: from TeamWare to Mercurial, what we learned along the way 22 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  23. 23. Developer-focused tools Three major scripts needed updating: nightly wx webrev Many wx features were ported to Mercurial extensions (thus written in Python) and called “Cadmium” nightly and webrev were enhanced to understand Mercurial as well as TeamWare The Great Migration: from TeamWare to Mercurial, what we learned along the way 23 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  24. 24. Why call it Cadmium? http://www.csudh.edu/oliver/chemdata/periodic/periodic-1.htm The Great Migration: from TeamWare to Mercurial, what we learned along the way 24 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  25. 25. Gate source changes While managed by TeamWare, all files had embedded SCCS ident strings: %Z%%M% %I% %E% SMI Since these are an anachronism with Mercurial, they were removed in files that were under active development at the migration point The Great Migration: from TeamWare to Mercurial, what we learned along the way 25 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  26. 26. Gate infrastructure changes The infrastructure saw a lot of the heavy lifting for the migration, leveraging the changes for the wx to Cdm work Scripts (ksh88) were re-written in Python as Mercurial hooks These are still being tweaked, but the majority of changes are bedded down The Great Migration: from TeamWare to Mercurial, what we learned along the way 26 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  27. 27. Gate back-end model Images by David Marker, used with permission The Great Migration: from TeamWare to Mercurial, what we learned along the way 27 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  28. 28. New model: general The Gate hg push Development process The Clone Updated after every successful hg push (almost immediately) hg pull The Great Migration: from TeamWare to Mercurial, what we learned along the way 28 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  29. 29. New model: Open and Closed Gate (Closed) hg push Gate (Open) Separate hg push and hg pull/clone required for Closed and Open parts Development process Clone (Open) Clone (Closed) hg pull Updated after every successful hg push (ie, almost immediately) The Great Migration: from TeamWare to Mercurial, what we learned along the way 29 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  30. 30. ON Go-live ON went live with Mercurial on 6 August 2008, when build snv_97 opened: 6733918 Teamware has retired, please welcome your new manager, Mercurial The Great Migration: from TeamWare to Mercurial, what we learned along the way 30 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  31. 31. What did we learn? The Great Migration: from TeamWare to Mercurial, what we learned along the way 31 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  32. 32. What did we learn? (1) The OpenSolaris community now knows a lot more about Sun's internal processes Switching to a new SCM is like a sausage (it's really ugly while it's happening) Corporate firewalls are necessary, but get in the way of migrating The longer you've been using the old system, the harder it is to change The longer you've been using the old system, the more obscure its foibles The Great Migration: from TeamWare to Mercurial, what we learned along the way 32 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  33. 33. What did we learn? (2) Your time estimates are too optimistic Push up to date tools to the community as soon as possible We had (and have!) essential developer tools that very few people understood We have developer tools so old that the last people to understand them properly have retired The better you know SCMs in general, the easier your migration will be The Great Migration: from TeamWare to Mercurial, what we learned along the way 33 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  34. 34. What did we learn? (3) The existing SCM dictates workflow implementation, which influences what developers see as gaps needing plugs Engineers don't like change. At least, not too much change A consistent developer tool experience requires SCM-agnosticism Migrating to a modern, sophisticated SCM is a great opportunity to change lots of niggles that require coordination The Great Migration: from TeamWare to Mercurial, what we learned along the way 34 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  35. 35. What did we learn? (4) The tools in use must support both the old and new SCMs developers need to migrate existing workspaces Non-integrated projects need to change and sync up Sustaining organisations need to work with both SCMs Productivity will dip while people work their way up the learning curve The Great Migration: from TeamWare to Mercurial, what we learned along the way 35 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  36. 36. Finally.... SCM migration is an all-or-nothing change Your bags must be packed for the journey The Great Migration: from TeamWare to Mercurial, what we learned along the way 36 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  37. 37. References http://www.opensolaris.org/os/community/tools/scm SCM Migration project page http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/ SCM Migration requirements list http://www.opensolaris.org/os/community/tools/scm/scm_milestone SCM Migration milestones http://www.opensolaris.org/os/community/tools/scm/on-scm-tools ON consolidation tools The Great Migration: from TeamWare to Mercurial, what we learned along the way 37 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  38. 38. Further reading http://opensolaris.org/os/community/on/os_dev_process/ OpenSolaris development process http://opensolaris.org/os/community/on/devref_toc OpenSolaris ON developer's reference guide An extended version of this presentation will appear at http://blogs.sun.com/jmcp The Great Migration: from TeamWare to Mercurial, what we learned along the way 38 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  39. 39. What Questions Do You Have? What Questions Do You Have? What Questions Do You Have? What Questions Do You Have? What Questions Do You Have? What What Questions Do You Have? Questions What What Questions Questions Do Do You You Have? Have? Do You Have? What Questions Do You Have? What Questions Do You Have? What Questions Do You Have? The Great Migration: from TeamWare to Mercurial, what we learned along the way 39 James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  40. 40. The Great Migration: from TeamWare to Mercurial James C. McPherson Senior Kernel Engineer Sun Microsystems jmcp@Sun.COM The Great Migration: from TeamWare to Mercurial, what we learned along the way James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×