A tale about a Big SVN to Git Migration
Upcoming SlideShare
Loading in...5
×
 

A tale about a Big SVN to Git Migration

on

  • 1,906 views

In November 2012 I helped the giant JBoss Tools repo containing 400+ eclipse plugins, 2+ millions line of code with 7+ year of history migrate from SVN into a multitude of Git repositories. ...

In November 2012 I helped the giant JBoss Tools repo containing 400+ eclipse plugins, 2+ millions line of code with 7+ year of history migrate from SVN into a multitude of Git repositories.

Most existing large migrations I could find info about was about the size as one of our 30+ modules in JBoss tools and it turned out the advice in most of these just did not cope with a migration as large as this.

During this tale I cover what tools I examined, rejected and in the end selected to do the migration. I will outline the migration process we did as a team, what things Git enabled immediately and which things we are moving towards now that Git have opened up for a more flexible build and development model and structure.

Before this move a 6+ months long preparation and even longer thinking process was started to ensure all data would be migrated properly in an efficient and repeatable manner with as little possible impact on developer productivity as possible.

After this talk you should be well prepared to do both small and large migrations to git for Eclipse based plugin projects and know what dangers to look out for and what tools to consider.

If you want to take a sneakpeak you can see the scripts and instructions for the migration at https://github.com/maxandersen/jbosstools-gitmigration

Statistics

Views

Total Views
1,906
Views on SlideShare
1,669
Embed Views
237

Actions

Likes
2
Downloads
19
Comments
4

4 Embeds 237

https://twitter.com 193
http://eventifier.co 23
http://www.twylah.com 19
http://eventifier.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

14 of 4 Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • @tamarmot ah yes - if that is the only thing you need then yeah you can do that trick too ;) The link I posted before is more if you need to rename *subparts* of a move/conversion.
    Are you sure you want to
    Your message goes here
    Processing…
  • @tamarmot happy to hear you are liking this. For some reason I don't get notifications for these comments thus I didn't see this sooner.

    If I understand you correctly your problem is you want to remove trunk/plugins/nasa completely and simply just have trunk/plugins/nasa as the root of your repo ?

    If that is the case then I think this might help you: https://github.com/maxandersen/jbosstools-gitmigration/blob/master/filter_tests.py#L42

    That line and following is what we use to rename some of our files when converting/spliiting our test repos.

    Hope that helps.
    Are you sure you want to
    Your message goes here
    Processing…
  • Ah, I posted too soon -- I can just customize this to include the directory:
    git filter-branch --subdirectory-filter foodir -- --all

    so for example
    git filter-branch --subdirectory-filter plugins/nasa -- --all

    thanks!
    Are you sure you want to
    Your message goes here
    Processing…
  • Hi Max -- we are using your tools to do a similar migration from SVN to git here at NASA (THANK YOU!) and one issue I'm having is that when we break up our svn directories into git repositories, the paths from svn are subdirectories in the git repository. For example, in SVN the path is trunk/plugins/nasa; in our new git repository called nasa there is a path plugins/nasa/ before any of the projects. Is there a flag or something I should pass in to move things, or do I have to write my own script to move stuff around within git? Thanks for any help you can provide.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    A tale about a Big SVN to Git Migration A tale about a Big SVN to Git Migration Presentation Transcript

    • A tale about a Big SVN to Git migration Max rydahl Andersen @maxandersen Senior Principal Engineer, red HatWednesday 27 March 13
    • • Do not use svn2git • Learn about fast-filter.py • Scripts/Resources at: • https://github.com/maxandersen/jbosstools-gitmigrationWednesday 27 March 13
    • goals for git migration • Split up Big Repo to smaller ones • Keep the history • Get to use GitHub • Make it fast and fun again!Wednesday 27 March 13
    • Keeping it simple common plugins .core .ui ... features .feature .source.fe ature ... tests .tests ... siteWednesday 27 March 13
    • Keeping it simple common plugins .core .ui ... features .feature .source.fe ature ... tests .tests Svn ... site git svn clone -s git svn --authors-file=authors.txt http://svn.example.com/svn-repository/ svn-mirror svn-mirror.gitWednesday 27 March 13
    • ...a real project common plugins .core .ui ... features .feature .source.fe ature ... tests .tests ... siteWednesday 27 March 13
    • ...a real project JSF ... plugins .core ... plugins .core common plugins .core hibernate plugins .core CDI plugins .core plugins .core seam plugins .core .ui .ui .ui .ui .ui .ui .ui ... ... ... ... ... features .feature ... ... .source.fe features .feature features .feature ature features .feature features .feature features .feature features .feature ... .source.fe ature .source.fe ature .source.fe .source.fe .source.fe tests .tests .source.fe ature ature ature ... ature ... ... ... ... ... tests .tests site ... tests .tests ... tests .tests tests .tests tests .tests tests .tests ... site ... site ... ... ... site site site site ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui ... ... ... ... ... ... ... ... ... ... ... ... features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe ature ature ature ature ature ature ature ature ature ature ature ature ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... ... ... ... ... ... ... ... ... ... ... ... .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature site site site site site site site site site site site site .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe ature ature ature ature ature ature ature ature ature ature ature ature ... ... ... ... ... ... ... ... ... ... ... ... tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... ... ... ... ... ... ... ... ... ... ... ... .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui site site site site site site site site site site site site ... ... ... ... ... ... ... ... ... ... ... ... features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe ature ature ature ature ature ature ature ature ature ature ature ature ... ... ... ... ... ... ... ... ... ... ... ... tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests ... ... ... ... ... ... ... ... ... ... ... ... site site site site site site site site site site site siteWednesday 27 March 13
    • How hard can it be ? Svn git svn clone --prefix=svn/ -s git svn --authors-file=authors.txt http://svn.example.com/svn-repository/ svn-mirror svn-mirror.gitWednesday 27 March 13
    • Check list for migration 1.Create svn-mirror.git from mirrored svn 2.Cleanup Mirror 1. rename master to trunk, Delete dead branches, Checkout all branches, tag branches to real tags 3.Create individual Repositories 1. Use fast-filter.py to process content, Remove empty tags/branches, Garbage Collect 4.Clean up Garbage 1. Prune big files, Fix line endings, Garbage Collect 5. ? (Publish to Github) 6.Profit!Wednesday 27 March 13
    • SVN Mirroring svn sync Svn Svn Mirror svnsync init file://svn-mirror http://svn.example.com/repository while(true) svnsync sync file://svn-mirror http://svn.example.com/repositoryWednesday 27 March 13
    • SVN Mirroring svn sync Svn Svn Mirror svnsync init file://svn-mirror http://svn.example.com/repository while(true) git svn svnsync sync file://svn-mirror http://svn.example.com/repository cd svn-mirror git svn rebase --fetch-all svn-mirror.gitWednesday 27 March 13
    • SVN Mirroring svn sync Svn Svn Mirror svnsync init file://svn-mirror http://svn.example.com/repository while(true) git svn svnsync sync file://svn-mirror http://svn.example.com/repository cd svn-mirror git svn rebase --fetch-all svn-mirror.git Svn and Git now isolated and always in sync !!!Wednesday 27 March 13
    • svn mirroring • Atlassian blogs has excellent material, but... • Use --fetch-all to get full history! • Do not remove metadata - it will help you in case of failuresWednesday 27 March 13
    • Cleanup Branches/Tags svn-mirror.gitWednesday 27 March 13
    • Cleanup Branches/Tags svn-mirror.git • Branches: • trunk • deadbranch • tags/jbt-4.0.1 • tags/jbt-4.0.2 • jbt-4.0.x • jbt-3.03Wednesday 27 March 13
    • Cleanup Branches/Tags svn-mirror.git • Branches: • Branches: • trunk • master • jbt-4.0.x • deadbranch • jbt-3.03 • tags/jbt-4.0.1 • Tags: • tags/jbt-4.0.2 • jbt-4.0.1 • jbt-4.0.x • jbt-4.0.2 • jbt-3.03Wednesday 27 March 13
    • Cleanup branches/tags svn-mirror.git # Rename trunk to master $ git branch -m trunk master $ git branch -rd origin/trunk # Delete dead branches $ git branch -a -r | grep -E "dead|broken|weird|I_WAS_DRUNK" | xargs git branch -r -DWednesday 27 March 13
    • Cleanup branches/tags svn-mirror.git # Convert svn tag-branches into git-tags $ git branch | grep tags | sed -e "s/tags///g" | xargs -n 1 -I {} git tag -m "svn branch tag" {} tags/{} # delete svn tag-branches $ git branch | grep tags | xargs -n 1 git branch -dWednesday 27 March 13
    • To split or not to split ? svn-mirror.gitWednesday 27 March 13
    • To split or not to split ? svn-mirror.gitWednesday 27 March 13
    • Wednesday 27 March 13
    • split/filter process svn-mirror.git • /modulex • plugina • pluginb • /moduley • pluginaWednesday 27 March 13
    • split/filter process svn-mirror.git fastfilter • /modulex • /modulex • plugina • plugina • pluginb • pluginb • /moduley • pluginaWednesday 27 March 13
    • split/filter process svn-mirror.git fastfilter filter-branch • /modulex • /modulex • plugina • plugina • plugina • pluginb • pluginb • pluginb • /moduley • pluginaWednesday 27 March 13
    • fast-filter+filter-branch $ python filter_repo.py jbosstools-svn-mirror jbosstools-openshift "^openshift.*"Wednesday 27 March 13
    • fast-filter+filter-branch $ python filter_repo.py jbosstools-svn-mirror jbosstools-openshift "^openshift.*" $ git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter openshift -f -- --allWednesday 27 March 13
    • Clean up branches (again) • $ deleteemptybranches.sh openshift-repo • # delete empty commits • $ git filter-branch --tag-name-filter cat --prune-empty -- --allWednesday 27 March 13
    • “it’s simple”Wednesday 27 March 13
    • “it’s simple” svn sync Svn Svn MirrorWednesday 27 March 13
    • “it’s simple” svn sync Svn Svn Mirror git svn svn-mirror.gitWednesday 27 March 13
    • “it’s simple” svn sync Svn Svn Mirror git svn svn-mirror.git cleanupWednesday 27 March 13
    • “it’s simple” svn sync Svn Svn Mirror git svn svn-mirror.git cleanupWednesday 27 March 13
    • “it’s simple” svn sync Svn Svn Mirror cleanup git svn svn-mirror.git cleanupWednesday 27 March 13
    • Check list for migration 1.Create svn-mirror.git from mirrored svn 2.Cleanup Mirror 1. rename master to trunk, Delete dead branches, Checkout all branches, tag branches to real tags 3.Create individual Repositories 1. Use fast-filter.py to process content, Remove empty tags/branches, Garbage Collect 4.Clean up Garbage 1. Prune big files, Fix line endings, add .gitignore Garbage Collect 5. ? 6.Profit!Wednesday 27 March 13
    • Tips’n’TricksWednesday 27 March 13
    • Train your team • http://try.github.com • OReilly Videos • Understand the internals: https://vimeo.com/49444883 • Learn the CLI before using the UI’s • Ban git push --force • Green Button Considered HarmfulWednesday 27 March 13
    • guardians vs KingsWednesday 27 March 13
    • Git Hub api is your friendWednesday 27 March 13
    • Hub “github cli” http://defunkt.io/hub/ • $ hub clone maxandersen/autolinkjboss • $ hub fork • $ hub pull-request • “Fixed this bad presentation”Wednesday 27 March 13
    • Questions ? https://github.com/maxandersen/jbosstools-gitmigrationWednesday 27 March 13