• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Subversion on-the-fly replication
 

Subversion on-the-fly replication

on

  • 36,234 views

 

Statistics

Views

Total Views
36,234
Views on SlideShare
36,058
Embed Views
176

Actions

Likes
24
Downloads
543
Comments
5

2 Embeds 176

http://www.slideshare.net 166
http://www.linkedin.com 10

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

15 of 5 previous next Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Never mind that inotify-tools is also an choice, here's an example which works well for me:
    http://planet.admon.org/synchronize-subversion-repositories-with-inotify-tools/
    Are you sure you want to
    Your message goes here
    Processing…
  • WANdisco Subversion MultiSite http://www.wandisco.com/subversion/multisite is far easier.
    Are you sure you want to
    Your message goes here
    Processing…
  • The post-revprop-change script for the master is missing the copy-revprops command:

    /path/to/svnsync copy-revprops --source-username=svnsync --sync- svnsync username=svnsync http:///repos/proxy-sync $REV
    Are you sure you want to
    Your message goes here
    Processing…
  • a few minor points; exposing slave through apache, you'll want to sudo chown -R www-data:www-data /path/to/repos

    also, since you initiate the sync FROM the master server (ultimately in the post-commit), i find it's best to not force my sync-outs to use apache. that is:

    instead of:
    svnsync --sync-username syncuser --sync-password syncpassword --source-username specialsourceuser --source-password init http://slaveserver/svn-sync/reponame http://masterserver/svn/reponame

    i can simply push from file:
    svnsync --sync-username syncuser --sync-password syncpassword init http://slaveserver/svn-sync/reponame file:///path/to/repo

    [on a windows server that file path looks like file:///C:/whatever/ versus linux which is like file:///rootfolder/path]
    Are you sure you want to
    Your message goes here
    Processing…
  • If you set up the SVNMASTERURI on the slave prior to synching with the master you will probably run into problems.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Subversion on-the-fly replication Subversion on-the-fly replication Presentation Transcript

    • Subversion on-the-fly replication Do it like the Apache Software Foundation Infrastructure Team Norman Maurer ApacheCon EU 2009 March 27, 2009
    • Norman Maurer Member of the Apache Software Foundation  Apache Software Foundation Infrastructure Team  PMC Apache JAMES  Senior Unix System Engineer  norman@apache.org norman.maurer@heagmedianet.de http://www.heagmedianet.com http://myblog.kicks-ass.org 2
    • Subversion – The Opensource Version Control System Keep track of your sourcecode 3
    • What are the benefits of using Subversion Opensource  Runs many OS  Flexible plugin mechanism via hooks  Very good integration with IDEs  Many users / Well tested  4
    • Replication Replicate Repositories on as many servers as you like 5
    • Benefits when using Subversion replication Copy of the data on more than one server  Eliminate single point of failure  Share load across servers  Repositories on many different GEO locations  Works with the usual load balancers in front  READ / WRITE possible  6
    • Drawbacks of using Subversion replication When the primary node fails, no commits are  possible, but read-only still works ;) Big commits can take some time to replay to  slaves Importing dumps will mean the slaves become  out of sync! More servers to maintain  7
    • How it works – The concept Replication Commit Subversion svn commit pass- Slave through Users Subversion Master svn update Replay changes svn commit svn update Users 8
    • Putting stuff together Installing all the needed bits 9
    • Installation of Apache HTTP Ubuntu / Debian  # apt-get install apache2 Freebsd  # portinstall apache22 Others  Check if there are prebuilt packages / build it on your own! Make sure you compile apache2 with mod_proxy support included. 10
    • Installation of Subversion and Apache HTTP module Ubuntu / Debian  # apt-get install subversion libapache2-svn Freebsd  # portinstall -m “-DMOD_DAV_SVN“ subversion Others  Check if there are prebuilt packages / build it yourself from source 11
    • Create the Subversion Repository Create Parent Directory  # mkdir /path/to/repos-parent # chown $wwwuser:$wwwgroup -R /path/to/repos-parent # chmod 770 -R /path/to/repos-parent Create Repository  # su $wwwuser -c 'svnadmin create /path/to/repository' 12
    • Setup the Master Version-Control for all your data 13
    • Configuration of Apache HTTP on the Master Setup Apache HTTPD config  Make sure # /path/to/apache/httpd.conf LoadModule dav_svn_module modules/mod_dav_svn.so Subversion ....... module is loaded <Location /repos> on startup DAV svn SVNPath /path/to/repos AuthName quot;SVN Reposquot; Subversion AuthType Basic Repository AuthBasicProvider file AuthUserFile /path/to/auth-file Require valid-user </Location> 14
    • Setup the Slave Mirror of your Main Subversion Repository 15
    • Configuration of Apache HTTPD on the Slave Modify Apache HTTPD config  # /path/to/apache/httpd.conf Make sure LoadModule dav_svn_module modules/mod_dav_svn.so Subversion ....... <Location /repos> module is loaded DAV svn on startup SVNPath /path/to/repos SVNMasterURI http://<MASTER>/repos AuthName quot;SVN Reposquot; AuthType Basic Subversion Master AuthBasicProvider file Repository AuthUserFile /path/to/auth-file Require valid-user </Location> <Location /repos/proxy-sync> DAV svn Subversion Sync SVNPath /path/to/repos Order deny,allow Location Deny from all Allow from <MASTER> </Location> 16
    • Add Subversion hooks Add hook on Slave  #/!bin/sh #/path/to/repos/hooks/pre-revprop-change Limit revprop USER=$3 changes to the if [ quot;$USERquot; != quot;svnsyncquot; ]; then svnsync user echo >&2 quot;Only the svnsync user is allowed to change revpropsquot; exit 1 fi exit 0 17
    • Set the correct permissions and group Make sure the hooks are owned by the right  group slave# chgrp $wwwgroup /path/to/repos/hooks/pre- revprop-change Get sure the hooks are executable  slave# chmod 750 /path/to/repos/hooks/pre-revprop- change 18
    • Final steps Last steps to complete replication 19
    • Init the Slave Repository Init the Slave Repository to set properties  master# svnsync --source-username=svnsync --sync- username=svnsync init http://<SLAVE>/repos http://<MASTER>/repos Check if this was successful  master# svn propget svn:sync-from-url --revprop -r 0 http://<SLAVE>/repos This should return the url to the master repository 20
    • First replication to the Slave Repository First sync should probably be started in a  “screen“ session, because it will take some time master# screen Start the initial sync by hand  master# svnsync --source-username=svnsync --sync- username=svnsync sync http://<SLAVE>/repos Sync started! 21
    • Add Subversion hooks to the Master for on-the-fly replication Add hooks needed for replication  #/!bin/sh Sync revprop #/path/to/repos/hooks/post-revprop-change changes to Slave via REV=$2 /path/to/svnsync --source-username=svnsync --sync- svnsync username=svnsync http://<SLAVE>/repos/proxy-sync $REV 2>>/var/log/svnsynclog & #!/bin/sh Sync commits to #/path/to/repos/hooks/post-commit Slave via svnsync /path/to/svnsync --source-username=svnsync --sync- username=svnsync sync http://<SLAVE>/repos/proxy-sync 2>>/var/log/svnsynclog & 22
    • Set the correct permissions and group Make sure the hooks are owned by the right  group master# chgrp $wwwgroup /path/to/repos/hooks/post- revprop-change master# chgrp $wwwgroup /path/to/repos/hooks/post- commit Get sure the hooks are executable  master# chmod 750 /path/to/repos/hooks/post-revprop- change master# chmod 750 /path/to/repos/hooks/post-commit 23
    • Subversion Replication in Production Our setup within the Apache Software Foundation 24
    • Details about the setup – Part I Master / Slave  OS: FreeBSD-7.0 Filesystem: ZFS (raidz2) Repository Size: ~65GB Apache HTTPD: 2.2.11 Subversion: 1.5.5 Master  3500000 – 4500000 Hits/day: SVN-Operations/day: 200000 - 250000 Slave  70000 – 130000 Hits/day: SVN-Operations/day: 9000 - 90000 25
    • Details about the setup – Part II Different GEO Locations  Master: US (OSUOSL) Slave: EU (SARA) 26
    • Known problems Problems we saw at Apache Software Foundation 27
    • Problems with file locking – Part I File locking will not work well enough with  svnsync Use this locking utility written by Joe Schaefer See „Useful resources“ for download link 28
    • Problems with file locking – Part II Svnsync died but the lock file is still present  Delete lock file and wait for the resync Make sure master# ps aux | grep svnsync svnsync is not running master# svn proplist -- username=$syncuser --revprop -r0 http://$slave/repos/proxy-sync/ | grep Check for svn:sync-lock lock master# svn propdel svn:sync-lock -- username=$syncuser --revprop -r0 http://$slave/repos/proxy-sync/ Delete the lock 29
    • Out-of-sync on big commits / imports Commit to slave server is not possible due to  the out-of-sync of revisions Nothing can really help here, but here are some suggestions: Schedule big imports to low-traffic time  Notify your users about the “downtime“ if  neccessary Poke the Subversion Development Team to  better handle such situations in the next release=P 30
    • Other useful stuff Random stuff you maybe find useful, just as we did! 31
    • Monitoring Monitor the sync-state of your repositories with  Nagios Send notification if the repositories are out-of- sync! SVN=/usr/bin/svn GREP=/bin/grep CUT=/usr/bin/cut MASTER_SVN=$1 SLACK=10 See „Useful SLAVE_ERROR= resources“ for # check if needed args are given download link if [ $# -gt 1 ] then # get the revision of the master repos MASTER_REV=`$SVN info $MASTER_SVN | grep -i ^revision | cut -f2 -d ' '` ..... 32
    • Useful resources Apache HTTPD  http://httpd.apache.org Subversion  http://subversion.tigris.org Better Locking script  http://people.apache.org/~norman/talks/setlock.pl Nagios monitoring script  http://people.apache.org/~norman/talks/check_svn_replication.sh 33
    • Questions at all? 34
    • Some last words.... Listen carefully 35
    • Some last words... Please use the closest Subversion Server to help us.. lower the load on svn.apache.org  split the bandwidth  improve our setup  36
    • Thank you for your attention! 37