Your SlideShare is downloading. ×
0
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Subversion on-the-fly replication
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Subversion on-the-fly replication

31,851

Published on

Published in: Technology
5 Comments
26 Likes
Statistics
Notes
  • Never mind that inotify-tools is also an choice, here's an example which works well for me:
    http://planet.admon.org/synchronize-subversion-repositories-with-inotify-tools/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • WANdisco Subversion MultiSite http://www.wandisco.com/subversion/multisite is far easier.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • The post-revprop-change script for the master is missing the copy-revprops command:

    /path/to/svnsync copy-revprops --source-username=svnsync --sync- svnsync username=svnsync http:///repos/proxy-sync $REV
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • a few minor points; exposing slave through apache, you'll want to sudo chown -R www-data:www-data /path/to/repos

    also, since you initiate the sync FROM the master server (ultimately in the post-commit), i find it's best to not force my sync-outs to use apache. that is:

    instead of:
    svnsync --sync-username syncuser --sync-password syncpassword --source-username specialsourceuser --source-password init http://slaveserver/svn-sync/reponame http://masterserver/svn/reponame

    i can simply push from file:
    svnsync --sync-username syncuser --sync-password syncpassword init http://slaveserver/svn-sync/reponame file:///path/to/repo

    [on a windows server that file path looks like file:///C:/whatever/ versus linux which is like file:///rootfolder/path]
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • If you set up the SVNMASTERURI on the slave prior to synching with the master you will probably run into problems.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
31,851
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
580
Comments
5
Likes
26
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Subversion on-the-fly replication Do it like the Apache Software Foundation Infrastructure Team Norman Maurer ApacheCon EU 2009 March 27, 2009
  • 2. Norman Maurer Member of the Apache Software Foundation  Apache Software Foundation Infrastructure Team  PMC Apache JAMES  Senior Unix System Engineer  norman@apache.org norman.maurer@heagmedianet.de http://www.heagmedianet.com http://myblog.kicks-ass.org 2
  • 3. Subversion – The Opensource Version Control System Keep track of your sourcecode 3
  • 4. What are the benefits of using Subversion Opensource  Runs many OS  Flexible plugin mechanism via hooks  Very good integration with IDEs  Many users / Well tested  4
  • 5. Replication Replicate Repositories on as many servers as you like 5
  • 6. Benefits when using Subversion replication Copy of the data on more than one server  Eliminate single point of failure  Share load across servers  Repositories on many different GEO locations  Works with the usual load balancers in front  READ / WRITE possible  6
  • 7. Drawbacks of using Subversion replication When the primary node fails, no commits are  possible, but read-only still works ;) Big commits can take some time to replay to  slaves Importing dumps will mean the slaves become  out of sync! More servers to maintain  7
  • 8. How it works – The concept Replication Commit Subversion svn commit pass- Slave through Users Subversion Master svn update Replay changes svn commit svn update Users 8
  • 9. Putting stuff together Installing all the needed bits 9
  • 10. Installation of Apache HTTP Ubuntu / Debian  # apt-get install apache2 Freebsd  # portinstall apache22 Others  Check if there are prebuilt packages / build it on your own! Make sure you compile apache2 with mod_proxy support included. 10
  • 11. Installation of Subversion and Apache HTTP module Ubuntu / Debian  # apt-get install subversion libapache2-svn Freebsd  # portinstall -m “-DMOD_DAV_SVN“ subversion Others  Check if there are prebuilt packages / build it yourself from source 11
  • 12. Create the Subversion Repository Create Parent Directory  # mkdir /path/to/repos-parent # chown $wwwuser:$wwwgroup -R /path/to/repos-parent # chmod 770 -R /path/to/repos-parent Create Repository  # su $wwwuser -c 'svnadmin create /path/to/repository' 12
  • 13. Setup the Master Version-Control for all your data 13
  • 14. Configuration of Apache HTTP on the Master Setup Apache HTTPD config  Make sure # /path/to/apache/httpd.conf LoadModule dav_svn_module modules/mod_dav_svn.so Subversion ....... module is loaded <Location /repos> on startup DAV svn SVNPath /path/to/repos AuthName quot;SVN Reposquot; Subversion AuthType Basic Repository AuthBasicProvider file AuthUserFile /path/to/auth-file Require valid-user </Location> 14
  • 15. Setup the Slave Mirror of your Main Subversion Repository 15
  • 16. Configuration of Apache HTTPD on the Slave Modify Apache HTTPD config  # /path/to/apache/httpd.conf Make sure LoadModule dav_svn_module modules/mod_dav_svn.so Subversion ....... <Location /repos> module is loaded DAV svn on startup SVNPath /path/to/repos SVNMasterURI http://<MASTER>/repos AuthName quot;SVN Reposquot; AuthType Basic Subversion Master AuthBasicProvider file Repository AuthUserFile /path/to/auth-file Require valid-user </Location> <Location /repos/proxy-sync> DAV svn Subversion Sync SVNPath /path/to/repos Order deny,allow Location Deny from all Allow from <MASTER> </Location> 16
  • 17. Add Subversion hooks Add hook on Slave  #/!bin/sh #/path/to/repos/hooks/pre-revprop-change Limit revprop USER=$3 changes to the if [ quot;$USERquot; != quot;svnsyncquot; ]; then svnsync user echo >&2 quot;Only the svnsync user is allowed to change revpropsquot; exit 1 fi exit 0 17
  • 18. Set the correct permissions and group Make sure the hooks are owned by the right  group slave# chgrp $wwwgroup /path/to/repos/hooks/pre- revprop-change Get sure the hooks are executable  slave# chmod 750 /path/to/repos/hooks/pre-revprop- change 18
  • 19. Final steps Last steps to complete replication 19
  • 20. Init the Slave Repository Init the Slave Repository to set properties  master# svnsync --source-username=svnsync --sync- username=svnsync init http://<SLAVE>/repos http://<MASTER>/repos Check if this was successful  master# svn propget svn:sync-from-url --revprop -r 0 http://<SLAVE>/repos This should return the url to the master repository 20
  • 21. First replication to the Slave Repository First sync should probably be started in a  “screen“ session, because it will take some time master# screen Start the initial sync by hand  master# svnsync --source-username=svnsync --sync- username=svnsync sync http://<SLAVE>/repos Sync started! 21
  • 22. Add Subversion hooks to the Master for on-the-fly replication Add hooks needed for replication  #/!bin/sh Sync revprop #/path/to/repos/hooks/post-revprop-change changes to Slave via REV=$2 /path/to/svnsync --source-username=svnsync --sync- svnsync username=svnsync http://<SLAVE>/repos/proxy-sync $REV 2>>/var/log/svnsynclog & #!/bin/sh Sync commits to #/path/to/repos/hooks/post-commit Slave via svnsync /path/to/svnsync --source-username=svnsync --sync- username=svnsync sync http://<SLAVE>/repos/proxy-sync 2>>/var/log/svnsynclog & 22
  • 23. Set the correct permissions and group Make sure the hooks are owned by the right  group master# chgrp $wwwgroup /path/to/repos/hooks/post- revprop-change master# chgrp $wwwgroup /path/to/repos/hooks/post- commit Get sure the hooks are executable  master# chmod 750 /path/to/repos/hooks/post-revprop- change master# chmod 750 /path/to/repos/hooks/post-commit 23
  • 24. Subversion Replication in Production Our setup within the Apache Software Foundation 24
  • 25. Details about the setup – Part I Master / Slave  OS: FreeBSD-7.0 Filesystem: ZFS (raidz2) Repository Size: ~65GB Apache HTTPD: 2.2.11 Subversion: 1.5.5 Master  3500000 – 4500000 Hits/day: SVN-Operations/day: 200000 - 250000 Slave  70000 – 130000 Hits/day: SVN-Operations/day: 9000 - 90000 25
  • 26. Details about the setup – Part II Different GEO Locations  Master: US (OSUOSL) Slave: EU (SARA) 26
  • 27. Known problems Problems we saw at Apache Software Foundation 27
  • 28. Problems with file locking – Part I File locking will not work well enough with  svnsync Use this locking utility written by Joe Schaefer See „Useful resources“ for download link 28
  • 29. Problems with file locking – Part II Svnsync died but the lock file is still present  Delete lock file and wait for the resync Make sure master# ps aux | grep svnsync svnsync is not running master# svn proplist -- username=$syncuser --revprop -r0 http://$slave/repos/proxy-sync/ | grep Check for svn:sync-lock lock master# svn propdel svn:sync-lock -- username=$syncuser --revprop -r0 http://$slave/repos/proxy-sync/ Delete the lock 29
  • 30. Out-of-sync on big commits / imports Commit to slave server is not possible due to  the out-of-sync of revisions Nothing can really help here, but here are some suggestions: Schedule big imports to low-traffic time  Notify your users about the “downtime“ if  neccessary Poke the Subversion Development Team to  better handle such situations in the next release=P 30
  • 31. Other useful stuff Random stuff you maybe find useful, just as we did! 31
  • 32. Monitoring Monitor the sync-state of your repositories with  Nagios Send notification if the repositories are out-of- sync! SVN=/usr/bin/svn GREP=/bin/grep CUT=/usr/bin/cut MASTER_SVN=$1 SLACK=10 See „Useful SLAVE_ERROR= resources“ for # check if needed args are given download link if [ $# -gt 1 ] then # get the revision of the master repos MASTER_REV=`$SVN info $MASTER_SVN | grep -i ^revision | cut -f2 -d ' '` ..... 32
  • 33. Useful resources Apache HTTPD  http://httpd.apache.org Subversion  http://subversion.tigris.org Better Locking script  http://people.apache.org/~norman/talks/setlock.pl Nagios monitoring script  http://people.apache.org/~norman/talks/check_svn_replication.sh 33
  • 34. Questions at all? 34
  • 35. Some last words.... Listen carefully 35
  • 36. Some last words... Please use the closest Subversion Server to help us.. lower the load on svn.apache.org  split the bandwidth  improve our setup  36
  • 37. Thank you for your attention! 37

×