Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ResourceSync: Leveraging Sitemaps for Resource Synchronization


Published on

Published in: Technology, News & Politics
  • Be the first to comment

ResourceSync: Leveraging Sitemaps for Resource Synchronization

  1. 1. ResourceSync: Leveraging Sitemapsfor Resource SynchronizationWWW 2013, Rio de Janeiro, May 17thBernhard Haslhofer | University ofViennaSimeon Warner | Cornell UniversityCarl Lagoze | University of MichiganMartin Klein, Robert Sanderson | Los Alamos National LabsMichael L. Nelson | Old Dominion UniversityHerbert van de Sompel | Los Alamos National Labs
  2. 2. WWW 2013, May 17thResourceSync• What and Why?• Synchronization Scenarios• ResourceSync Basics• Demos• Status and Next Steps2
  3. 3. WWW 2013, May 17thWhat?• A framework for synchronizing Webresources from a Source to a Destination3Websync$ resync
  4. 4. WWW 2013, May 17thWhy?• rsync: filesystem sync, but not Web• OAI-PMH: metadata, but not resources• Web-DAV: extends HTTP, requires serverinstallation at source• ...4… because lots of projects and services are doingsynchronization but rely on ad-hoc solutions!
  5. 5. WWW 2013, May 17thResourceSync• What and Why?• Synchronization Scenarios• ResourceSync Basics• Demos• Status and Next Steps5
  6. 6. WWW 2013, May mirroring• 2.4M resources (PDF,metadata, Latex src)• ~800/day created orupdated• uses homebrewmirroring since 1994 (!)• look for more generalsolution to supportindependent destinations6
  7. 7. WWW 2013, May 17thWikipedia• 1.4 updates / sec• many dependentservices reusingWikipedia content (e.g.,DBPedia, Freebase, etc.)• harvest articles via OAI-PMH, retrieve changesvia IRC, downloaddumps7
  8. 8. WWW 2013, May• aggregates metadatafrom >200 dataproviders in Europe• 10 largest providerscontribute 80%• >190 providerscontribute 20%8
  9. 9. WWW 2013, May 17thDesign Guidelines• Sync small websites / repositories (fewresources) but also large data collections(millions of resources)• Support low change frequency (weeks /months) to high change frequency(seconds) sources• Low adoption barrier!9
  10. 10. WWW 2013, May 17thResourceSync• What and Why?• Synchronization Scenarios• ResourceSync Basics• Demos• Status and Next Steps10
  11. 11. WWW 2013, May 17thResource List11DestinationSource<?xml version="1.0" encoding="UTF-8"?><urlset xmlns=""xmlns:rs=""><rs:md capability="resourcelist"modified="2013-01-03T09:00:00Z"/><url><loc></loc></url><url><loc></loc></url></urlset>$ resync -b http://example.comXML Sitemap
  12. 12. WWW 2013, May 17thResource List12<?xml version="1.0" encoding="UTF-8"?><urlset xmlns=""xmlns:rs=""><rs:md capability="resourcelist"modified="2013-01-03T09:00:00Z"/><url><loc></loc><lastmod>2013-01-02T13:00:00Z</lastmod><rs:md hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"/></url><url><loc></loc><lastmod>2013-01-02T14:00:00Z</lastmod><rs:md hash="md5:1e0d5cb8ef6ba40c99b14c0237be735e"/></url></urlset>Source
  13. 13. WWW 2013, May 17thChange List13DestinationSource<?xml version="1.0" encoding="UTF-8"?><urlset xmlns=""xmlns:rs=""><rs:md capability="changelist"modified="2013-01-03T11:00:00Z"/><url><loc></loc><lastmod>2013-01-02T13:00:00Z</lastmod><rs:md change="updated"/></url><url><loc></loc><lastmod>2013-01-02T18:00:00Z</lastmod><rs:md change="deleted"/></url></urlset>$ resync -b$ resync -i http://example.comXML Sitemap
  14. 14. WWW 2013, May 17thResource Dump14Source<?xml version="1.0" encoding="UTF-8"?><urlset xmlns=""xmlns:rs=""><rs:md capability="resourcedump"modified="2013-01-03T09:00:00Z"/><url><loc></loc><lastmod>2013-01-03T09:00:00Z</lastmod></url></urlset>XML Sitemap
  15. 15. WWW 2013, May 17thResource Dump15|- manifest.xml|- resources|- res1|- res2
  16. 16. WWW 2013, May 17thResource Dump Manifest16<?xml version="1.0" encoding="UTF-8"?><urlset xmlns=""xmlns:rs=""><rs:md capability="resourcedump-manifest"modified="2013-01-03T09:00:00Z"/><url><loc></loc><lastmod>2013-01-03T03:00:00Z</lastmod><rs:md hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"path="/resources/res1"/></url><url><loc></loc><lastmod>2013-01-03T04:00:00Z</lastmod><rs:md hash="md5:1e0d5cb8ef6ba40c99b14c0237be735e"path="/resources/res2"/></url></urlset>manifest.xml (XML Sitemap)
  17. 17. WWW 2013, May 17thCapability List17DestinationSource<?xml version="1.0" encoding="UTF-8"?><urlset xmlns=""xmlns:rs=""><rs:ln href=""rel="describedby"type="application/xml"/><rs:md capability="capabilitylist"modified="2013-01-02T14:00:00Z"/><url><loc></loc><rs:md capability="resourcelist"/></url><url><loc></loc><rs:md capability="resourcedump"/></url><url><loc></loc><rs:md capability="changelist"/></url></urlset>$ resync -x http://example.comXML Sitemap
  18. 18. WWW 2013, May 17thLarge Resource Lists18<?xml version="1.0" encoding="UTF-8"?><sitemapindex xmlns=""xmlns:rs=""><rs:md capability="resourcelist"modified="2013-01-03T09:00:00Z"/><sitemap><loc></loc><lastmod>2013-01-03T09:00:00Z</lastmod></sitemap><sitemap><loc></loc><lastmod>2013-01-03T09:00:00Z</lastmod></sitemap></sitemapindex>Source
  19. 19. WWW 2013, May 17thOther Capabilities
  20. 20. WWW 2013, May 17thResourceSync• What and Why?• Synchronization Scenarios• ResourceSync Basics Walkthrough• Demos• Status and Next Steps20
  21. 21. WWW 2013, May 17thAvailable code• ResourceSync clientand library (Python)• ResourceSync sourcesimulator21
  22. 22. WWW 2013, May 17thInstall resync client/library22$ git clone git://$ cd resync/$ python build$ sudo python install$ sudo easy_install resync$ sudo pip install resyncoror
  23. 23. WWW 2013, May 17thInstall resync simulator23$ git clone git://$ cd simulator/$ chmod u+x simulate-source$ ./simulate-source$ sudo easy_install tornado
  24. 24. WWW 2013, May 17thRun client against simulator24$ resync -b http://localhost:8888$ resync -i http://localhost:8888
  25. 25. WWW 2013, May 17thresync @ arxiv.org25resync -v --noauth
  26. 26. WWW 2013, May 17thresync @ en.wikipedia.org26
  27. 27. WWW 2013, May 17thResourceSync• What and Why?• Synchronization Scenarios• ResourceSync Basics Walkthrough• Demos• Status and Next Steps27
  28. 28. WWW 2013, May 17thStatus• Beta spec (v.0.6) for public comment• Tool development started• Separate documents for archiving and pushdeployments28
  29. 29. WWW 2013, May 17thNext Steps• Continue tool development & deployment• Collect• public comments• implementation issues on• Version 0.9 to be released in Summer 2013• Version 1.0 in fall 2013 (NISO standard)29
  30. 30. WWW 2013, May 17thThanks!@bhaslhofer