sitemap4rdf is a command line tool that generates Sitemap files by querying a SPARQL endpoint for URIs and listing them. It allows search engines to discover Linked Data resources. It is open source and generates valid Sitemaps that can be submitted to search engines like Sindice and Google. Future work includes integrating it with Linked Data publishing tools and supporting more Sitemap extensions.

1. sitemap4rdf
generate Sitemap files from a SPARQL
endpoint
http://www.deri.ie/
http://www deri ie/
Boris Villazón-Terrazas and Richard Cyganiak (DERI)
Facultad de Informática, Universidad Politécnica de Madrid
Campus de Montegancedo sn 28660 Boadilla del Monte Madrid
sn, Monte,
http://www.oeg-upm.net
Phone: 34.91.3366605, Fax: 34.91.3524819

2. ToC
• Publishing Linked Data from a triple store
• Search engines
• The Sitemap protocol
• sitemap4rdf
• Summary
S
• Future work
2

3. Linked Data frontends for triple stores
Source: Pubby website, http://www4.wiwiss.fu-berlin.de/pubby/
3

4. ToC
• Publishing Linked Data from a triple store
• Search engines
• The Sitemap protocol
• sitemap4rdf
• Summary
S
• Future work
4

5. Sindice: the best RDF search engine
5

6. Sindice: the best RDF search engine
• 120M+ documents
• Continuously updating since 2006
C ti l d ti i
• Search API
• RDF/XML, Turtle, RDFa, microformats
6

7. ToC
• Publishing Linked Data from a triple store
• Search engines
• The Sitemap protocol
• sitemap4rdf
• Summary
S
• Future work
7

8. Sitemap Protocol
• Used by web crawlers
• Efficiently find all your content & discover
what has been updated
http://sitemaps.org/
A sitemap fil contains i f
i file i information regarding one or more URL on
i di URLs
your Web site. The information that is stored there helps search
engines better spider your website.
8

9. Sitemap Protocol: Simple example
<?xml version="1.0" encoding="UTF-8"?>
<urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://yoursite/</loc>
</url>
<url>
oc ttp://you s te/p oducts/535 6 / oc
<loc>http://yoursite/products/53546</loc>
</url>
<url>
<loc>http://yoursite/products/98421</loc>
</url>
<url>
<loc>http://yoursite/products/41003</loc>
</url>
</urlset>
9

10. Sitemap Protocol: Optional parts
<?xml version="1.0" encoding="UTF-8"?>
<urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://yoursite/</loc>
<lastmod>2010-06-24</lastmod>
<changefreq>daily</changefreq>
< h f >d il </ h f >
</url>
</urlset>
10

11. Sitemap Protocol: Huge sitemaps
• Gzip-compress your sitemap
• Limit: 50k URLs or 10MB
• split into multiple sitemap files
• add a sitemap index file
11

12. Sitemap Protocol: Discovery
• Publish the sitemap file
• Add a line to http://yoursite/robots.txt
• Web site owners use the /robots.txt file to give instructions about their site
to web robots; this is called The Robots Exclusion Protocol.
Sitemap: http://yoursite/sitemap.xml
12

13. ToC
• Publishing Linked Data from a triple store
• Search engines
• The Sitemap protocol
• sitemap4rdf
• Summary
S
• Future work
13

14. sitemap4rdf
• Simple command line tool
• Sends a SPARQL query to list all URIs
• Generates sitemap
sitemap4rdf htt //
it 4 df http://yoursite/sparql htt //
it / l http://yoursite/resource/
it / /
Example:
sitemap4rdf http://geo.linkeddata.es/sparql http://geo.linkeddata.es/
• run sitemap4rdf specifying th SPARQL endpoint
it 4 df if i the d i t
and the prefix of the URLs to include in the Sitemap
14

15. Submit the sitemap location - Sindice
• http://sindice.com/main/submit
15

16. Submit the sitemap location - Google
• https://www.google.com/webmasters/tools/
16

17. ToC
• Publishing Linked Data from a triple store
• Search engines
• The Sitemap protocol
• sitemap4rdf
• Summary
S
• Future work
17

18. Summary
• Sitemap protocol informs search engines about
available pages
• Supported by Sindice!
• sitemap4rdf generates Sitemap files by listing URIs
in a SPARQL endpoint
• Open source, Java
• http://lab.linkeddata.deri.ie/2010/sitemap4rdf/
• http://mccarthy dia fi upm es/sitemap4rdf/
http://mccarthy.dia.fi.upm.es/sitemap4rdf/
• http://www.oeg-upm.net/index.php/en/downloads/122-sitemap4rdf
18

19. ToC
• Publishing Linked Data from a triple store
• Search engines
• The Sitemap protocol
• sitemap4rdf
• Summary
S
• Future work
19

20. Future Work
• Integrate sitemap4rdf with Pubby
• Generate voiD file automatically from a SPARQL
endpoint
• Generate an entry in CKAN (registry of open
knowledge packages) automatically through CKAN-
API
• http://ckan net/package/geolinkeddata
http://ckan.net/package/geolinkeddata
• Interact with prefix cc ( service for remembering and
prefix.cc
looking up URI prefixes) through its API
• geoes: < http://geo.linkeddata.es/ontology>
http://geo.linkeddata.es/ontology
20

21. Future Work
• Support the semantic sitemap extension (when it will
be compatible with google)
• http://sw.deri.org/2007/07/sitemapextension/
21

22. sitemap4rdf
generate Sitemap files from a SPARQL
endpoint
http://www.deri.ie/
http://www deri ie/
Boris Villazón-Terrazas and Richard Cyganiak (DERI)
Facultad de Informática, Universidad Politécnica de Madrid
Campus de Montegancedo sn 28660 Boadilla del Monte Madrid
sn, Monte,
http://www.oeg-upm.net
Phone: 34.91.3366605, Fax: 34.91.3524819