Introduction on how to use open data and Python, with examples of RDFLib, SuRF and RDF-Alchemy. http://softwarelivre.org/fisl13

1. Linking the world with Python and Semantics
@tati_alchueyr (Globo.com)
25th July 2012, FISL 13

2. how do you store your data?

3. how do you store your data?
[ ] data... what data?!
[ ] raw files (csv, json, xml)
[ ] database (eg. Relational Data Base)
[ ] graphs (eg. Resource Description Framework)
[ ] other...

4. how do you search for...?
Apartments near English-Portuguese bilingual
childcare in Rio de Janeiro state.
ERP service providers with offices in São Paulo
and New York.
Researchers working on artificial intelligence in
Southeast of Brazil.
GNU GPL software for image processing
developed from 2009 to 2010 authored also by
Brazilian developers

5. how do you search for...?
Apartments near English-Portuguese bilingual
childcare in Rio de Janeiro state.
ERP service providers with offices in São Paulo
and New York.
Researchers working on artificial intelligence in
Southeast of Brazil.
GNU GPL software for image processing
developed from 2009 to 2010 authored also by
Brazilian developers

6. how do you search for...?
Apartments near English-Portuguese bilingual
childcare in Rio de Janeiro state.
ERP service providers with offices in São Paulo
and New York.
Researchers working on artificial intelligence in
Southeast of Brazil.
GNU GPL software for image processing
developed from 2009 to 2010 authored also by
Brazilian developers

7. how do you search for...?
Apartments near English-Portuguese bilingual
childcare in Rio de Janeiro state.
ERP service providers with offices in São Paulo
and New York.
Researchers working on artificial intelligence in
Southeast of Brazil.
GNU GPL software for image processing
developed from 2009 to 2010 authored also by
Brazilian developers

8. what ^ have in common?

9. linked open data in 2007

10. linked open data in 2008

11. linked open data in 2009

12. linked open data in 2011

13. traditional RDMS

14. linked data graph

15. linked data modelling

16. modelling

17. modelling

18. quering RDB
select bookID, authorName
from books, authors
where books.aid = authors.aid
and books.isbn = ‘006251587X’.

19. quering RDF
select ?authName ?authEmail
where {
<amazon:book#006251587X> <amazon:hasAuthor>
<foaf:name#TimBerners-Lee>
<foaf:name#TimBerners-Lee> <foaf:name> ?
authName
<foaf:name#TimBerners-Lee> <foaf:email>?
authEmail
}

20. globo.com developers before using
web semantics

21. globo.com developers while learning
web semantics
(?w ?t ?f)

22. globo.com developers after using
web semantics

23. Sample hard to test code

24. approach 1
# queries isolation

26. approach 2
# data as object
DAO

28. Y U NO make
SPARQL queries?!

29. Y U NO make
data access easy?!

30. Y U NO make
things testable?!

31. product developers evaluating
web semantics

33. fact 1: we don't have an
out-of-box solution

34. fact 2: but we do have
some options

35. some options
#1: create a solution
from scratch
#2: study existing
solutions and then
[ ] contribute to them
[ ] develop on top of
them
[ ] goto #1

36. the final decision is not only ours

37. but we chose starting from #2
#2: study existing solutions and then (...)

38. ok, lmgfy

39. a few results from google
ActiveRDF PyRdfa
active-semantic pysparql
Django4Store RDFAlchemy
Django-RDF
RdfLib
Django-RDFAlchemy
Redland
Djubby
semantic-django
EasyRDF
SPARQLWrapper
Jena
FuXi Sparrow
Oort Sparta
Pymantic SuRF

40. click to know more
ActiveRDF PyRdfa
active-semantic pysparql
Django4Store RDFAlchemy
Django-RDF
RdfLib
Django-RDFAlchemy
Redland
Djubby
semantic-django
EasyRDF
SPARQLWrapper
Jena
FuXi Sparrow
Oort Sparta
Pymantic SuRF

41. {?project :by_author ?author .
?author :works_at :globocom . }
ActiveRDF PyRdfa
active-semantic pysparql
Django4Store RDFAlchemy
Django-RDF
RdfLib
Django-RDFAlchemy
Redland
Djubby
semantic-django
EasyRDF
SPARQLWrapper
Jena
FuXi Sparrow
Oort Sparta
Pymantic SuRF

42. {?project :use_language :python . }
ActiveRDF PyRdfa
active-semantic pysparql
Django4Store RDFAlchemy
Django-RDF
RdfLib
Django-RDFAlchemy
Redland
Djubby
semantic-django
EasyRDF
SPARQLWrapper
Jena
FuXi Sparrow
Oort Sparta
Pymantic SuRF

43. {?project :use_language :python ;
:last_commit ?commit .
FILTER (?commit >= "2011-12-01"^^xsd:date) }
ActiveRDF PyRdfa
active-semantic pysparql
Django4Store RDFAlchemy
Django-RDF
RdfLib
Django-RDFAlchemy
Redland
Djubby
semantic-django
EasyRDF
SPARQLWrapper
Jena
FuXi Sparrow
Oort Sparta
Pymantic SuRF

44. relation between these tools

45. team filtering
ActiveRDF PyRdfa
active-semantic pysparql
Django4Store RDFAlchemy
Django-RDF
RdfLib
Django-RDFAlchemy
Redland
Djubby
semantic-django
EasyRDF
SPARQLWrapper
Jena
FuXi Sparrow
Oort Sparta
Pymantic SuRF

46. SPARQLWrapper
problem: list all predicates of a class
# List all predicates of dbonto:Band
query = """
SELECT distinct ?subject
FROM <http://dbpedia.org>
{
?subject rdfs:domain ?object .
<http://dbpedia.org/ontology/Band> rdfs:subClassOf ?object
OPTION (TRANSITIVE, t_distinct, t_step('step_no') as ?n, t_min
(0) ).
}""" http://live.dbpedia.org/sparql
sparql = SPARQLWrapper("http://dbpedia.org/sparql")
sparql.setQuery(query)
sparql.setReturnFormat(JSON)
results = sparql.query().convert()
for result in results["results"]["bindings"]:
print(result["subject"]["value"])

47. SPARQLWrapper
abstract endpoint returns dict
# List all predicates of dbonto:Band
query = """
SELECT distinct ?subject
FROM <http://dbpedia.org>
{
?subject rdfs:domain ?object .
<http://dbpedia.org/ontology/Band> rdfs:subClassOf ?object
OPTION (TRANSITIVE, t_distinct, t_step('step_no') as ?n, t_min
(0) ).
}""" http://live.dbpedia.org/sparql
sparql = SPARQLWrapper("http://dbpedia.org/sparql")
sparql.setQuery(query)
sparql.setReturnFormat(JSON)
results = sparql.query().convert()
for result in results["results"]["bindings"]:
print(result["subject"]["value"])

48. SPARQLWrapper
Ok, not different from what we have...

49. SPARQLWrapper
just a wrapper around a SPARQL server
well tested ;)

50. SPARQLWrapper
problem: list all subjects given ?p ?o
from SPARQLWrapper import SPARQLWrapper, JSON
# List all instances (eg. bands) with genre Metal
query = """
PREFIX db: <http://dbpedia.org/resource/>
PREFIX dbonto: <http://dbpedia.org/ontology/>
SELECT DISTINCT ?who
FROM <http://dbpedia.org>
WHERE {
?who dbonto:genre db:Metal .
}
"""
sparql = SPARQLWrapper("http://dbpedia.org/sparql")
sparql.setQuery(query)
sparql.setReturnFormat(JSON)
results = sparql.query().convert()
for result in results["results"]["bindings"]:
print(result["who"]["value"])

51. RdfLib
problem: list all subjects given ?p ?o
import rdflib
import rdfextras.store.SPARQL
# SPARQL endpoint setup
endpoint = "http://dbpedia.org/sparql"
store = rdfextras.store.SPARQL.SPARQLStore(endpoint)
graph = rdflib.Graph(store)
# Definitions
genre = rdflib.URIRef("http://dbpedia.org/ontology/genre")
metal = rdflib.URIRef("http://dbpedia.org/resource/Metal")
# Query
for label in graph.subjects(genre, metal):
print label

52. RdfLib
abstract endpoint returns dict namespace
import rdflib
import rdfextras.store.SPARQL
# SPARQL endpoint setup
endpoint = "http://dbpedia.org/sparql"
store = rdfextras.store.SPARQL.SPARQLStore(endpoint)
graph = rdflib.Graph(store)
# Namespaces to clear up definitions
DBONTO = rdflib.Namespace("http://dbpedia.org/ontology/")
DB = rdflib.Namespace("http://dbpedia.org/resource/")
# Query
for label in graph.subjects(DBONTO.genre, DB.Metal):
print label

53. RdfLib
abstract endpoint returns dict namespace
import rdflib
import rdfextras.store.SPARQL
# SPARQL endpoint setup
endpoint = "http://dbpedia.org/sparql"
store = rdfextras.store.SPARQL.SPARQLStore(endpoint)
graph = rdflib.Graph(store)
# Namespaces to clear up definitions
DBONTO = rdflib.Namespace("http://dbpedia.org/ontology/")
DB = rdflib.Namespace("http://dbpedia.org/resource/")
# Query
for label in graph.subjects(DBONTO.genre, DB.Metal):
print label
subjects
predicates
objects
subject_predicates
subject_objects
predicates_objects

54. RdfLib
abstract endpoint returns dict namespace
import rdflib
import rdfextras.store.SPARQL
# SPARQL endpoint setup
endpoint = "http://dbpedia.org/sparql"
store = rdfextras.store.SPARQL.SPARQLStore(endpoint)
graph = rdflib.Graph(store)
# Namespaces to clear up definitions
DBONTO = rdflib.Namespace("http://dbpedia.org/ontology/")
DB = rdflib.Namespace("http://dbpedia.org/resource/")
# Using triples
for musician, _, _ in graph.triples((None, DBONTO.genre, DB.Metal)):
print musician

55. RdfLib
abstract endpoint returns dict namespace query by triples
import rdflib
import rdfextras.store.SPARQL
# SPARQL endpoint setup
endpoint = "http://dbpedia.org/sparql"
store = rdfextras.store.SPARQL.SPARQLStore(endpoint)
graph = rdflib.Graph(store)
# Namespaces to clear up definitions
DBONTO = rdflib.Namespace("http://dbpedia.org/ontology/")
DB = rdflib.Namespace("http://dbpedia.org/resource/")
# Query
for label in graph.subjects(DBONTO.genre, DB.Metal):
print label

56. RdfLib
abstract endpoint returns dict namespace query by triples add / remove
import rdflib
import rdfextras.store.SPARQL
# n3 fixture file
graph = rdflib.Graph()
graph.parse("fixture_genre_metal.nt", format="nt")
# Namespace
DBONTO = rdflib.Namespace("http://dbpedia.org/ontology/")
DB = rdflib.Namespace("http://dbpedia.org/resource/")
# Add nodes
graph.add((DB.AndrewsMedina, DBONTO.genre, DB.Metal))
graph.add((DB.Siminino, DBONTO.genre, DB.Metal))
graph.add((DB.Herman, DBONTO.genre, DB.Metal))
# Remove nodes
graph.remove((DB.AndrewsMedina, DBONTO.genre, DB.Metal))

57. RdfLib
concentrates on
providing the core
RDF types and
interfaces, through
plugin interface

58. RdfLib
makes testing
simple, allowing
fixtures using n3
files, add triples
and remove triples

59. RdfLib
but...
each triple query
requires a new
connection to
SPARQL

60. RdfLib
therefore
too many access to
SPARQL endpoint

61. RdfLib
and...
doesn't provide an
ORM (object
relational mapping)

62. SuRF
abstract endpoint returns dict namespace query by triples add / remove
from surf import Store, Session, ns, query
store = Store(reader='sparql_protocol',
endpoint='http://dbpedia.org/sparql')
session = Session(store, {})
session.enable_logging = False
ns.register(db='http://dbpedia.org/resource/')
ns.register(dbonto='http://dbpedia.org/ontology/')
MusicalArtist = session.get_class(ns.DB['MusicalArtist'])
artistas_metal = MusicalArtist.get_by(dbonto_genre=ns.DB["Metal"])
print artistas_metal
ORM

63. SuRF
problem: list all subjects given ?p ?o
from surf import Store, Session, ns, query
store = Store(reader='sparql_protocol',
endpoint='http://dbpedia.org/sparql')
session = Session(store, {})
ns.register(db='http://dbpedia.org/resource/')
ns.register(dbonto='http://dbpedia.org/ontology/')
query_surf = query.select("?who").distinct()
query_surf.where(("?who", ns.DBONTO.genre, ns.DB.Metal))
metal_bands = session.default_store.execute(query_surf)
for band in metal_bands:
print band
composed
ORM
queries

64. SuRF
various approaches
ORM
programaticaly

65. SuRF
simple ORM
no need to redeclare
TTL definitions

66. SuRF
“complex” queries
using
lazy evalutation

67. SuRF
documentation
&
community

68. SuRF
but...
no django-style
models

69. SuRF
verbose syntax

70. RDFAlchemy
problem: list all subjects given ?p ?o
from rdfalchemy.sparql import SPARQLGraph
from rdflib import Namespace
endpoint = "http://dbpedia.org/sparql"
graph = SPARQLGraph(endpoint)
DB = Namespace("http://dbpedia.org/resource/")
DBONTO = Namespace("http://dbpedia.org/ontology/")
metal_bands = graph.subjects(predicate=DBONTO.genre,
object=DB.Metal)
for band in metal_bands:
print band

71. RDFAlchemy
abstract endpoint returns dict namespace query by triples add / remove
from rdfalchemy.sparql import SPARQLGraph
from rdfalchemy import rdfSubject, rdfSingle
from rdflib import Namespace
DB = Namespace('http://dbpedia.org/resource/')
DBONTO = Namespace("http://dbpedia.org/ontology/")
RDFS = Namespace('http://www.w3.org/2000/01/rdf-schema#')
endpoint = "http://live.dbpedia.org/sparql"
graph = SPARQLGraph(endpoint)
rdfSubject.db = graph
class MusicalArtist(rdfSubject):
rdfs_label = rdfSingle(RDFS.label, 'label')
genre = rdfSingle(DBONTO.genre, 'genre')
metal_artists = MusicalArtist.filter_by(genre=DB.Metal)
for band in metal_artists:
print band
ORM django-like

72. RDFAlchemy
django-like
models

73. RDFAlchemy
simple syntax

74. RDFAlchemy
but...
non-lazy

75. RDFAlchemy
we have to declare
all data already
described in TTL files
as python classes

76. semantic-django
abstract endpoint returns dict namespace query by triples add / remove
# Classes similar to django model's are created from TTL
# files (using manage.py)
class BaseLugar(BaseEntidade):
latitude = models.UriField()
longitude = models.UriField()
geonameid = models.UriField()
tem_mapa = models.UriField()
apelido = models.UriField()
ImagemMapa = models.UriField()
genero_gramatical = models.UriField()
class Meta:
semantic_graph = 'http://semantica.globo.com/base/Lugar'
ORM django-like

77. semantic-django
https://github.com/rfloriano/semantic-django

78. semantic-django
dream of
many
product developers

79. semantic-django
but...
just started to be
developed

80. study existing solutions, and now?
[ ] contribute to them
[ ] develop on top of them
[ ] create a solution from scratch
[ ] other, _________________

81. grab your post-it, it's review time!
=) =( comments
shows no my
SuRF query models favorite
not my
nice
models lazy choice
RDFAlchemy API
name low
RDFlib space layer
django just
semantic-django like started
(...)

83. any questions...?
@tati_alchueyr

84. casting by
(click to know more about each meme)