Getting Started With The Talis Platform

Getting Started With The Talis Platform - Presentation Transcript

Getting Started with the Talis Platform

Leigh Dodds

Platform Programme Manager

Talis

December 2008

http://creativecommons.org/licenses/by/2.0/uk/

Agenda

Platform Overview

Core Concepts

Review of the RDF Model

Managing binary data

Managing structured metadata

Exploring RDF data with SPARQL

Extra Features

Store Administration

Summary

Platform Overview

Software as a Service Multi-Tenant Data Storage Service

Unstructured Data Storage e.g. binary files, including images, documents, etc

Structured Data Storage RDF metadata

Access Control All data is open (to read) by default Configurable access options

Full-Text Searching and Querying

Standards Compliance RDF, SPARQL, HTTP

Platform Architecture Web API Metabox Contentbox

REST, RDF Authentication & Authorization Content Negotiation Core Concepts aka “The Science Bit”

REST Re presentational S tate T ransfer Correct Use of HTTP

Resource-Centric API Everything has a unique URI

Interact with resources using HTTP GET = read PUT = write POST = update/modify DELETE = delete

Use HTTP Response Codes 200 = OK 201 = Created (new resource) 202 = Accepted (for processing) 400 = Bad Request 500 = Server Error

Mime Types Used to identifiy content & meaning of request and response body

Content Negotiation Majority of services support multiple output options, list varies by resource Accept header output parameter

Our Service Checklist

Consistent URI structure

Every service has human interface

Plain text error messages for easy debugging

Cacheable

… etc

Authentication HTTP Digest Authentication

Authentication Example

Authorization By default stores are world-readable, Store owner writable Customisable roles and privileges per-Store

Review of the RDF Model

Apollo 11 was launched from Cape Canaveral

Apollo 11 was launched from Cape Canaveral Subject Predicate Object

<http://purl.org/net/schemas/space/spacecraft/apollo-11> <http://purl.org/net/schemas/space/launchsite> <http://purl.org/net/schemas/space/launchsite/capecanaveral>.

space: spacecraft/apollo-11 space: launchsite space: launchsite/capecanaveral.

space:spacecraft/apollo-11 space:launchsite space:launchsite/capecanaveral. space:spacecraft/apollo-11 rdfs:label “Apollo 11” . space:launchsite/capecanaveral rdfs:label “Cape Canaveral” .

Benefits of RDF?

Good for Semi-structured Data “Schema-Free” Very Flexible

Extensible New properties New resources New types of resource New statements

Encourages Convergence Reuse of vocabularies (i.e. properties) Reuse of identifiers (i.e. talk about the same things)

Simplifies Data Integration and Aggregation Shared identifiers Common data model Common query language Common data formats

Several Different Ways to Serialize RDF Optimized for different purposes

Turtle Simple to read and hand-author Used in SPARQL query language

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> @prefix space: <http://purl.org/net/schemas/space/> @ @prefix dc: <http://purl.org/dc/elements/1.1/> <http://purl.org/net/schemas/space/spacecraft/1969-059A> rdf:type <http://purl.org/net/schemas/space/Spacecraft>; dc:description "Apollo 11 was…”; space:agency "United States" .

RDF/XML Best for data interchange Harder to read

<rdf:RDF xmlns:j.0="http://xmlns.com/foaf/0.1/“ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:space="http://purl.org/net/schemas/space/" xmlns:dc="http://purl.org/dc/elements/1.1/" xml:base="http://purl.org/net/schemas/space"> <rdf:Description rdf:about="/spacecraft/1969-059A"> <dc:description>Apollo 11 was…</dc:description> <rdf:type rdf:resource="http://purl.org/net/schemas/space/Spacecraft"/> <space:agency>United States</space:agency> </rdf:Description> </rdf:RDF>

The Content Box Managing unstructured, binary data

Store any stream of binary data Images, documents, Javascript, etc

Full HTTP Caching Support ETags Efficient retrieval Conditional updates

Server or Client Assignment of Identifiers Provides full control over how URIs assigned

ContentBox URLs

/storename/items

The Contentbox container

/storename/items/<id>

An individual item

Adding Content

Deleting Content

Metadata for Contentbox Resources Minimum is URI and ETag Extract height & width of images … more metadata extraction in future

The Meta Box Managing structured metadata

Full RDF Data Storage Create, read, update, delete RDF resources Query RDF data

Configurable Full Text Indexing of RDF Indexes updated whenever new metadata added

Versioned and Un-Versioned Updates By submitting data to separate resources Maintain audit trail

Can be Divided into Sub-Graphs Separate access control options

Metabox URLs

/storename/meta

The metabox

/storename/meta/changesets

The collection of changesets associated with this metabox

/storename/meta/graphs

The collection of sub-graphs

/storename/meta/graphs/{id}

A sub-graph

/storename/meta/graphs/{id}/changesets

The collection of changesets associated with a sub-graph

/storename/services/sparql

SPARQL endpoint for metabox

/storename/services/multisparql

SPARQL endpoint for querying across all sub-graphs

Storing RDF POST application/rdf+xml Changes saved immediately Search indexing asynchronous

Triples are Merged into Store Can catch out the unwary Updates happen through separate mechanism

Retrieving Metadata /meta?about=…URI… Can select RDF serialization

Updating Resources POST application/vnd.talis.changeset+xml

ChangeSets Vocabulary that specifies removals/additions to an RDF graph

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"

xmlns:cs="http://purl.org/vocab/changeset/schema#">

<cs:ChangeSet rdf:about="http://example.com/changesets#change">

< cs:subjectOfChange

rdf:resource="http://purl.org/net/schema/space/launch/1969-059"/>

< cs:createdDate >2008-12-08T00:00:00Z</ cs:createdDate >

< cs:creatorName >Leigh Dodds</ cs:creatorName >

< cs:changeReason >More accurate launch time</ cs:changeReason >

<cs:removal>

<rdf:Statement>

<rdf:subject rdf:resource="http://purl.org/net/schema/space/launch/1969-059"/>

<rdf:predicate rdf:resource="http://purl.org/net/schema/space/launched"/>

<rdf:object>1969-07-16</rdf:object>

</rdf:Statement>

</cs:removal>

<cs:addition>

<rdf:Statement>

<rdf:object>1969-07-16T13:32:00</rdf:object>

</rdf:Statement>

</cs:addition>

</cs:ChangeSet>

</rdf:RDF>

<cs:subjectOfChange

<cs:createdDate>2008-12-08T00:00:00Z</cs:createdDate>

<cs:creatorName>Leigh Dodds</cs:creatorName>

<cs:changeReason>More accurate launch time</cs:changeReason>

< cs:removal >

<rdf:Statement>

< rdf:subject rdf:resource="http://purl.org/net/schema/space/launch/1969-059"/>

< rdf:predicate rdf:resource="http://purl.org/net/schema/space/launched"/>

< rdf:object >1969-07-16</ rdf:object >

</rdf:Statement>

</ cs:removal >

<cs:addition>

<rdf:Statement>

<rdf:object>1969-07-16T13:32:00</rdf:object>

</rdf:Statement>

</cs:addition>

</cs:ChangeSet>

</rdf:RDF>

<cs:subjectOfChange

<cs:createdDate>2008-12-08T00:00:00Z</cs:createdDate>

<cs:creatorName>Leigh Dodds</cs:creatorName>

<cs:changeReason>More accurate launch time</cs:changeReason>

<cs:removal>

<rdf:Statement>

<rdf:object>1969-07-16</rdf:object>

</rdf:Statement>

</cs:removal>

< cs:addition >

<rdf:Statement>

< rdf:subject rdf:resource="http://purl.org/net/schema/space/launch/1969-059"/>

< rdf:predicate rdf:resource="http://purl.org/net/schema/space/launched"/>

< rdf:object >1969-07-16T13:32:00</ rdf:object >

</rdf:Statement>

</ cs:addition >

</cs:ChangeSet>

</rdf:RDF>

Versioned Updates POST to /meta/changesets Apply update and stores changeset for later retrieval

Batch Updates Combine several changesets into single POST Linked together to define ordering

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:cs="http://purl.org/vocab/changeset/schema#"> <cs:ChangeSet rdf:about="http://example.com/changesets/1"> <cs:subjectOfChange rdf:resource="http://purl.org/net/schema/space/launch/1969-059"/> <cs:changeReason>More accurate launch time</cs:changeReason> < cs:precedingChangeset rdf:resource=" http://example.com/changesets/2 "/> <!– changes --> </cs:ChangeSet> <cs:ChangeSet rdf:about=" http://example.com/changesets/2 "> <cs:subjectOfChange rdf:resource="http://purl.org/net/schema/space/launch/1969-059"/> <cs: precedingChangeset rdf:resource=" http://example.com/changesets/3 "/> <!– changes --> </cs:ChangeSet> <cs:ChangeSet rdf:about=" http://example.com/changesets/3 "> <cs:subjectOfChange rdf:resource="http://purl.org/net/schema/space/spacecraft/1969-059D"/> <!– changes --> ... </cs:ChangeSet> </rdf:RDF>

Data Extraction & Exploration with SPARQL

SPARQL RDF query language; HTTP protocol; Results format 4 different forms of query

ASK Test whether the graph contains some data of interest

#Was there a launch on 16 th July 1969? PREFIX space: <http://purl.org/net/schemas/space/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> ASK WHERE { ?launch space:launched "1969-07-16"^^xsd:date. }

<?xml version="1.0"?> <sparql xmlns="http://www.w3.org/2005/sparql-results#"> <head> </head> <boolean>true</boolean> </sparql>

DESCRIBE Generate an RDF description of a resource(s)

#Describe launch(es) that occurred on 16 th July 1969 PREFIX space: <http://purl.org/net/schemas/space/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> DESCRIBE ?launch WHERE { ?launch space:launched "1969-07-16"^^xsd:date. }

#Describe spacecraft launched on 16 th July 1969 PREFIX space: <http://purl.org/net/schemas/space/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> DESCRIBE ?spacecraft WHERE { ?launch space:launched "1969-07-16"^^xsd:date. ?spacecraft space:launch ?launch. }

CONSTRUCT Create a custom RDF graph based on query criteria

PREFIX space: <http://purl.org/net/schemas/space/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> CONSTRUCT { ?spacecraft foaf:name ?name; space:agency ?agency; space:mass ?mass. } WHERE { ?launch space:launched "1969-07-16"^^xsd:date. ?spacecraft space:launch ?launch; foaf:name ?name; space:agency ?agency; space:mass ?mass. }

SELECT SQL style result set retrieval

PREFIX space: <http://purl.org/net/schemas/space/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> SELECT ?name ?agency ?mass WHERE { ?launch space:launched "1969-07-16"^^xsd:date. ?spacecraft space:launch ?launch; foaf:name ?name; space:agency ?agency; space:mass ?mass. }

…as XML <?xml version="1.0"?> <sparql xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2005/sparql-results#" > <head> <variable name="name"/> <variable name="agency"/> <variable name="mass"/> </head> <results> <result> <binding name="name"> <literal>Apollo 11 Command and Service Module (CSM)</literal> </binding> <binding name="agency"> <literal>United States</literal> </binding> <binding name="mass"> <literal>28801.0</literal> </binding> </result> <!– more results --> </results> </sparql>

…as JSON { "head": { "vars": [ "name" , "agency" , "mass" ] } , "results": { "bindings": [ { "name": { "type": "literal" , "value": "Apollo 11 Command and Service Module (CSM)" } , "agency": { "type": "literal" , "value": "United States" } , "mass": { "type": "literal" , "value": "28801.0" } } , { "name": { "type": "literal" , "value": "Apollo 11 SIVB" } , "agency": { "type": "literal" , "value": "United States" } , "mass": { "type": "literal" , "value": "13300.0" } } , { "name": { "type": "literal" , "value": "Apollo 11 Lunar Module / EASEP" } , "agency": { "type": "literal" , "value": "United States" } , "mass": { "type": "literal" , "value": "15065.0" } } ] } }

Tour of Extra Features Searching, browsing, augmentation

Searching Full text index over RDF literals Configurable indexing options

/items?query=[query] &max=[10] &offset=[0] &sort=[comma-separated fieldnames] &xsl=[XSLT stylesheet] &content-type=[mimetype for XSLT results]

Query Syntax

lunar

luna*

“ apollo 11”

lunar OR apollo

name:apollo

(lunar OR apollo) AND agency:united states

Query Results RSS 1.0 feed OpenSearch extensions (paging, relevance) Full description of each resource

<rdf:RDF xmlns="http://purl.org/rss/1.0/" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:relevance="http://a9.com/-/opensearch/extensions/relevance/1.0/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:ns.1="http://purl.org/net/schemas/space/"> <channel rdf:about=“…"> <title>lunar</title> <link>…</link> <description>Results of a search for lunar on space</description> <items> <rdf:Seq rdf:about="urn:uuid:eae4ead8-ca6a-4b12-b714-fe631d38e447"> <rdf:li resource="http://purl.org/net/schemas/space/spacecraft/LUNAR-A" /> </rdf:Seq> </items> < os:startIndex >0</ os:startIndex > < os:itemsPerPage >10</ os:itemsPerPage > < os:totalResults >118</ os:totalResults > </channel> <item rdf:about="http://purl.org/net/schemas/space/spacecraft/LUNAR-A"> <title>Item</title> <link>http://purl.org/net/schemas/space/spacecraft/LUNAR-A</link> < relevance:score >1.0</ relevance:score > <foaf:name>Lunar-A</foaf:name> <space:mass>520.0</space:mass> <space:internationalDesignator>LUNAR-A</space:internationalDesignator> </item> </rdf:RDF>

Facetted Search Similar to Amazon product search, etc Group search results by specific fields

/services/facet?query=[query] &fields=[comma-separated fieldnames] &top=[10] &format=[xml|html]

<facet-results xmlns="http://schemas.talis.com/2007/facet-results#"> <head> <query>name:luna*</query> <fields>agency</fields> <top>10</top> <output>xml</output> </head> <fields> <field name="agency"> <term value="U.S.S.R" number="25" facet-uri=“…" search-uri=“…"/> <term value="United States" number="9" facet-uri=“…" search-uri=“…"/> <term value="Japan" number="1" facet-uri=“…" search-uri=“…"/> <term value="India" number="1" facet-uri=“…" search-uri=“…"/> </field> </fields> </facet-results>

Augmentation Annotate an RSS 1.0 feed against a store Automatically add a description of each referenced resource

Store Administration Job Control, Store Configuration

Field Predicate Map Associate a short name to a RDF property Properties in field predicate map are indexed for searching Short name used in query syntax, sort order, etc

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:bf="http://schemas.talis.com/2006/bigfoot/configuration#" xmlns:frm="http://schemas.talis.com/2006/frame/schema#“ xml:base=“http://api.talis.com/stores/space”> <bf:FieldPredicateMap rdf:about="/indexes/default/fpmaps/default"> <frm:mappedDatatypeProperty> <rdf:Description rdf:about="/indexes/default/fpmaps/default#agency"> <frm:property rdf:resource="http://purl.org/net/schema/space/agency"/> <frm:name>agency</frm:name> </rdf:Description> </frm:mappedDatatypeProperty> </bf:FieldPredicateMap> </rdf:RDF>

Query Profile Assign weightings to fields for searching

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:bf="http://schemas.talis.com/2006/bigfoot/configuration#" xmlns:frm="http://schemas.talis.com/2006/frame/schema#“ xml:base=“http://api.talis.com/stores/space”> <bf:QueryProfile rdf:about=""> <bf:fieldWeight> <rdf:Description rdf:about="/indexes/default/queryprofiles/default#name"> <bf:weight>10.0</bf:weight> <frm:name>name</frm:name> </rdf:Description> </bf:fieldWeight> <bf:fieldWeight> <rdf:Description rdf:about="/indexes/default/queryprofiles/default#agency"> <bf:weight>5.0</bf:weight> <frm:name>agency</frm:name> </rdf:Description> </bf:fieldWeight> </bf:QueryProfile> </rdf:RDF>

Job Control Reindex, Reset, Snapshot, Restore POST Job Request to /jobs

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:bf="http://schemas.talis.com/2006/bigfoot/configuration#"> <bf:JobRequest> <rdfs:label>Reset the data in my store</rdfs:label> <bf:jobType rdf:resource="http://schemas.talis.com/2006/bigfoot/configuration#ResetDataJob"/> <bf:startTime>2008-12-01T15:10:00Z</bf:startTime> </bf:JobRequest> </rdf:RDF>

Jobs Each job is a resource, with a URI GET to monitor status, DELETE to remove

Summing Up Summary, Additional Resources

The Talis Platform…