1. Jon Iturrioz, Iker Azpeitia, Oscar Díaz
University of the Basque Country (UPV/EHU)
San Sebastián (Spain)
YQL as a Platform for
Linked-Data Wrapper
Development

2. Outline
Ø Motivation: API vs LD Cloud
Ø LDW development and maintenance
Ø YQL platform
Ø Evaluation
Ø Conclusion
2

3. Outline
Ø Motivation: API vs LD Cloud
Ø LDW development and maintenance
Ø YQL platform
Ø Evaluation
Ø Conclusion
3

4. Chicken-egg problem
LD Producer
4
LD Consumer
No use cases
Complex
technology
No interesting
data
No applications

5. The LD picture
5

6. The LD picture
6

7. The LD picture
7
Academia: DBpedia
Public institutions: Europeana LOData
A few companies: BBC
General purpose and domain specific data
Data dumps: static data
Dbpedia-live

8. The API picture
8

9. The API picture
9
13,634 APIs

10. The API picture
10
Application programmers
Mash up makers
User-centered data and Dynamic data
No semantic data
Programatic access

11. From the API to the LD
11

12. Chicken-egg problem
LD Producer
12
LD Consumer
USE CASE
Complex
technology
INTERESTING
DATA
APPLICATIONS

13. Linked Data Wrappers
DEVELOPMENT
A LDW is an API wrapper serving Linked
Data as a result of dereferencing an URI
API
LDW
URI deref Linked Data CONSUMER
Call XML
JSON
PRODUCERLOWERING
PROCESS
LIFTING
PROCESS

14. Linked Data Wrappers
DEVELOPMENT
A LDW is an API wrapper serving Linked
Data as a result of dereferencing an URI
API
LDW
URI deref Linked Data CONSUMER
Call XML
JSON
PRODUCERLOWERING
PROCESS
LIFTING
PROCESS

15. Linked Data Wrappers
MAINTENANCE
API
LDW
URI deref Linked Data CONSUMER
Call XML
JSON
PRODUCERLOWERING
PROCESS
LIFTING
PROCESS
*API evolution: signature, protocol, data structure,…
*Errors on data mappings
*Consumer requirements change

16. API evolution
16

17. Errors on data mappings
17

18. The Challenge
LD Wrappers maintenance: keep
wrappers working and useful for
consumers.
18
Help PRODUCERs
developing and maintaining wrappers

19. Outline
Ø Mativation: API vs LD Cloud
Ø LDW development and maintenance
Ø YQL platform
Ø Evaluation
Ø Conclusion
19

20. LDW Development effort
20
Development:
• Programming skills (Wrapper
Language, API)
• Development tools and deployment
infrastructure

21. LDW Maintenance effort
21
Maintenance:
• Programming skills (errors
detection/correction)
• Time
• Debugging tools

22. Producers' effort
Development:
• Programming skills (Wrapper
Language, API)
• Development tools and deployment
infrastructure
Maintenance:
• Programming skills (errors
detection/correction)
• Time
• Debugging tools
22

23. Producers' effort
Development:
• Programming skills (Wrapper
Language, API)
• Development tools and deployment
infrastructure
Maintenance:
• Programming skills (errors
detection/correction)
• Time
• Debugging tools
23
INCREMENTAL &
COLLABORATIVE
DEVELOPMENT
OF DECLARATIVELY
DEFINED WRAPPERS:
*Producers community
*Shared artifacts
*Common Wrappers
Definition Language

24. Producers' effort
Development:
• Programming skills (Wrapper
Language, API)
• Development tools and deployment
infrastructure
Maintenance:
• Programming skills (errors
detection/correction)
• Time
• Debugging tools
24
A free online
Wrappers
Development and
Deployment
Infrastructure

25. Collaborative LDW development
and maintenance
Yahoo! Query Language platform
INCREMENTAL &
COLLABORATIVE
DEVELOPMENT
OF DECLARATIVELY
DEFINED WRAPPERS:
*Producers community
*Shared artifacts
*Common Wrappers
Definition Language
A free online
Wrappers
Development and
Deployment
Infrastructure

26. Outline
Ø Motivation: API vs LD Cloud
Ø LDW development and maintenance
Ø YQL platform
Ø Evaluation
Ø Conclusion
26

27. YQL: Yahoo Query Language
framework
A SQL-like query language (YQL) to retrieve data from
APIs as they were DB tables.
An execution infrastructure for invoking YQL statement
through a REST call.
A programming platform to abstract the APIs specificities
in order to see them as tables (ODT).
An ODTs repository on Github (1068 ODTs).
A Producers community.
27

28. LDW CREATION
INCREMENTAL CREATION:
Deref Lastfm resources (events)

29. ODT
select * from lastfm.event.getinfo
where event='3986264'
and api_key ='54839b2971aef0f6b99ebb410a0ec588'
A SQL-like query language

30. YQL console
ODTs repository
Programming platform:
• Testing console
• Editor
• ODT
• JS libraries
• Environment files
Execution infrastructure

31. host/table/class/id
URI Pattern: http://rdf.onekin.org/lastfm/event/{event}/{api_key}
URI Example: http://rdf.onekin.org/lastfm/event/3986264/5483…a0ec588
Lowering: the URI
select * from lastfm
where event='3986264'
and api_key ='54839b2971aef0f6b99ebb410a0ec588'
URI Pattern: http://rdf.onekin.org/lastfm/event/{event}
URI Example: http://rdf.onekin.org/lastfm/event/3986264

32. env 'store://FvY1ygoNJgkqrWEvZWQR5w';
select * from lastfm
where event='3986264'

33. Lowering:
host/table/class/id
URI Pattern: http://rdf.onekin.org/lastfm/event/{event}
URI Example: http://rdf.onekin.org/lastfm/event/3986264
env 'store://FvY1ygoNJgkqrWEvZWQR5w';
select * from lastfm
where event='3986264’

35. LIFTING: pipe functions
SELECT statement with the pipe symbol (“|”).
These functions are applied to the result set after
the SELECT statement performs all other
operations.
35
env 'store://FvY1ygoNJgkqrWEvZWQR5w';
select * from lastfm
where event='3986264'
| lastfm.lifting (http://rdf.onekin.org/lastfm/event/
3986264)

36. LDW
Interlinkage
Xpath
Semantic
property

37. Mapping
Interlinkage

38. Registering LDW and tokens
Register the LDW URL
Register a token if the LDW needs it

39. Deref sequence diagram

40. Outline
Ø Motivation: API vs LD Cloud
Ø LDW development and maintenance
Ø YQL platform
Ø Evaluation
Ø Conclusion
40

41. Effectiveness
20’ incremenal creation vs. 50’ from scatch
Problems identifying suitable ontologies

42. Effectiveness

43. Latency
Direct call
One indirection
Two indirections
1000 calls (one/sg)
Our server
Yahoo! server farm

44. Outline
Ø Motivation: API vs LD Cloud
Ø LDW development and maintenance
Ø YQL platform
Ø Evaluation
Ø Conclusion
44

45. Conclusion
Ø YQL platform supports:
Ø INCREMENTAL development.
Ø COLLABORATIVE development (browser script)
Ø Programmers community.
Ø ODT are
Ø DECLARATIVELY defined
Ø Shared on a repository.
45
YQL is a free powerful online Infrastructure

46. Chicken-egg problem
LD Producer
46
LD Consumer
USE CASE
EASY
TECHNOLOGY
INTERESTING
DATA
APPLICATIONS

47. 47

48. Deref process:
http://rdf.onekin.org/lastfm/event/3986264
env 'store://FvY1ygoNJgkqrWEvZWQR5w';
select * from lastfm
where event='3986264'
| lastfm.lifting (http://rdf.onekin.org/lastfm/event/3986264)
Identify the LDW and its token
Create the select statement
Call the REST
Execute the select statement
Aply lifting function
Add metada
(provenance, void)
Onekin deref server
YQL platform