Tutorial "Linked Data Query Processing" Part 4 "Execution Process" (WWW 2013 Ed.)

949 views

Published on

These are the slides from my WWW 2013 Tutorial "Linked Data Query Processing" http://db.uwaterloo.ca/LDQTut2013/

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
949
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
57
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Tutorial "Linked Data Query Processing" Part 4 "Execution Process" (WWW 2013 Ed.)

  1. 1. Linked Data Query ProcessingTutorial at the 22nd International World Wide Web Conference (WWW 2013)May 14, 2013http://db.uwaterloo.ca/LDQTut2013/4. Execution ProcessOlaf HartigUniversity of Waterloo
  2. 2. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 2Outline Separated Execution Integrated Execution➢ General Idea and Properties➢ Push-Based Implementation [LT10, LT11]➢ Link Traversal Based Query Execution
  3. 3. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 3… clearly separatedata retrievalandresult constructioninto twoconsecutive phasesSeparated Execution ApproachesQuery-local datahttp://mdb.../Paul http://geo.../Berlinhttp://mdb.../Ric http://geo.../Rome?loc?actorGET http://.../movie244912
  4. 4. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 4Properties of Separated ExecutionQuery-local datahttp://mdb.../Paul http://geo.../Berlinhttp://mdb.../Ric http://geo.../Rome?loc?actorGET http://.../movie244912● Advantage: straightforward to implement● Can be combined with anysource selection strategy● A traditional query executionplan might then be used forconstructing the result● Downside: First solutions cannotbe reported before data retrievalhas been completed
  5. 5. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 5Query-local datahttp://mdb.../Paul http://geo.../Berlinhttp://mdb.../Ric http://geo.../Rome?loc?actorGET http://.../movie2449Integrated Execution Approaches… intertwinedata retrieval andresult construction
  6. 6. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 6Query-local datahttp://mdb.../Paul http://geo.../Berlinhttp://mdb.../Ric http://geo.../Rome?loc?actorGET http://.../movie2449Integrated Execution Approaches● Implementations may reportfirst solutions early● For monotonic queries● Implementations may processdata in a streaming manner● May require less query-localmemory (b/c query-local datasetneed not be materialized)● Can also be combined with anysource selection strategy… intertwinedata retrieval andresult construction
  7. 7. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 7Outline Separated Execution Integrated Execution➢ General Idea and Properties➢ Push-Based Implementation [LT10, LT11]➢ Link Traversal Based Query Execution√√
  8. 8. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 8Push-Based ImplementationTP Operator TP OperatorTP OperatorJoinOperatorJoinOperator● Each operator runs as anasynchronous thread● Connected viamessage queuesData RetrievalOperator
  9. 9. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 9Push-Based ImplementationTP Operator TP OperatorTP OperatorJoinOperatorJoinOperator● Each operator runs as anasynchronous thread● Connected viamessage queues● Data retrieval operator● May apply any possiblesource selection strategy● Usually, implemented viamultiple lookup threads● Pushes any incomingmatching triple to thecorresponding triplepattern operatorData RetrievalOperator
  10. 10. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 10Push-Based Implementation (Example)TP Operator( ?p, interested in, ?i )TP Operator( ?p, affiliated with, orgax )TP Operator( ?i, rdf:type, Book )JoinOperatorJoinOperatorData RetrievalOperator
  11. 11. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 11Push-Based Implementation (Example)TP Operator( ?p, interested in, ?i )TP Operator( ?p, affiliated with, orgax )TP Operator( ?i, rdf:type, Book )JoinOperatorJoinOperatorData RetrievalOperator{ ?p → alice }( alice, affiliated with, orgax )( alice, affiliated with, orgax )
  12. 12. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 12Push-Based Implementation (Example)TP Operator( ?p, interested in, ?i )TP Operator( ?p, affiliated with, orgax )TP Operator( ?i, rdf:type, Book )JoinOperatorJoinOperatorData RetrievalOperator{ ?p → alice }( alice, interested in, yoga ){ ?p → alice, ?i → yoga }( alice, interested in, yoga ){ ?p → alice, ?i → yoga }
  13. 13. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 13Symmetric Hash Join (SHJ)SHJ Operator# Bucket1 {?p → alice,?i → yoga}23 {?p → bob,?i → tea}, {?p → bob,?i → chess}# Bucket1 {?p → alice,?o → orgax}2 {?p → eve,?o → orgax}3● Maintains a hash table for each input● Same hash function for both tables● Each bucket contains a set of input solutions
  14. 14. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 14How SHJ Processes Incoming SolutionSHJ Operator# Bucket1 {?p → alice,?i → yoga}23 {?p → bob,?i → tea}, {?p → bob,?i → chess}# Bucket1 {?p → alice,?o → orgax}2 {?p → eve,?o → orgax}3● Hash the value that μinput binds to the join variableμinput = {?p → bob,?o → acme}
  15. 15. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 15How SHJ Processes Incoming SolutionSHJ Operator# Bucket1 {?p → alice,?i → yoga}23 {?p → bob,?i → tea}, {?p → bob,?i → chess}# Bucket1 {?p → alice,?o → orgax}2 {?p → eve,?o → orgax}3● Hash the value that μinput binds to the join variable● Insert μinput into the corresponding input hash tablehfct(bob) = 3μinput = {?p → bob,?o → acme}
  16. 16. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 16How SHJ Processes Incoming SolutionSHJ Operator# Bucket1 {?p → alice,?i → yoga}23 {?p → bob,?i → tea}, {?p → bob,?i → chess}# Bucket1 {?p → alice,?o → orgax}2 {?p → eve,?o → orgax}3 {?p → bob,?o → acme}● Hash the value that μinput binds to the join variable● Insert μinput into the corresponding input hash table● Probe into the other hash table to find join candidates● Merge μinput with each join candidate, push results to output{?p → bob,?o → acme,?i → tea}hfct(bob) = 3{?p → bob,?o → acme,?i → chess}μinput = {?p → bob,?o → acme}
  17. 17. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 17Outline Separated Execution Integrated Execution➢ General Idea and Properties➢ Push-Based Implementation [LT10, LT11]➢ Link Traversal Based Query Execution√√√
  18. 18. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 18Link Traversal Based Query Execution… is the combination ofintegrated execution andlive exploration(i.e., the push-based approach can beused as an implementation of LTBQE)An even closer combination presents thefollowing pull-based implementation approach ...
  19. 19. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 19?p ex:affiliated_with <http://.../orgaX>?p ex:interested_in ?b?b rdf:type <http://.../Book>QuerySeed: <http://.../orgaX>Pipeline of Link Traversing IteratorsI3I2I1
  20. 20. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 20?p ex:affiliated_with <http://.../orgaX>?p ex:interested_in ?b?b rdf:type <http://.../Book>Querytp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3Seed: <http://.../orgaX>Pipeline of Link Traversing Iterators
  21. 21. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 21?p ex:affiliated_with <http://.../orgaX>?p ex:interested_in ?b?b rdf:type <http://.../Book>QuerySeed: <http://.../orgaX>tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3query-localdatasetPipeline of Link Traversing Iterators
  22. 22. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 22?p ex:affiliated_with <http://.../orgaX>?p ex:interested_in ?b?b rdf:type <http://.../Book>QuerySeed: <http://.../orgaX>tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3query-localdatasetPipeline of Link Traversing Iterators
  23. 23. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 23?p ex:affiliated_with <http://.../orgaX>?p ex:interested_in ?b?b rdf:type <http://.../Book>QuerySeed: <http://.../orgaX>tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3query-localdatasetPipeline of Link Traversing Iterators
  24. 24. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 24query-localdatasetNext?tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3Pipeline of Link Traversing Iterators
  25. 25. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 25query-localdatasetNext?Next?tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3Pipeline of Link Traversing Iterators
  26. 26. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 26query-localdatasetNext?Next?Next?tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3Pipeline of Link Traversing Iterators
  27. 27. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 27Next?Next?Next?tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1:<http://.../alice> ex:affiliated_with <http://.../orgaX>:query-localdatasetPipeline of Link Traversing Iterators
  28. 28. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 28Next?tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1:<http://.../alice> ex:affiliated_with <http://.../orgaX>:query-localdataset{ ?p = <http://.../alice> }Next?tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3Pipeline of Link Traversing Iterators
  29. 29. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 29tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1query-localdataset{ ?p = <http://.../alice> }Next?Next?tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3Pipeline of Link Traversing Iterators
  30. 30. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 30Next?Next?tp3= ( ?b , rdf:type , <http://.../Book> ) I3tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2query-localdataset{ ?p = <http://.../alice> }Pipeline of Link Traversing Iterators
  31. 31. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 31Next?Next?tp3= ( ?b , rdf:type , <http://.../Book> ) I3tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2query-localdataset{ ?p = <http://.../alice> }:<http://.../alice> ex:interested_in <http://.../b1>:Pipeline of Link Traversing Iterators
  32. 32. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 32Next?Next?tp3= ( ?b , rdf:type , <http://.../Book> ) I3tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2{ ?p = <http://.../alice> }:<http://.../alice> ex:interested_in <http://.../b1>:query-localdatasetPipeline of Link Traversing Iterators
  33. 33. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 33Next?tp3= ( ?b , rdf:type , <http://.../Book> ) I3tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2{ ?p = <http://.../alice> }:<http://.../alice> ex:interested_in <http://.../b1>:query-localdataset{ ?p = <http://.../alice> , ?b = <http://.../b1> }Pipeline of Link Traversing Iterators
  34. 34. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 34tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1Next?{ ?p = <http://.../alice> }query-localdatasettp3= ( ?b , rdf:type , <http://.../Book> ) I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2{ ?p = <http://.../alice> , ?b = <http://.../b1> }Pipeline of Link Traversing Iterators
  35. 35. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 35tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1Next?{ ?p = <http://.../alice> }query-localdatasettp3= ( ?b , rdf:type , <http://.../Book> )tp3 = ( <http://.../b1> , rdf:type , <http://.../Book> )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2{ ?p = <http://.../alice> , ?b = <http://.../b1> }Pipeline of Link Traversing Iterators
  36. 36. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 36tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1Next?{ ?p = <http://.../alice> }tp3= ( ?b , rdf:type , <http://.../Book> )tp3 = ( <http://.../b1> , rdf:type , <http://.../Book> )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2{ ?p = <http://.../alice> , ?b = <http://.../b1> }query-localdataset:<http://.../Book> rdfs:subClassOf <http://.../CreativeWork>:Pipeline of Link Traversing Iterators
  37. 37. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 37tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1Next?{ ?p = <http://.../alice> }tp3= ( ?b , rdf:type , <http://.../Book> )tp3 = ( <http://.../b1> , rdf:type , <http://.../Book> )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2{ ?p = <http://.../alice> , ?b = <http://.../b1> }query-localdataset:<http://.../b1> rdf:type <http://.../Book>:Pipeline of Link Traversing Iterators
  38. 38. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 38tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1Next?{ ?p = <http://.../alice> }tp3= ( ?b , rdf:type , <http://.../Book> )tp3 = ( <http://.../b1> , rdf:type , <http://.../Book> )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2{ ?p = <http://.../alice> , ?b = <http://.../b1> }:<http://.../b1> rdf:type <http://.../Book>:query-localdatasetPipeline of Link Traversing Iterators
  39. 39. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 39{ ?p = <http://.../alice> , ?b = <http://.../b1> }tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1{ ?p = <http://.../alice> }tp3= ( ?b , rdf:type , <http://.../Book> )tp3 = ( <http://.../b1> , rdf:type , <http://.../Book> )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2{ ?p = <http://.../alice> , ?b = <http://.../b1> }query-localdatasetPipeline of Link Traversing Iterators
  40. 40. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 40tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp3= ( ?b , rdf:type , <http://.../Book> )tp3 = ( <http://.../b1> , rdf:type , <http://.../Book> )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2Pipeline of Link Traversing Iterators{ ?p = <http://.../alice> }{ ?p = <http://.../alice> , ?b = <http://.../b1> }Next?query-localdataset
  41. 41. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 41tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp3= ( ?b , rdf:type , <http://.../Book> ) I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2Pipeline of Link Traversing Iterators{ ?p = <http://.../alice> }Next?Next?query-localdataset
  42. 42. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 42tp1= ( ?p , ex:affiliated_with , <http://.../orgaX> ) I1tp3= ( ?b , rdf:type , <http://.../Book> ) I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( <http://.../alice> , ex:interested_in , ?b )I2Pipeline of Link Traversing Iterators{ ?p = <http://.../alice> }Next?Next?query-localdataset
  43. 43. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 43● Deeply intertwines integrated execution + live exploration● Data retrieval is a side-effect of calling link traversing iterators● No need for a separate data retrieval operator● Makes adding LTBQE to existing SPARQL engines easy● Because those usually use iterators for execution● Caveat: does not achieve the full flexibilitythat is possible with LTBQE● Iterators evaluate the triple patterns in a fixed order● Iterators discard each input solution after using itPipeline of Link Traversing Iterators
  44. 44. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 44?p ex:affiliated_with <http://.../orgaX>?p ex:interested_in ?b?b rdf:type <http://.../Book>QuerySeed: <http://.../orgaX>tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , <http://.../Book> ) I3tp1= ( ?p , ex:affiliated_with , <http://.../orgaX>) I1Alternative Execution Order
  45. 45. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 45?p ex:affiliated_with <http://.../orgaX>?p ex:interested_in ?b?b rdf:type <http://.../Book>QuerySeed: <http://.../orgaX>tp2= ( ?p , ex:interested_in , ?b ) I2Alternative Execution Ordertp3= ( ?p , ex:affiliated_with , <http://.../orgaX>) I3tp1= ( ?b , rdf:type , <http://.../Book> ) I1
  46. 46. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 46?p ex:affiliated_with <http://.../orgaX>?p ex:interested_in ?b?b rdf:type <http://.../Book>QuerySeed: <http://.../orgaX>tp2= ( ?p , ex:interested_in , ?b ) I2query-localdatasetIterator Based Executiontp1= ( ?b , rdf:type , <http://.../Book> ) I1tp3= ( ?p , ex:affiliated_with , <http://.../orgaX>) I3
  47. 47. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 47tp3= ( ?p , ex:affiliated_with , <http://.../orgaX>) I3tp2= ( ?p , ex:interested_in , ?b ) I2tp1= ( ?b , rdf:type , <http://.../Book> ) I1:<http://.../alice> ex:affiliated_with <http://.../orgaX>:query-localdatasetAlternative Execution Order
  48. 48. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 48Next?:<http://.../alice> ex:affiliated_with <http://.../orgaX>:query-localdatasetNext?tp1= ( ?b , rdf:type , <http://.../Book> ) I1Alternative Execution OrderNext?tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?p , ex:affiliated_with , <http://.../orgaX>) I3
  49. 49. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 49Next?:<http://.../alice> ex:affiliated_with <http://.../orgaX>:query-localdatasetNext?tp1= ( ?b , rdf:type , <http://.../Book> ) I1Alternative Execution OrderEND!tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?p , ex:affiliated_with , <http://.../orgaX>) I3
  50. 50. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 50query-localdatasettp1= ( ?b , rdf:type , <http://.../Book> ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?p , ex:affiliated_with , <http://.../orgaX>) I3END!END!END!Computed queryresult may dependon the order of triple patterns= logical query execution planAlternative Execution Order
  51. 51. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 51Outline Separated Execution Integrated Execution➢ General Idea and Properties➢ Push-Based Implementation [LT10, LT11]➢ Link Traversal Based Query Execution√√√√Next part: 5. Query Planning and Optimization ...
  52. 52. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 52These slides have been created byOlaf Hartigfor theWWW 2013 tutorial onLink Data Query ProcessingTutorial Website: http://db.uwaterloo.ca/LDQTut2013/This work is licensed under aCreative Commons Attribution-Share Alike 3.0 License(http://creativecommons.org/licenses/by-sa/3.0/)

×