SemTech 2011 Semantic Search tutorial


Published on

SemTech 2011 tutorial on Semantic Search by Peter Mika (Yahoo! Research) and Thanh Tran (AIFB Institute at KIT)

Published in: Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Search is a form of content aggregation
  • - In recent years we have witnessed tremendous interest and substantial economic exploitation of search technologies, both at web and enterprise scale. In this regard, technologies for Information Retrieval (IR) can be distinguished from solutions in the field of Database (DB) and Knowledge-based Expert Systems (KB). Whereas IR applications support the retrieval of documents (document retrieval), DB and KB systems deliver more precise answers (data retrieval). The technical differences between these two paradigms can be broken down into three main dimensions, i.e. the representation of the user need (query model), the underlying resources (data model) and the matching technique. The representation of user need and resource content in current IR systems is still almost exclusively achieved by the lightweight syntax-centric models such as the predominant keyword paradigm (i.e. keyword queries matched against bag-of-words document representation). While these systems have shown to work well for topical search, i.e. retrieve documents based on a topic, they usually fail to address more complex information needs. Using more expressive models for the representation of the user need, and leveraging the structure and semantics inherent in the data, DB and KB systems allow for complex queries, and for the retrieval of concrete answers that precisely match them.
  • Semantic search can be seen as a retrieval paradigm Centered on the use of semanticsIncorporates the semantics entailed by the query and (or) the resources into the matching process, it essentially performs semantic search.
  • Another trend resulting from this convergence of textual, structured and semantic data is the need for hybrid semantic search systems. Whilestandard IR focuses on the retrieval of documents, the DB and KB systems are built for data retrieval. As opposed to these types of systems, hybrid search systems manage the different types of data in a holistic way, and support the retrieval of answers that are integrated units of information, possibly assembled from different types of data. In such a system, there is not only a convergence at the
  • Close to the topic of keyword-search in databases, except knowledge-bases have a schema-oblivious designDifferent papers assume vastly different query needs even on the same type of data
  • >> >> Intro Session: motivation, overview etc.: 10 min >> >>1)Representation of the Search Space (M) 45>> >>2)Offline Preprocessing: Crawling and Indexing (P) 60>> >>3)Query Processing (T) 4)Matching (T) 90>> >>5)Ranking 60 >> >>6)Result Presentation (45)>> >>7)Evaluation (45)>> >>Demo Session (30)>> >> Wrap-up Session (10)
  • Approximate many results  need rankingRanking also needed in the case where qp is complete and sound, but queries and data representation so imprecise such that we have to deal with too many results
  • Miss structural information in textsHyperlinksLinguistic structurePositional information
  • - Real world objects
  • SELECT Returns all, or a subset of, the variables bound in a query pattern match. CONSTRUCT Returns an RDF graph constructed by substituting variables in a set of triple templates. ASK Returns a boolean indicating whether a query pattern matches or not. DESCRIBE Returns an RDF graph that describes the resources found. Graph patterns are defined recursively. A graph pattern may have zero or more optional graph patterns, and any part of a query pattern may have an optional part. In this example, there are two optional graph patterns.Section 6 introduces the ability to make portions of a query optional; Section 7 introduces the ability to express the disjunction of alternative graph patterns; and Section 8 introduces the ability to constrain portions of a query to particular source graphs. Section 8 also presents SPARQL's mechanism for defining the source graphs for a query.Basic graph patterns allow applications to make queries where the entire query pattern must match for there to be a solution. For every solution of a query containing only group graph patterns with at least one basic graph pattern, every variable is bound to an RDF Term in a solution. However, regular, complete structures cannot be assumed in all RDF graphs. It is useful to be able to have queries that allow information to be added to the solution where the information is available, but do not reject the solution because some part of the query pattern does not match. Optional matching provides this facility: if the optional part does not match, it creates no bindings but does not eliminate the solution.The UNION pattern combines graph patterns; each alternative possibility can contain more than one triple pattern:SPARQL provides a means of combining graph patterns so that one of several alternative graph patterns may match. If more than one of the alternatives matches, all the possible pattern solutions are found.
  • Web data: Text+ Linked Data+ Semi-structured RDF+ Hybrid datathat can be conceived as forming data graphsHear abour bob and alice all the time (in computer science literatures), want to find out more… build Semantic Web search engine. To address complex information needs by exploiting Web data:- Query as a set of constrains Match structured data Match text
  • - Less than 5 percent of IR papers deal with query processing and the aspect of efficiency
  • Partitioning has impact on performance!)Blocking: iterator-based approachesNon-blocking: good for streaming, good we cannot wait for some parts of the results to be completely worked-offLink data: cannot wait for sources, (some are slower then other) thus better to push data into query processing as the they come instead of pulling data and wait (busy waiting)Top-k:
  • -phrase patterns (e.g., “X is the capital of Y”) for large scale extraction. Such simple patterns, when coupled with the richness and redundancy of theWeb, can be very useful in scraping millions or even billions of facts from the Web.- patterns: Matched when keywords or data types Xi appear in sequence. Matched if all keywords/data types/patterns appear within an m-words window.For extraction: relation patternsFor text search: entity patterns -When not assuming that all relevant data can be extracted such matching against text still needed: Hybrid search
  • Given some materialized indexes  no joins at all Given sorted inputs  sorted merge join
  • Given some materialized indexes  no joins at all Given sorted inputs  sorted merge joinjoin
  • Every task is a challenge of itself, some more some less well elaboratedThere are separate challenges for every problems
  • >> >> Intro Session: motivation, overview etc.: 10 min >> >>1)Representation of the Search Space (M) 45>> >>2)Offline Preprocessing: Crawling and Indexing (P) 60>> >>3)Query Processing (T) 4)Matching (T) 90>> >>5)Ranking 60 >> >>6)Result Presentation (45)>> >>7)Evaluation (45)>> >>Demo Session (30)>> >> Wrap-up Session (10)
  • Approximate many results  need rankingRanking also needed in the case where qp is complete and sound, but queries and data representation so imprecise such that we have to deal with too many results
  • Web data: Text+ Linked Data+ Semi-structured RDF+ Hybrid datathat can be conceived as forming data graphsHear abour bob and alice all the time (in computer science literatures), want to find out more… build Semantic Web search engine. To address complex information needs by exploiting Web data:- Query as a set of constrains Match structured data Match text
  • Syntactic works at Semantic works vs. employ
  • text which contains a large mentions of “Berlin” is likely to be about “Berlin”i is more likely to be the correct interpretation of K when terms in K co-occur in a large number of context (bag of words) associated with i“Berlin and apartment” more often co-occur in the geographic location context/topic than in the context of people“Berlin and apartment” more often co-occur in the geographic location context/topic than in the context of people
  • SemTech 2011 Semantic Search tutorial

    1. 1. Peter Mika| Yahoo Research, Spain<br /><br />Thanh Tran | Institute AIFB, KIT, Germany<br /><br />Semantic Search TutorialIntroduction<br />
    2. 2. About the speakers<br />Peter Mika<br />Semantic Search group at Yahoo! Barcelona<br />Semantic Search, Web Object Retrieval, Natural Language Processing<br />Tran Duc Thanh<br />Semantic Search group at AIFB<br />Semantic Search, Semantic Data Management, Linked Data Query Processing<br />
    3. 3. Agenda<br />Introduction (5 min)<br />Semantic Web data (50 min)<br />The RDF data model<br />Publishing RDF<br />Crawling and indexing RDF data<br />Query processing (35 min)<br />Ranking (25 min) <br />Result presentation (15 min)<br />Semantic Search evaluation (15 min)<br />Demos (15 min)<br />Questions (5 min)<br />
    4. 4. Why Semantic Search? I.<br />“We are at the beginning of search.“ (Marissa Mayer)<br />Solved large classes of queries, e.g. navigational<br />Heavy investment in computational power<br />Remaining queries are hard, not solvable by brute force, and require a deep understanding of the world and human cognition<br />Background knowledge and metadata can help to address poorly solved queries<br />
    5. 5. Poorly solved information needs<br />Ambiguous searches<br />parishilton<br />Long tail queries<br />george bush (and I mean the beer brewer in Arizona)<br />Multimedia search<br />parishilton sexy<br />Imprecise or overly precise searches <br />jimhendler<br />pictures of strong adventures people<br />Precise searches for descriptions<br />countries in africa<br />32 year old computer scientist living in barcelona<br />reliable digital camera under 300 dollars<br />Many of these queries would not be asked by users, who learned over time what search technology can and can not do.<br />
    6. 6. Example: multiple interpretations<br />
    7. 7. Why Semantic Search? II.<br />The Semantic Web is now a reality<br />Large amounts of data published in RDF<br />Heterogeneous data of varying quality<br />Users who are not skilled in writing complex queries (e.g. SPARQL) and may not be experts in the domain<br />Searching data instead or in addition to searching documents<br />Direct answers<br />Novel search tasks<br />
    8. 8. Information box with content from and links to Yahoo! Travel<br />Example: direct answers in search<br />Points of interest in Vienna, Austria<br />Shopping results from <br />Yahoo! Shopping<br />Since Aug, 2010, ‘regular’ search results are ‘Powered by Bing’<br />
    9. 9. Novel search tasks<br />Aggregation of search results<br />e.g. price comparison across websites<br />Analysis and prediction<br />e.g. world temperature by 2020<br />Semantic profiling<br />recommendations based on particular interests<br />Semantic log analysis<br />understanding user behavior in terms of objects <br />Support for complex tasks<br />e.g. booking a vacation using a combination of services<br />
    10. 10. Document retrieval and data retrieval<br />Information Retrieval (IR) support the retrieval of documents (document retrieval)<br />Representation based on lightweight syntax-centric models <br />Work well for topical search<br />Not so well for more complex information needs<br />Web scale<br />Database (DB) and Knowledge-based Systems (KB) deliver more precise answers (data retrieval)<br />More expressive models <br />Allow for complex queries<br />Retrieve concrete answers that precisely match queries<br />Not just matching and filtering, but also joins<br />Limitations in scalability<br />
    11. 11. Combination of document and data retrieval <br />Documents with metadata<br />Metadata may be embeddedinside the document<br />I’m looking for documents that mention countries in Africa.<br />Data retrieval<br />Structured data, but searchable text fields<br />I’m looking for directors, who have directed movies where the synopsis mentions dinosaurs.<br />
    12. 12. Semantic Search<br />Target (combination of) document and data retrieval<br />Semantic search is a retrieval paradigm that<br />Exploits the structure/semantics of the data or explicit background knowledge to understand user intent and the meaning of content<br />Incorporates the intent of the query and the meaning of content into the search process (semantic models)<br />Wide range of semantic search systems<br />Employ different semantic models, possibly at different steps of the search process and in order to support different tasks<br />
    13. 13. Semantic models<br />Semantics is concerned with the meaning of the resources made available for search<br />Various representations of meaning<br />Linguistic models: models of relationships among words<br />Taxonomies, thesauri, dictionaries of entity names<br />Inference along linguistic relations, e.g. broader/narrower terms<br />Natural language search<br />Conceptual models: models of relationships among objects<br />Ontologies capture entities in the world and their relationships<br />Inference along domain-specific relations<br />Knowledge-based search<br />We will focus on conceptual models in this tutorial<br />In particular, the RDF/OWL conceptual model for representing classes in a domain, and describing their instances<br />
    14. 14. Semantic Search – a process view<br />Knowledge Representation<br />Semantic Models<br />Resources<br />Documents<br />DocumentRepresentation<br />
    15. 15. Semantic Search systems<br /> For data / document retrieval, semantic search systems might combine a range of techniques, ranging from statistics-based IR methods for ranking, database methods for efficient indexing and query processing, up to complex reasoning techniques for making inferences! <br />
    16. 16. Semantic Web data<br />
    17. 17. Semantic Web<br />Sharing data across the Web<br />Standard data model<br />RDF<br />A number of syntaxes (file formats)<br />RDF/XML, RDFa<br />Powerful, logic-based languages for schemas<br />OWL, RIF<br />Query languages and protocols<br />HTTP, SPARQL<br />
    18. 18. Resource Description Framework (RDF)<br />Each resource (thing, entity) is identified by a URI<br />Globally unique identifiers<br />Locators of information<br />Data is broken down into individual facts<br />Triples of (subject, predicate, object)<br />A set of triples (an RDF graph) is published together in an RDF document<br />RDF document<br />foaf:Person<br />type<br />example:roi<br />name<br />“Roi Blanco”<br />
    19. 19. Linking resources<br />Friend-of-a-Friend ontology<br />Roi’s homepage<br />type<br />example:roi<br />foaf:Person<br />name<br />“Roi Blanco”<br />sameAs<br />Yahoo<br />type<br />worksWith<br />example:roi2<br />example:peter<br />email<br />“”<br />
    20. 20. Publishing RDF<br />Linked Data<br />Data published as RDF documents linked to other RDF documents<br />Community effort to re-publish large public datasets (e.g. Dbpedia, open government data)<br />RDFa<br />Data embedded inside HTML pages<br />Recommended for site owners by Yahoo, Google, Facebook<br />SPARQL endpoints<br />Triple stores (RDF databases) that can be queried through the web<br />
    21. 21. Linked Data<br />A web of RDF documents in parallel to the current Web<br />often implemented as wrappers around databases or APIs<br />The four rules of Linked Data:<br />Use URIs to identify things.<br />Use HTTPURIs so that these things can be referred to and looked up ("dereference") by people and user agents.<br />Provide useful information about the thing when its URI is dereferenced, using standard formats such as RDF-XML.<br />Include links to other, related URIs in the exposed data to improve discovery of other related information on the Web.<br />
    22. 22. Linked Data<br />Advantages: <br />No change to the publishing of the HTML documents<br />Data can be published by third party (e.g. Dbpedia)<br />Disadvantages:<br />Web servers need to be configured to properly handle URIs that identify concepts instead of documents<br />Not favored by search engines <br />Lack of use cases<br />Crawling needs to be changed<br />Authority is difficult to determine<br />Tools<br />Triple stores (Virtuoso, Oracle etc.) and front-ends (Pubby)<br />RDB-to-RDF mappers (e.g. D2RQ, Triplify)<br />Validators (Vapour)<br />Linked Data browsers (many)<br />
    23. 23. The state of Linked Data<br />Rapidly growing community effort to (re)publish open datasets as Linked Data<br />In particular, scientific and government datasets<br />see<br />Less commercial interest, real usage<br />
    24. 24. Metadata in HTML<br />1995: HTML meta tags<br />1996: Simple HTML Ontology Extensions (SHOE)<br />1998: RDF/XML<br />RDF/XML in HTML<br />RDF linked from HTML<br />2003: Web 2.0<br />Tagging<br />Microformats<br />Metadata in Wikipedia<br />Machine tags in Flickr<br />2005: eRDF <br />2008: RDFa 1.0<br />2011: RDFa 1.1<br />2012: Microdata?<br />
    25. 25. HTML meta tags<br /><HTML><br /><HEAD profile=""><br /><META name="" content="Peter Mika"><br /><LINK rel="DC.rights copyright" href="" /> <br /><LINK rel="meta" type="application/rdf+xml" title="FOAF" <br /> href= ""> <br /></HEAD> <br />…<br /></HTML><br />
    26. 26. Microformats (μf)<br />Agreements on the way to encode certain kinds metadata in HTML<br />Reuse of semantic-bearing HTML elements<br />Based on existing standards<br />Minimality<br />Microformats exist for a limited set of objects<br />hCard (persons and organizations)<br />hCalendar (events)<br />hResume<br />hProduct<br />hRecipe<br />Varying degrees of support and stability<br />hCard and rel-tag are widely supported<br />Community centered around<br />Specifications and discussions are hosted there<br />
    27. 27. Microformats: limitations<br />No shared syntax<br />Each microformat has a separate syntax tailored to the vocabulary <br />No formal schemas<br />Limited reuse, extensibility of schemas<br />Unclear which combinations are allowed<br />No datatypes<br />No namespaces, unique identifiers (URIs) <br />no interlinking<br />mapping between instances is required<br />Always appears in the HTML <body> <br />
    28. 28. Example: the hCard microformat<br /><div class="vcard"> <br /> <a class="email fn" href="">Joe Friday</a> <br /> <div class="tel">+1-919-555-7878</div> <br /> <div class="title">Area Administrator, Assistant</div> <br /></div> <br /><cite class="vcard"><br /><a class="fn url" rel="friend colleague met” href=""><br />Eric Meyer</a> </cite> wrote a post (<cite><br /><a href=""><br />Tax Relief</a></cite>) about an unintentionally humorous letter he received from <br />the <span class="vcard”> <a class="fn org url" href=""><br />Internal Revenue Service</a> <br /></span>. <br />
    29. 29. RDFa<br />W3C standard for embedding RDF data in HTML documents<br />A set of new HTML attributes to be used in head or body<br />A specification of how to extract the data from these attributes <br />RDFa is just a syntax, you have to choose a vocabulary separately<br />RDFa 1.0 is a W3C Recommendation since October, 2008<br />RDFa Primer<br />RDFa 1.1 is a small update on RDFa to make it easier to use<br />Currently Working Draft (March 31, 2011)<br />Updated version of the RDFa Primer (April 19, 2011)<br />RDFa API for accessing RDFa data in a webpage in the browser from JavaScript<br />Currently Working Draft (April 19, 2011)<br />
    30. 30. RDFa 1.1<br />Changes<br />New vocab attribute to define the default namespace for the document or subtree<br />Profile documents to define multiple namespace prefixes<br />The prefix attribute as a recommended replacement of xmlns<br />You can use URIs even where only CURIEs where allowed before<br />RDFa 1.1 is backward compatible with RDFa 1.0<br />RDFa 1.1 is recommended if you want to use HTML5<br />
    31. 31. When to use RDFa<br />Choose microformats when you find a microformat that fits your needs and supported by your consumers<br />Microformats are first option because they are simple<br />Yahoo supports all major microformats, see the documentation<br />It’s a common misconception that RDFa requires XHTML or that it’s compatible with HTML5<br />It’s compatible with HTML4, HTML5, XHTML<br />If you find none that perfectly fits your needs then you need RDFa<br />Microformats have a fixed schema: you can not add your own attributes<br />Example: a social networking site with user profiles<br />VCard is a good candidate, but for example it doesn’t have a way to express the user’s social connections<br />You either live without this, or go with RDFa<br />
    32. 32. Example: Facebook’s Like and the Open Graph Protocol<br />The ‘Like’ button provides publishers with a way to promote their content on Facebook and build communities <br />Shows up in profiles and news feed<br />Site owners can later reach users who have liked an object<br />Facebook Graph API allows 3rd party developers to access the data <br />Open Graph Protocol is an RDFa-based format that allows to describe the object that the user ‘Likes’<br />
    33. 33. Example: Facebook’s Open Graph Protocol<br />RDF vocabulary to be used in conjunction with RDFa<br />Simplify the work of developers by restricting the freedom in RDFa<br />Activities, Businesses, Groups, Organizations, People, Places, Products and Entertainment<br />Only HTML <head> accepted<br /><html xmlns:og=""> <br /><head> <br /> <title>The Rock (1996)</title> <br /> <meta property="og:title" content="The Rock" /> <br /> <meta property="og:type" content="movie" /> <br /> <meta property="og:url" content="" /> <br /> <meta property="og:image" content="" /> …<br /></head> ... <br />
    34. 34. Example: Yahoo! Enhanced Results (was: SearchMonkey)<br />Guide for publishers to mark-up their pages for common types of objects<br />Product, Local, News, Video, Events, Documents, Discussion, Games<br />Using popular microformats and RDF vocabularies<br />Copy-paste code <br />Validator<br />Yahoo as a consumer<br />See later<br />
    35. 35. Example: Google’s Rich Snippets<br />Google accepts popular microformats and its own RDFa vocabulary<br />Similar approach to RDFa as Facebook<br />Validator to check if the markup is correct<br />Google displays enhanced results based on this metadata<br />Rich Snippets<br />
    36. 36. RDFa on the rise<br />510% increase between March, 2009 and October, 2010<br />Percentage of URLs with embedded metadata in various formats<br />
    37. 37. Microdata<br />Currently under standardization at the W3C<br />Originally part of the HTML5 spec, but now a separate document<br />Similar to microformats, but with the extensibility of RDFa<br />Introduce new terms using reverse domain names or full URIs<br />HTML5 also has a number of “semantic” elements such as <time>, <video>, <article>…<br />
    38. 38. Microdata example<br /><div itemscope itemid=“”><br /> <p>My name is <span itemprop="name">Neil</span>.</p><br /> <p>My band is called <br /> <span itemprop="band">Four Parts Water</span>.<br /> I was born on <br /> <time itemprop="birthday" datetime="2009-05-10">May 10th 2009</time>.<br /> <img itemprop="image" src=”me.png" alt=”me”><br /> </p><br /></div<br />
    39. 39. The state of metadata in HTML<br />5-10% of webpages contain some explicit metadata<br />Depending on how you count…<br />Too many competing approaches<br />Too many formats: microformats vs RDFa vs Microdata<br />When using RDFa, publishers may need to use multiple different vocabularies to satisfy everyone<br />
    40. 40. Crawling the Semantic Web<br />Linked Data<br />Similar to HTML crawling, but the the crawler needs to parse RDF/XML (and others) to extract URIs to be crawled<br />Semantic Sitemap/VOID descriptions<br />RDFa<br />Same as HTML crawling, but data is extracted after crawling<br />Mika et al. Investigating the Semantic Gap through Query Log Analysis, ISWC 2010.<br />SPARQL endpoints<br />Endpoints are not linked, need to be discovered by other means<br />Semantic Sitemap/VOID descriptions<br />
    41. 41. Data fusion<br />Ontology matching<br />Widely studied in Semantic Web research, see e.g. list of publications at<br />Unfortunately, not much of it is applicable in a Web context due to the quality of ontologies<br />Entity resolution<br />Logic-based approaches in the Semantic Web<br />Studied as record linkage in the database literature <br />Machine learning based approaches, focusing on attributes<br />Graph-based approaches, see e.g. the work of Lisa Getoor are applicable to RDF data<br />Improvements over only attribute based matching<br />Blending<br />Merging objects that represent the same real world entity and reconciling information from multiple sources<br />
    42. 42. Data quality assessment and curation<br />Heterogeneity, quality of data is an even larger issue<br />Quality ranges from well-curated data sets (e.g. Freebase) to microformats <br />In the worst of cases, the data becomes a graph of words<br />Short amounts of text: prone to mistakes in data entry or extraction<br />Example: mistake in a phone number or state code<br />Quality assessment and data curation<br />Quality varies from data created by experts to user-generated content<br />Automated data validation<br />Against known-good data or using triangulation<br />Validation against the ontology or using probabilistic models<br />Data validation by trained professionals or crowdsourcing<br />Sampling data for evaluation<br />Curation based on user feedback<br />
    43. 43. Indexing<br />Search requires matching and ranking<br />Matching selects a subset of the elements to be scored<br />The goal of indexing is to speed up matching<br />Retrieval needs to be performed in milliseconds<br />Without an index, retrieval would require streaming through the collection<br />The type of index depends on the query model to support<br />DB-style indexing<br />IR-style indexing <br />
    44. 44. IR-style indexing<br />Index data as text<br />Create virtual documents from data<br />One virtual document per subgraph, resource or triple<br />typically: resource<br />Key differences to Text Retrieval<br />RDF data is structured<br />Minimally, queries on property values are required<br />
    45. 45. Horizontal index structure<br />Two fields (indices): one for terms, one for properties<br />For each term, store the property on the same position in the property index<br />Positions are required even without phrase queries<br />Query engine needs to support the alignment operator<br />✓ Dictionary is number of unique terms + number of properties<br />Occurrences is number of tokens * 2<br />
    46. 46. Vertical index structure<br />One field (index) per property<br />Positions are not required<br />But useful for phrase queries<br />Query engine needs to support fields<br />Dictionary is number of unique terms<br />Occurrences is number of tokens<br />✗ Number of fields is a problem for merging, query performance<br />
    47. 47. Indexing using MapReduce<br />MapReduce is the perfect model for building inverted indices<br />Map creates (term, {doc1}) pairs<br />Reduce collects all docs for the same term: (term, {doc1, doc2…}<br />Sub-indices are merged separately<br />Term-partitioned indices<br />Peter Mika. Distributed Indexing for Semantic Search, SemSearch 2010.<br />
    48. 48. Query Processing <br />
    49. 49. Structure<br />Taxonomy of retrieval approaches<br />Query processing for semantic search<br />Types of semantic data<br />Formalisms for querying semantic data<br />Approaches<br />General task: hybrid graph pattern matching<br />Matching keyword query against text<br />Matching structured query against structured data<br />Matching keyword query against structured data<br />Matching structured query against text (a hybrid case)<br />Main tasks, challenges and opportunities<br />
    50. 50. Taxonomy of retrieval approaches (1)<br />Data retrieval problem<br />A collection of resources represented by the data model G<br />Information needs expressed as queries in Q<br />Retrieval is the task of efficiently computing results from G that are relevant to the queries in Q<br />Document retrieval vs. data retrieval <br />Differences in query and data representation and matching<br />Efficiently retrieve structured data that exactly match formal information needs expressed as structured queries<br />Effectively rank textual results that match ambiguous NL / keyword queries to a certain degree (notions of relevance) <br />
    51. 51. Taxonomy of retrieval approaches (2)<br />Exact<br />Complete <br />Sound <br />Query<br />Matching<br /><ul><li>Approximate
    52. 52. Not complete
    53. 53. Not sound
    54. 54. Ranked
    55. 55. Best effort
    56. 56. Top-k</li></ul>Data<br />Query processing mainly focuses on efficiency of matching whereas ranking deals with degree of matching (relevance)!<br />
    57. 57. Query processing for Semantic Search (1)<br />The underlying collection of resources is represented by semantic data G ranging from<br />Structured data with well defined schemas<br />Semi-structured data with incomplete or no schemas<br />Data that largely comprise text<br />Hybrid / embedded data<br />Targeted information needs Q are of varying complexity, captured using different formalisms and querying paradigms <br />Natural language texts and keywords <br />Form-based inputs <br />Formal structured queries<br />Semantic search, mainly, is the task of efficiently computing results(query processing) from G that are relevantto the queries in Q (ranking)<br />
    58. 58. Query processing for Semantic Search (2)<br />Keywords<br />NL Questions<br />Form- / facet-based Inputs<br />Structured Queries (SPARQL)<br />Query<br />Matching<br />Data<br />OWL ontologies with rich, formal semantics<br />Structured RDF data<br />Semi-Structured RDF data<br />RDF data embedded in text (RDFa)<br />
    59. 59. Query processing for Semantic Search (3)<br />Textual Data<br />Keyword query on textual data <br />(IR/document retrieval) <br />Structured query on textual data <br />Semantic Search target different group of users, information needs, and types of data. Query processing for semantic search is hybrid combination of techniques!<br />Unstructured Query<br />Structured<br />Query<br />Keyword query on structured data <br />Structured query on structured data <br />(DB/data retrieval)<br />Structured Data<br />
    60. 60. Types of data models (1)<br />Textual<br />Bag-of-words<br />Represent documents, text in structured data,…, real-world objects (captured as structured data)<br />Lacks “structure” <br />in text, e.g. linguistic structure, hyperlinks, (positional information)<br />Structure in structured data representation<br />term (statistics)<br />In combination with Cloud Computing technologies, promising solutions for the management of `big data' have emerged. Existing industry solutions are able to support complex queries and analytics tasks with terabytes of data. For example, using a Greenplum.<br />combination <br />Cloud <br />Computing <br />Technologies<br />solutions <br />management <br />`big data' <br />industry <br />solutions <br />support <br />complex <br />……<br />
    61. 61. Types of data models (2)<br />Textual<br />Structured <br />Resource Description Framework (RDF) <br />Represent real-world objects, services, applications, …. documents<br />Resource attribute values and relationships between resources<br />Schema<br />Picture<br />creator<br />Person<br />Bob<br />
    62. 62. Types of data models (3)<br />Textual<br />Structured<br />Hybrid<br />RDF data embedded in text (RDFa)<br />
    63. 63. Types of data models – RDFa (1)<br />…<br /><div about="/alice/posts/trouble_with_bob"><br /> <h2 property="dc:title">The trouble with Bob</h2><br /><h3 property="dc:creator">Alice</h3><br />Bob is a good friend of mine. We went to the same university, and <br /> also shared an apartment in Berlin in 2008. The trouble with Bob is that he takes much better photos than I do:<br /> <div about=""><br /><imgsrc="" /><br /><span property="dc:title">Beautiful Sunset</span><br /> by <span property="dc:creator">Bob</span>.<br /></div><br /></div><br />…<br />adoptedfrom :<br />
    64. 64. Types of semantic data – RDFa (2) <br />Bob is a good friend of mine. We went to the same university, and also shared an apartment in Berlin in 2008. The trouble with Bob is that he takes much better photos than I do:<br />content<br />content<br />adoptedfrom :<br />
    65. 65. Types of semantic data - conclusion<br />Semantic data in general can be conceived as a graph with text and structured data items as nodes, and edges represent different types of relationships including explicit semantic relationships and vaguely specified ones such as hyperlinks!<br />
    66. 66. Formalisms for querying semantic data (1)<br />Example information need<br />“Information about a friend of Alice, who shared an apartment with her in Berlin and knows someone working at KIT.”<br />Unstructured queries<br />Fully-structured queries<br />Hybrid queries: unstructured + structured<br />
    67. 67. Formalisms for querying semantic data (2)<br />Example information need<br />“Information about a friend of Alice, who shared an apartment with her in Berlin and knows someone working at KIT.”<br />Unstructured<br />NL<br />Keywords <br />apartment<br />Berlin<br />Alice<br />shared<br />
    68. 68. Formalisms for querying semantic data (3)<br />Example information need<br />“Information about a friend of Alice, who shared an apartment with her in Berlinand knows someone working at KIT.”<br />Unstructured<br />Fully-structured<br />SPARQL: BGP, filter, optional, union, select, construct, ask, describe <br />PREFIX ns: <> <br /> SELECT ?x <br /> WHERE { ?x ns:knows ? y. ?y ns:name “Alice”. <br /> ?x ns:knows ?z. ?z ns: works ?v. ?v ns:name “KIT” }<br />
    69. 69. Formalisms for querying semantic data (4)<br />Fully-structured<br />Unstructured <br />Hybrid: content and structure constraints<br />“shared apartment Berlin Alice”<br />?x ns:knows ? y. ?y ns:name “Alice”. <br />?x ns:knows ?z. ?z ns: works ?v. <br />?v ns:name “KIT”<br />
    70. 70. Formalisms for querying semantic data (5)<br />Fully-structured<br />Unstructured <br />Hybrid: content and structure constraints<br />“shared apartment Berlin Alice”<br />?x ns:knows ? y. ?y ns:name “Alice”. <br />?x ns:knows ?z. ?z ns: works ?v. <br />?v ns:name “KIT”<br />
    71. 71. Formalisms for querying semantic data - conclusion<br />Semantic search queries can be conceived as graph patterns with nodes referring to text and structured data items, and edges referring to relationships between these items!<br />
    72. 72. Processing hybrid graph patterns (1)<br />Example information need<br />“Information about a friend of Alice, who shared an apartment with her in Berlin and knows someone working at KIT.”<br />?x ns:knows ?z. ?z ns: works ?v. ?v ns:name “KIT”<br />apartment shared Berlin Alice<br />?y ns:name “Alice”. ?x ns:knows ? y<br />age<br />works<br />34<br />trouble with bob<br />FluidOps<br />Peter<br />sunset.jpg<br />Bob is a good friend of mine. We went to the same university, and also shared an apartment in Berlin in 2008. The trouble with Bob is that he takes much better photos than I do:<br />Beautiful <br />Sunset<br />author<br />title<br />Semantic Search<br />Germany<br />Alice<br />creator<br />author<br />creator<br />knows<br />year<br />Germany<br />2009<br />Bob<br />Thanh<br />knows<br />located<br />works<br />KIT<br />
    73. 73. Processing hybrid graph patterns (2)<br />Matching hybrid graph patterns against data<br />
    74. 74. Matching keyword query against text<br /><ul><li>Retrieve documents
    75. 75. Inverted list (inverted index)</li></ul>keyword  {<doc1, pos, score, ...>, <br /><doc2, pos, score, ...>, ...}<br /><ul><li>AND-semantics: top-k join</li></ul>shared<br />Berlin<br />Alice<br />shared<br />Berlin<br />Alice<br />D1<br />D1<br />D1<br />shared<br />berlin<br />alice<br />=<br />=<br />shared<br />
    76. 76. Matching structured query against structured data<br /><ul><li>Retrieve data for triple patterns
    77. 77. Index on tables
    78. 78. Multiple “redundant” indexes to cover different access patterns
    79. 79. Join (conjunction of triples)
    80. 80. Blocking, e.g. linear merge join (required sorted input)
    81. 81. Non-blocking, e.g. symmetric hash-join
    82. 82. Materialized join indexes</li></ul>?x ns:knows ?y. ?x ns:knows ?z. <br />?z ns: works ?v. ?v ns:name “KIT”<br />Per1 ns:works?v<br />?vns:name “KIT”<br />SP-index<br />PO-index<br />=<br />=<br />=<br />Per1 ns:worksIns1<br />Ins1ns:name KIT<br />Per1 ns:works Ins1<br />Ins1 ns:name KIT<br />
    83. 83. Matching keyword query against structured data<br /><ul><li>Retrieve keyword elements
    84. 84. Using inverted index</li></ul> keyword  {<el1, score, ...>, <el2, score, ...>,…} <br /><ul><li>Exploration / “Join”
    85. 85. Data indexes for triple lookup
    86. 86. Materialized index (paths up to graphs)
    87. 87. Top-k Steiner tree search, top-k subgraph exploration </li></ul>Alice<br />Bob<br />Bob<br />KIT<br />Alice<br />KIT<br />↔<br />↔<br />=<br />=<br />Alice ns:knowsBob<br />Inst1ns:name KIT<br />Bobns:worksInst1<br />
    88. 88. Matching structured query against text<br /><ul><li>Based on offline IE (offline see Peter’s slides)
    89. 89. Based on online IE, i.e., “retrieve “ is as follows
    90. 90. Derive keywords to retrieve relevant documents
    91. 91. On-the-fly information extraction, i.e., phrase pattern matching “X title Y”
    92. 92. Retrieve extracted data for structured part
    93. 93. Retrieve documents for derived text patterns, e.g. sequence, windows, reg. exp.
    94. 94. Index
    95. 95. Inverted index for document retrieval and pattern matching
    96. 96. Join index  inverted index for storing materialized joins between keywords
    97. 97. Neighborhood indexes for phrase patterns</li></ul>?x ns:knows ?y. ?x ns:knows ?z. <br />?z ns: works ?v. ?v ns:name “KIT”<br />KIT<br />name<br />knows<br />name<br />KIT<br />Hybrid case<br />
    98. 98. Query processing – main tasks<br />Retrieval<br />Documents , data elements, triples, paths, graphs<br />Inverted index,…, but also other indexes (B+ tree)<br />Index documents, triples materialized join paths<br />Join<br />Different join implementations, efficiency depends on availability of indexes<br />Non-blocking join good for early result reporting and for “unpredictable” linked data scenario<br />Query<br />Matching<br />Data<br />
    99. 99. Query processing – more tasks<br />Disjunction, aggregation, grouping<br />Join order optimization<br />Approximate <br />Approximate the search space <br />Approximate the results (matching, join)<br />Parallelization<br />Top-k <br />Use only some entries in the input streams to produce k results<br />Multiple sources<br />Federation, routing <br />On-the-fly mapping, similarity join <br />Hybrid<br />Join text and data<br />Query<br />Matching<br />Data<br />
    100. 100. Query processing on the Web - research challenges and opportunities <br />Large amount of semantic data<br />Data inconsistent, redundant, and low quality <br />Large amount of data embedded in text <br />Large amount of sources<br />Large amount of links between sources<br /><ul><li>Optimization parallelization,
    101. 101. Approximation
    102. 102. Hybrid querying and data management
    103. 103. Federation, routing
    104. 104. Online schema mappings
    105. 105. Similarity join </li></li></ul><li>Ranking<br />
    106. 106. Structure<br />Problem definition<br />Types of ambiguities<br />Ranking paradigms<br />Model construction<br />Content-based<br />Structure-based<br />
    107. 107. Ranking – problem definition<br />Query<br /><ul><li>Ambiguities arise when representation is incomplete / imprecise
    108. 108. Ambiguities at the level of
    109. 109. elements (content ambiguity)
    110. 110. structure between elements (structure ambiguity)</li></ul>Matching<br />Data<br />Due to ambiguities in the representation of the information needs and the underlying resources, the results cannot be guaranteed to exactly match the query. Ranking is the problem of determining the degree of matching using some notions of relevance.<br />
    111. 111. Content ambiguity<br />?x ns:knows ?z. ?z ns: works ?v. ?v ns:name “KIT”<br />apartment shared Berlin Alice<br />?y ns:name “Alice”. ?x ns:knows ? y<br />age<br />works<br />34<br />trouble with bob<br />FluidOps<br />Peter<br />sunset.jpg<br />Bob is a good friend of mine. We went to the same university, and also shared an apartment in Berlin in 2008. The trouble with Bob is that he takes much better photos than I do:<br />Beautiful <br />Sunset<br />author<br />title<br />Semantic Search<br />Germany<br />Alice<br />creator<br />author<br />creator<br />knows<br />year<br />Germany<br />2009<br />Bob<br />Thanh<br />knows<br />located<br />works<br />KIT<br />What is meant by “Berlin” in the query?<br />What is meant by “Berlin” in the data?<br />A city with the name Berlin? a person? <br />What is meant by “KIT” in the query?<br />What is meant by “KIT” in the data?<br />A research group? a university? a location?<br />
    112. 112. Structure ambiguity<br />?x ns:knows ?z. ?z ns: works ?v. ?v ns:name “KIT”<br />apartment shared Berlin Alice<br />?y ns:name “Alice”. ?x ns:knows ? y<br />age<br />works<br />34<br />trouble with bob<br />FluidOps<br />Peter<br />sunset.jpg<br />Bob is a good friend of mine. We went to the same university, and also shared an apartment in Berlin in 2008. The trouble with Bob is that he takes much better photos than I do:<br />Beautiful <br />Sunset<br />author<br />title<br />Semantic Search<br />Germany<br />Alice<br />creator<br />author<br />creator<br />knows<br />year<br />Germany<br />2009<br />Bob<br />Thanh<br />knows<br />located<br />works<br />KIT<br />What is the connection between “Berlin” and “Alice”? <br />Friend? Co-worker? <br />What is meant by “works”? <br />Works at? employed? <br />
    113. 113. Ambiguity<br />Recall: query processing is matching at the level of syntax and semantics <br />Ambiguities arise when data or query allow for multiple interpretations, i.e. multiple matches<br />Syntactic, e.g. works vs. works at<br />Semantic, e.g. works vs. employ<br />“Aboutness”, i.e., contain some elements which represent the correct interpretation <br />Ambiguities arise when matching elements of different granularities<br />Does icontains the interpretation for j, given some part(s) of i (syntactically/semantically) match j<br />E.g. Berlin vs. “…we went to the same university, and also, we shared an apartment in Berlin in 2008…”<br />Strictly speaking, ranking is performed after syntactic / semantic matching is done!<br />
    114. 114. Features: What to use to deal with ambiguities?<br />What is meant by “Berlin”? What is the connection between “Berlin” and “Alice”? <br />Content features<br />Frequencies of terms: d more likely to be “about” a query term k when d more often, mentions k (probabilistic IR)<br />Co-occurrences: terms K that often co-occur form a contextual interpretation, i.e., topics (cluster hypothesis) <br />Structure features<br />Consider relevance at level of fields<br />Linked-based popularity<br />
    115. 115. Ranking paradigms<br />Explicit relevance model <br />Foundation: probability ranking principle<br />Ranking results by the posterior probability (odds) of being observed in the relevant class:<br />P(w|R) varies in different approaches, e.g., binary independence model, 2-poisson model, relevancemodel<br />
    116. 116. Ranking paradigms<br />No explicit notion of relevance: similarity between the query and the document model<br />Vector space model (cosine similarity)<br />Language models (KL divergence)<br />
    117. 117. Model construction<br />How to obtain<br />Relevance models?<br />Weights for query / document terms?<br />Language models for document / queries?<br />
    118. 118. Content-based model construction<br />Document statistics, e.g. <br />Term frequency <br />Document length <br />Collection statistics, e.g. <br />Inverse document frequency<br />Background language models<br /><ul><li> An object is more likely about “Berlin”?
    119. 119. When it contains a relatively high number of mentions of the term “Berlin”
    120. 120. When the number of mentions of this term in the overall collection is relatively low</li></li></ul><li>Structure-based model construction<br />Consider structure of objects during content-based modeling, i.e., to obtain structured content-based model<br />Content-based model for structured objects, documents and for general tuples<br /><ul><li> An object is more likely about “Berlin”?
    121. 121. When one of its (important) fields contains a relatively high number of mentions of the term “Berlin”</li></li></ul><li>Structure-based model construction<br />PageRank<br />Link analysis algorithm<br />Measuring relative importance of nodes<br />Link counts as a vote of support <br />The PageRank of a node recursively depends on the number and PageRank of all nodes that link to it (incoming links)<br />ObjectRank<br />Types and semantics of links vary in structured data setting <br />Authority transfer schema graph specifies connection strengths <br />Recursively compute authority transfer data graph<br /><ul><li> An object about “Berlin” is more important than one another?
    122. 122. When a relatively large number of objects are linked to it</li></li></ul><li>Taxonomy of ranking approaches<br />Explicitly vs. non-explicitly relevance-based<br />Content-based ranking<br />Structure-based ranking<br />Content- and-structure-based ranking<br />
    123. 123. Result Presentation<br />
    124. 124. Search interface<br />Input and output functionality<br />helping the user to formulate complex queries<br />presenting the results in an intelligent manner<br />Semantic Search brings improvements in<br />Query formulation<br />Snippet generation<br />Adaptive and interactive presentation<br />Presentation adapts to the kind of query and results presented<br />Object results can be actionable, e.g. buy this product<br />Aggregated search<br />Grouping similar items, summarizing results in various ways<br />Filtering (facets), possibly across different dimensions<br />Task completion<br />Help the user to fulfill the task by placing the query in a task context<br />
    125. 125. Query interpretation<br />“Snap-to-grid”: find the most likely interpretation of the query given the ontology or a summary of the data<br />See Query Processing <br />Display the system’s interpretation of the user query<br />Offer one or more interpretations, possibly while the user is typing<br />
    126. 126. Example: Freebase suggest<br />
    127. 127. Example: TrueKnowledge<br />Q: “How many people live in Shanghai?”<br />I: What is the population of Shanghai (Shanghainese: Zånhae), the metropolis in eastern China and a direct-controlled municipality of the People's Republic of China? <br />A: The population of Shanghai on November 7th 2010 is approximately 19,300,389. (Extrapolated from a population of 18,884,600 in 2008 and a population of 19,210,000 on June 6th 2010.)<br />
    128. 128. Snippet generation using metadata<br />Yahoo displays enriched search results for pages that contain microformat or RDFa markup using recognized ontologies<br />Displaying data, images, video<br />Example: GoodRelations for products<br />Enhanced results also appear for sites from which we extract information ourselves<br />Also used for generating facets that can be used to restrict search results by object type<br />Example: “Shopping sites” facet for products<br />Documentation and validator for developers<br /><br />Formerly: SearchMonkey allowed developers to customize the result presentation and create new ones for any object type<br />
    129. 129. Example: Yahoo! Enhanced Results<br />Enhanced result with deep links, rating, address.<br />
    130. 130. Automated snippet summarization<br />Generate search result snippets given a query and a search result<br />Penin et al. Snippet Generation for Semantic Web Search Engines, ASWC 2010<br />Search results are ontologies<br />
    131. 131. Example: Facets in Yahoo! Search<br />Click to restrict results to shopping sites<br />
    132. 132. Example: Yahoo! Vertical Intent Search<br />Related actors and movies<br />
    133. 133. Adaptive presentation: semantic bookmarking<br />Extract objects from pages tagged/bookmarked by a user<br />Visualize the extracted objects<br />Tabular display<br />Sorting on attributes<br />Map<br />Tracking changes in data <br />Alert me when the price drops below…<br />Prototype: house search application<br />Delicious profiles<br />Extracting housing data from popular Spanish real-estate sites<br />
    134. 134. Adaptive presentation: semantic bookmarking<br />
    135. 135. Interactive presentation: Time Explorer<br />Deliverable of the LivingKnowledge European Project <br />Not a Yahoo product<br /> <br />Won the HCIR 2010 challenge<br />Tool for understanding current news stories <br />what are the events that led to a particular situation? <br />what are the important entities for a given topic? (people,places,dates, etc.) <br />what entities are important at a given time? How do their relationships change? <br />what are the predictions made of a given topic? <br />
    136. 136. Interactive presentation: Time Explorer<br />Technology<br />Named Entity Recognition (persons, organizations)<br />Temporal expression mining<br />Inverted (sentence and document) index <br />Forward index (archive) for retrieving relevant entities<br />Ranking of both documents and relevant entities<br />Display<br />Two synchronized timelines showing relevant documents and the volume of documents<br />Entity relationships<br />Sentiments (future work)<br />
    137. 137. Example: Time Explorer<br />
    138. 138. Evaluation<br />
    139. 139. Semantic Search evaluation at SemSearch 2010/2011<br />Started at SemSearch 2010<br />Two tasks<br />Entity Search<br />Queries where the user is looking for a single real world object<br />Pound et al. Ad-hoc Object Retrieval in the Web of Data, WWW 2010.<br />List search (new in 2011)<br />Queries where the user is looking for a class of objects<br />Billion Triples Challenge 2009 dataset<br />Evaluated using Amazon’s Mechanical Turk<br />See Halpin et al. Evaluating Ad-Hoc Object Retrieval, IWEST 2010<br />Prizes sponsored by Yahoo! Labs (2011) <br />
    140. 140. Entity Search Track<br />Entity Search<br />retrieval of data related to a single entity<br />Queries<br />Selected from the Search Query Tiny Sample v1.0 dataset, provided by the Yahoo! Webscope program<br />Real web search queries sampled from the US query log of January, 2009<br />Queries asked by at least three different users and with long number sequence removed (privacy reasons)<br />50 selected queries that name an entity explicitly (but may also provide context)<br />Last year: same type of queries, but a mix of Microsoft and Yahoo! Logs<br />
    141. 141. List query track<br />List queries<br />Queries that describe a set of entities<br />The answer is a closed set <br />Relatively small number of possible answers<br />The answer is not likely to change <br />Hand-picked but not hand-written<br />Yahoo! Search logs<br />Queries from the Tiny Sample v1.0 dataset<br />Queries with clicks on Wikipedia<br />TrueKnowledge<br />Recent queries<br />
    142. 142. Data set<br />Same as Billion Triples Challenge 2009 data set<br />Blank nodes are encoded as URIs<br />A data set combining crawls of multiple semantic search engines<br />doesn’t necessarily match the current state of the Web<br />doesn’t necessarily match the coverage of any particular search engine<br />Final dataset<br />
    143. 143. Collecting the results<br />Submissions via<br />NTNU, IIIT Hyderabad, DERI, U of Delaware, Daiict<br />max. 3 submissions per team per track<br />Pooling of results<br />Top 20 results are evaluated<br />Despite validation, still problems<br />e.g. N-Triples encoded URIs, lowercased URIs<br />Collecting triples for each result<br />All triples where the URI is the subject<br />Discarded URIs that didn’t appear as subject<br />Rendering result display<br />Values are clipped at 300 chars (last # or / for object-properties)<br />RDF built-ins shown first<br />Preference to English language values<br />
    144. 144.<br />
    145. 145. Assessment with Amazon Mechanical Turk<br />Evaluation using non-expert judges<br />Paid $0.2 per 12 results<br />Typically done in 1-2 minutes<br />$6-$12 an hour<br />Sponsored by the European SEALS project<br />Each result is evaluated by 5 workers<br />Workers are free to choose how many tasks they do<br />Makes agreement difficult to compute<br />Number of tasks completed per worker (2010)<br />
    146. 146. Evaluation form<br />
    147. 147. Evaluation form<br />
    148. 148. Catching the bad guys<br />Payment can be rejected for workers who try to game the system<br />An explanation is commonly expected, though cheaters rarely complain<br />We opted to mix control questions into the real results<br />Gold-win cases that are known to be perfect<br />Gold-loose cases that are known to be bad<br />Metrics<br />Avg. and std. dev on gold-win and gold-loose results<br />Time to complete<br />
    149. 149. Lessons learned<br />Finding complex queries is not easy<br />Query logs from web search engines contain simple queries<br />Semantic Web search engines are not used by the general public<br />Computing agreement is difficult<br />Each judge evaluates a different number of items<br />Result rendering is critically important<br />The Semantic Web is not necessarily all DBpedia<br />sub30-RES.3 40.6% DBpedia<br />sub31-run3 93.8% Dbpedia<br />Follow up experiments validated the Mechanical Turk approach<br />Blanco et al. Repeatable and Reliable Search System Evaluation using Crowd-Sourcing, SIGIR2011<br />
    150. 150. Next steps<br />Achieve repeatability<br />Simplify our process and publish our tools<br />Automate as much as possible… except the Turks ;)<br />Web site for evaluation<br />Continuous submission?<br />Positioning compared to other evaluation campaigns<br />TREC Entity Track<br />Question Answering over Linked Data<br />SEALS campaigns<br />Join the discussion at<br />
    151. 151. Demos<br />