This document discusses entity-oriented search as an alternative to traditional string-based search. It explains that entity-based search can improve precision and recall by indexing entities with unique IDs rather than text strings. The document introduces Elasticsearch's annotated text field type, which allows text to be annotated with entity links. This enables indexing of both text and entities, as well as highlighted search results that show matching text contexts linked to relevant entities.

1. Annotated text
Searching for “things not strings”

2. aka: Entity oriented search
https://www.springer.com/la/book/9783319939339

3. Text strings = unstructured data
.. he visited JFK later that day

4. Entities = structured data
https://en.wikipedia.org/wiki/John_F._Kennedy
https://en.wikipedia.org/wiki/John_F._Kennedy_International_Airport

5. Strings vs things
35th U.S. President
John Fitzgerald Kennedy
JFK
• Synonymy: thing has multiple string forms (bad for recall)
• Polysemy: - string has multiple meanings (bad for precision)

6. Strings vs things (it gets worse….)
John
JFK
• Query and document text strings are tokenized
(matching all Johns is bad for precision)
U.S
Fitzgerald
President
Kennedy
35th

7. Precision and Recall
Relevant docs
All docs
Search results
John F. Kennedy
High recall Low precision

8. Precision and Recall
Relevant docs
All docs
Search
results
“35th US President”
Low recall High precision

9. Precision and Recall
Relevant docs
All docs
Search results
JFK
High recall Low precision

10. Precision and Recall
Relevant docs
All docs
Search results
https://en.wikipedia.org/wiki/John_F._Kennedy
High recall High precision

11. Why use Entity IDs?
https://en.wikipedia.org/wiki/John_F._Kennedy
• One canonical ID
• One meaning

12. The ideal match:
Documents index entity IDs
Searches use entity IDs
== ==
https://en.wikipedia.org/wiki/John_F._Kennedy_International_Airport

13. How to rewrite free-text searches?
Documents use structured info
Searches use structured criteria
== ==
https://en.wikipedia.org/wiki/John_F._Kennedy_International_Airport
“JFK”

14. How to structure free text in docs?
Documents use structured info
Searches use structured criteria
== ==
https://en.wikipedia.org/wiki/John_F._Kennedy_International_Airport

15. Annotated text = text + entity IDs
.. he visited JFK later that day
https://en.wikipedia.org/wiki/John_F._Kennedy_International_Airport

16. How do I enable it?
• New field type in elasticsearch 6.5
• Enabled by installing a plugin
sudo bin/elasticsearch-plugin install mapper-annotated-text

17. New field type:
"mappings": {
"_doc": {
"properties": {
"my_rich_text_field": {
"type": "annotated_text",
"analyzer": "my_analyzer"
}
}
}
}

18. Markdown-like syntax:
{
“my_rich_text_field" : "Today [elastic](Elastic+Inc.) announced”
}

19. What gets indexed?
POST wikipediaannotated/_analyze
{
"field": "text",
"text": "Today [elastic](Elastic+Inc.&Company) announced"
}
today elastic
Elastic inc.
Company
announced

20. Example use: show entities in context

21. Example use: show entities in context
{
"person": [“Lee Harvey Oswald", "John F. Kennedy"],
"text": "...the [35th President of the United States](John%20F.%20Kennedy) ..."
}
{
"query": {
"term": {"person": "John F. Kennedy"}
},
"highlight": {
"fragment_size": 200,
"require_field_match": false,
"type": "annotated",
"fields": { "text": {} }
}
}
Query
Document

22. Detour: DBPedia ontology
• Things
• Living things
• People
• Office Holder
• John F. Kennedy
• Dwight D. Eisenhower
• DBPedia ontology provides classes for Wikipedia entries
• Can be used to enrich entity IDs with type
See http://mappings.dbpedia.org/server/ontology/classes/
{
"text": “.. General [Dwight D. Eisenhower](Dwight%20D.%20Eisenhower&OfficeHolder)…”
}

23. Example use: find things near strings
{
"text": “.. photographs of [Abraham Lincoln](Abraham%20Lincoln&OfficeHolder)..."
}
Document
{
"span_near": {
"slop": 1,
"in_order": true,
"clauses": [
{"span_term": {"text": {"value": "photographs"}}},
{"span_term": {"text": {"value": "of"}},
{"span_term": {"text": {"value": "OfficeHolder"}}}
]
}
}
Query

24. Further reading
https://www.elastic.co/guide/en/elasticsearch/plugins/6.6/mapper-annotated-text-usage.html
https://www.elastic.co/blog/search-for-things-not-strings-with-the-annotated-text-plugin