Suzan Verberne discusses limitations of using Google to search specialized text collections for the humanities. Current search systems only highlight query terms and do not provide guidance through retrieved documents. The presentation proposes extracting important entities, labeling them, and collecting facts from texts to enrich documents with contextual information. This would allow search systems to better guide users through relevant information in response to their specific questions. Technology like named entity recognition and parsing sentences into facts could help realize this vision of more specialized search engines for the humanities.

1. Search engines for the humanities that go beyond Google Suzan Verberne Centre for Language and Speech Technology Radboud University Nijmegen Brainstorm Meeting e-Humanities, March 29 2011 29.03.2011 Suzan Verberne 1

2. Outline Searching with Google Limitations of Google search Searching in text collections Better guidance through texts What technology is needed? 29.03.2011 2 Suzan Verberne

3. Searching with Google 29.03.2011 3 Suzan Verberne

4. Searching with Google 29.03.2011 4 Suzan Verberne

5. How does Google work? Index Relevance model query 29.03.2011 5 Suzan Verberne

6. How does Google work? Google calculates the relevance of web pages using word counts and popularity estimates. So, Google does not ‘understand’ the texts it sees; it can efficiently estimate a document’s relevance based on the words it contains. This is very effective and efficient for retrieving full documents (web pages). 29.03.2011 Suzan Verberne 6

7. Limitations of Google But what if I have a more specific information need: Which books did Multatuli write? How did other writers respond to Multatuli’s work? To which events did Multatuli refer in ‘Max Havelaar’? Then I need a more specialized text collection than the web a search engine that guides me through the retrieved documents. 29.03.2011 Suzan Verberne 7

8. Specialized text collection: DBNL DBNL: The Digital Library of Dutch Literature A website about Dutch literature, language and cultural history. Contains literary texts, secondary literature and additional information such as biographies, portraits and hyperlinks. http://www.dbnl.org 29.03.2011 Suzan Verberne 8

9. Searching in DBNL 29.03.2011 Suzan Verberne 9

13. The document is 6 pages long.

15. Better guidance through texts Step 1: Label important terms and entities in the text Person and place names Book and journal titles Events Other terms of interest This task is called ‘named entity recognition’. It is well developed in the field of computational linguistics. 29.03.2011 Suzan Verberne 15

16. Better guidance through texts 29.03.2011 Suzan Verberne 16 Journal title Book title Person name Person name

17. Better guidance through texts Step 2: collect information about entities in the text: Factual information: what is it and to whom does it relate? Links to external sources (biographies, encyclopaedias) Links to other mentions in the collection Automatically collecting large amounts of factual information is a current research topic in computational linguistics/artificial intelligence. 29.03.2011 Suzan Verberne 17

18. Better guidance through texts 29.03.2011 Suzan Verberne 18 Vaderlandsche Letteroefeningen was meer dan een eeuw lang een van de toonaangevende literair-culturele tijdschriften van Nederland. Verscheen maandelijks. Het laatste nummer kwam van de pers in december 1876. Het doel was in de eerste plaats om de lezers te wijzen op nuttige publicaties. Dat betrof zowel recente werken als boeken die lang geleden verschenen waren en niet meer in de aandacht stonden. http://www.kb.nl/dossiers/vaderlandscheletteroefeningen/

19. Better guidance through texts 29.03.2011 Suzan Verberne 19 Max Havelaar, of de koffij-veilingen der NederlandscheHandel-Maatschappij is een in 1860 gepubliceerde roman van Multatuli. Het boek gaat over een man die probeert te vechten tegen het corrupte regeringssysteem van Nederlands-Indië, en zou grote invloed hebben op de Nederlandse literatuur, maar ook op de Nederlandsekoloniale politiek. Max Havelaar geldt als een van de belangrijkste werken uit de Nederlandse literatuur. http://nl.wikipedia.org/wiki/Max_Havelaar_(boek)

20. Collecting facts from text Dutch Wikipedia: 678.683 articles (March 2011) Articles are categorized by topic Number of articles about Dutch writers: 439 29.03.2011 Suzan Verberne 20

21. Collecting facts from text Split the texts in sentences Analyze the sentences with a parser that indicates the most important syntactic parts of each sentence. Generate (nuclear) facts from the syntactic analysis: SUBJECT | VERB | OBJECT/PREDICATE | COMPLEMENTS Multatuli | write | Max Havelaar | in 1860, in Java 29.03.2011 Suzan Verberne 21

22. Collecting facts from text Hans Dekkers http://nl.wikipedia.org/wiki/Hans_Dekkers_(1954) “Hijschrijftromans, korteverhalen, gedichten en theaterstukken”“He writes novels, short stories, poems and plays” Factoids: hij | schrijven | theaterstukken | | hij | schrijven | gedichten | | hij | schrijven | romans | | hij | schrijven | korteverhalen | | 29.03.2011 Suzan Verberne 22

23. Collecting facts from text P.F. Thomése http://nl.wikipedia.org/wiki/P.F._Thom%C3%A9se “In 1991 en 2003 ontving hij literaire prijzen.”“In 1991 and 2003, hereceivedliteraryawards.” Factoids: hij | ontvangen | literaire prijzen | in 1991, in 2003 | 29.03.2011 Suzan Verberne 23

24. Better guidance through texts Step 3: enrich the text collection with this factual information. When the user clicks one of the labelled terms, the most important factual information will be shown, together with links to sources. 29.03.2011 Suzan Verberne 24

25. Better guidance through texts 29.03.2011 Suzan Verberne 25 Max Havelaar, of de koffij-veilingen der NederlandscheHandel-Maatschappij is een in 1860 gepubliceerde roman van Multatuli. Het boek gaat over een man die probeert te vechten tegen het corrupte regeringssysteem van Nederlands-Indië, en zou grote invloed hebben op de Nederlandse literatuur, maar ook op de Nederlandse koloniale politiek. Max Havelaar geldt als een van de belangrijkste werken uit de Nederlandse literatuur. http://nl.wikipedia.org/wiki/Max_Havelaar_(boek)

26. How to proceed? There are multiple initiatives (also in the Netherlands) to develop the described techniques. Challenges: What are the needs of the target group? Collaboration is essential. Older varieties of Dutch: development of resources and tools is needed (some already exist). User interfacing is very important: specialist knowledge needed. … 29.03.2011 Suzan Verberne 26

27. Thankyou! You can find more information on my web site (Google my name and you will get there) 29.03.2011 27 Suzan Verberne