This document summarizes a presentation on using dictionary searching to analyze texts, rather than relying on single keyword searches. Dictionary searching involves using predefined word lists or "dictionaries" to identify discourses represented by combinations of words. The presentation describes developing scripts to extract dictionaries from literature and visualize the results of dictionary searches conducted on a large newspaper corpus. Challenges addressed include running complex queries on very large datasets and accounting for variations in the newspaper data over time.

1. From
keyword searching
to
discourse mining
Pim Huijnen, Juliette Lonij
Encounters between the Humanities and
Computing, Utrecht University, 18 February 2016

2. The Keyword Problem

3. From: The Barre daily times, January 22 (1913), p. 1

4. Dictionary searching
using extensive and context-specific word
lists (‘dictionaries’) to replace the
contingency of single keywords

5. Why dictionary searching?
…to trace discursive shifts, represented by
combinations of words instead of
individual words
…to trace the persistence of discourses

6. Eugenics in Dutch
newspapers(?)
Query:
maatregel
nageslacht
eigenschap*
aanleg
theorie
bloed
invloed
NOT
eugenetica
eugenetiek
eugeniek
eugenese
ras*

7. Eugenics after eugenics
(Geref. gezinsblad 1965)
(De Tijd 1952)

8. Efficiency before efficiency
Query: "product* machine* verspilling bedrijf goedkoop kwaliteit” 01-01-1890 t/
m 31-12-1940
(1901)
(1906)

9. Developing a script to extract dictionaries
from literature
Experimenting with tools to visualise results
of dictionary searching in
kranten.delpher.nl
KB researcher-in-residence
project

10. Script to extract dictionaries
B
Topic
modeling
TF-IDF
A

11. B
C
Script to extract dictionaries

12. Visualising results of dictionary
searches in Delpher
Use OR-query to search Delpher
Visualise results on the basis of Solr’s relevancy-
score (min. nr. of words)
(arbeid* OR bedrij* OR beheer OR controle* OR factor* OR functie* OR kost* OR leiding* OR loon* OR maatregel*
OR management OR methode* OR model* OR norm* OR organisatie* OR plannen OR prijs OR productie OR
rationeel OR rendement OR reorganisatie OR statistiek OR taylor OR tijd OR werkbesparing OR werkverdeeling)

13. kbresearch.nl/dictionary

14. kbresearch.nl/dictionary

15. Challenges
Running an OR-query of 25+ (or, preferably, more)
words on a 100.000.000+ document dataset
Accounting for particularities of the corpus:
* number of newspaper titles per year
* changes in newspaper titles over the years
* changes in article length over the years
Getting an idea of the exact combination of words in
the visualised results