Presentation at the Digital Humanities in the Benelux 2018 conference, June 6-8, Amsterdam, Netherlands

1. Marijn Koolen (KNAW Humanities Cluster)
Jasmijn van Gorp (Utrecht University)
Jacco van Ossenbruggen (Centrum Wiskunde & Informatica)
DH Benelux 2018, Amsterdam, 7 June 2018
Lessons Learned from a Digital
Tool Criticism Workshop

3. Setting:
● Many datasets and tools available and used in (digital) humanities research
● Methods (hand)books in humanities primarily aimed at analogue research →
not yet up to date with digital research
Questions:
● Which methodological steps do we have to take to use digital tools and data responsibly?
● To what extent do these steps differ from the steps taken in the pre-digital era?
○ How to align tool impact with user’s intentions
○ How to reach shared understanding of tools in methodology
Goal: develop and test method for performing Digital Tool Criticism
Motivation workshop ‘Digital Tool Criticism’

4. Overview
- Digital Tool Criticism
- Workshop Setup
- Workshop Findings
- Recommendations
- Next Steps

5. Digital Tool Criticism
Definition:
With digital tool criticism we mean the reflection on the role of digital tools in the research
methodology and the evaluation of the suitability of a given digital tool for a specific research
goal.
The aim is to understand the impact of any limitation of the tool on the specific goal, not to improve
a tool’s performance.
That is, ensuring as a scholar to be aware of the impact of a tool on research design, methods,
interpretations and outcomes.
(Koolen, van Gorp & van Ossenbruggen, submitted)

6. ● Two experiments on exploratory research phase
○ Limited number of digital tools (overlap)
○ Broad research topic/theme: migration
● Do experiments in groups, keep track of findings
○ Participants log/write down steps and choices
○ Present and discuss findings
● Meta-discussion
○ General findings and lessons learned
○ Suitability of workshop format and setup
○ Ideas for next steps
Workshop Setup

7. Focus on Exploratory Phase of Research
Model taken from Maxwell (2013)
Trevor Owens (2014): developing research question is iterative process, can start anywhere

8. Guiding Questions
● Starting point: source criticism
○ Method / approach in the humanities and specifically in historical research (cf. Fickers, 2012)
○ Internal source criticism: content of the text
○ External source criticism: metadata of the text (context)
■ Who created the text?
■ What kind of document is it?
■ Where was it made and distributed?
■ When was it made?
■ Why was it made?
● Tool criticism
○ What makes digital tool criticism different from digital source criticism?
○ Question: what is a digital tool (versus a digital source)?

9. Two parts
1. Using explorative tools to refine a research question (90 minutes)
2. Using explorative tools to collect digital sources (90 minutes)
Single overarching research theme:
● Discourse around migration and refugees in broadcast media and politics
Logging process
● Write down reflections/insights/questions on Post-it notes and Google Doc
● Keep track of: goals, question, method, steps
Experiment

10. Tools for Specific Datasets
● Cultural heritage
○ Europeana (europeana.eu)
○ European Library (theeuropeanlibrary.org)
● Broadcast media
○ EuscreenXL (EU) (euscreen.eu)
○ AVResearcherXL (NL) (avresearcher.clariah.beeldengeluid.nl)
○ Plus: Delpher newspaper collection (NL) (delpher.nl)
● Politics:
○ Parliamentary debate search (http://search.politicalmashup.nl/)
○ Talk of Europe (http://www.talkofeurope.eu/data/)
○ Migration Flows - Europe (http://migration.iom.int/europe/)

11. ● Voyant-tools.org (https://voyant-tools.org/)
● OpenRefine (http://openrefine.org/, only available as desktop app)
● Digital Methods Initiative (DMI) tools
(https://wiki.digitalmethods.net/Dmi/ToolDatabase)
● Digital Research Tools Directory (https://dirtdirectory.org/)
● Excel
Generic Tools

12. 19 participants, broad range of backgrounds:
- Digital Heritage, Film studies, History, Linguistics, Literary Studies,
- Media Studies, Science & Technology Studies
- Information Science, Computer Science
6 groups (5 of 3 participants, 1 of 4)
Participants

13. Post-It Notes

14. Poster Presentations

15. Workshop Findings
Three parts:
1. General trends in research process
2. Impact of data and tools on research questions
3. Meta-discussion

16. Findings 1: General Trends in Research Process
We analysed the Google Doc notes of the individual groups
Colour-coded the notes based on 5 aspects:
Research question
Method
Tool
Dataset
Reflection (hard to read: it’s yellow and says “Reflection”)

17. Coding Collaborative Notes

18. Coding Collaborative Notes

19. Research DNA Visualizations

20. Findings 2: Impact of Data and Tools on RQs
● First steps in this exploratory phase:
○ All groups use rapid searches
■ Establish suitability of data/tool for a certain line of inquiry
■ Many dead ends: search reveals data is limited, tool lacks functionality
■ Iteratively adjusted questions, tools and data selections
○ Reflection on tools affects research questions
● Once questions, tools and data are aligned
○ Exploration continues in a specific direction
○ Use same strategy to refine questions and hypotheses
○ In line with findings by Solberg (2012, p. 64).
○ Continuous reflection required to keep alignment!

21. Findings 3: Meta-Discussion
● Collaborative reflection
○ Explicit discussion and questioning encourages reflection
○ Prompts questions that otherwise would not have been asked
○ Note taking helps understand process and choice points
● Data literacy
○ Need to understand how data is structured
○ “Give us the raw data!”: Scholars often want direct access to underlying data
○ Tools and data need better documentation
○ Tools and data are hard to separate
● Workshop format
○ (Collaborative) reflection-in-action increases awareness of tool impact and choices
○ Effective to group tool users and builders

22. Entanglement of Data and Tools

23. Entanglement of Data and Tools
Each step changes the underlying data!

24. We have to reflect on how digital tools organize, access and analyse our materials
Late 19th and early 20th century scholarship was dominated not by big ideas, but by methodological
refinement and disciplinary consolidation.
Denigrated in the later 20th century as unworthy of serious attention by scholars, the 19th and early
20th century, by contrast, took activities like philology, lexicology, and especially bibliography very
seriously.
Serious scholarship was concerned as much with organizing knowledge as it was with
framing knowledge in an ideological construct.
(Scheinfeldt 2008)
Need to Integrate Reflection in Methods

25. Model: Reflection as Integrative Practice

26. For researchers:
- Incorporate digital source, data and tool criticism in research process
- Explicitly ask and answer questions about assumptions, choices, limitations
- Document and share workarounds
- Develop method of experimentation with tool to test functioning
For tool developers and data providers:
- Add an “About” page and documentation on functionalities
- Describe selection criteria and transformations of data sets
Recommendations

27. Next steps
1. Elaborate methodology of DTC
a. More elaborate theoretical foundation of reflexivity in different disciplines and fields
b. Create a method-canvas
c. Use and test in different types of settings
2. DNA-visualisations
a. Create and interpret by researchers (e.g. workshop DH Benelux 2018)
b. Use and test in different types of settings
c. Look at ways to build them into software/tools (automated versus note-taking)
3. Plea for more reflexivity in tool development and training

28. Eijnatten, J. van ., Pieters, T. & Verheul, J., (2013). Big Data for Global History: The Transformative Promise of Digital Humanities.
BMGN - Low Countries Historical Review. 128(4), pp.55–77. DOI: http://doi.org/10.18352/bmgn-lchr.9350
Fickers, A. (2012). Towards a New Digital Historicism? Doing History in the Age of Abundance. View journal, volume 1 (1).
http://orbilu.uni.lu/bitstream/10993/7615/1/4-4-1-PB.pdf
Hitchcock, T. (2013). Confronting the Digital - Or How Academic History Writing Lost the Plot. Cultural and Social History, Volume 10,
Issue 1, pp. 9-23. https://doi.org/10.2752/147800413X13515292098070
Maxwell, J. (2013). Qualitative Research Design: An Interactive Approach, 3rd edition. SAGE publications.
Owens, T. (2014). Where to Start? On Research Questions in The Digital Humanities.
http://www.trevorowens.org/2014/08/where-to-start-on-research-questions-in-the-digital-humanities/
Putnam L. (2016). The Transnational and the Text-Searchable: Digitized Sources and the Shadows They Cast. American Historical
Review, Volume 121, Number 2, pp. 377-402.
Scheinfeldt, T.(2008). Sunset for Ideology, Sunrise for Methodology? Found History.
http://foundhistory.org/2008/03/sunset-for-ideology-sunrise-for-methodology/
Solberg, J. (2012). Googling the Archive: Digital Tools and the Practice of History. Advances in the History of Rethoric, Volume 15, pp.
53-76.
References

29. Thank You!
Questions?
Link to 2017 workshop slides: https://bit.ly/2oHsssK