Presentation during Digital Humanities Benelux, presenting RemBench and the results of a user study.

1. RemBench: A Digital Workbench for
Rembrandt Research
Suzan Verberne, Centre for Language and Speech Technology
Rudie van Leeuwen, Department of Art History
Radboud University Nijmegen
Digital Humanities Benelux, June 12, 2014

2. About RemBench
• RemBench: a Digital Workbench for Rembrandt Research
• Project financed by CLARIN-NL
• Goal: show the value of online digital data for art history research
- Subgoal 1: connect the metadata of four different databases
- Subgoal 2: build a demonstrator search interface to these databases
• Target group: historians, art historians and other humanities scholars and
students
Suzan Verberne, May 2014

4. Suzan Verberne, May 2014

5. Suzan Verberne, May 2014

6. About RemBench
RemBench connects four existing databases, all
accessible through APIs:
• RKDartists and RKDimages: http://explore.rkd.nl/
• RemDoc, a collection of original documents
related to Rembrandt (1475-1750):
http://www.remdoc.org/
• RUQuest, a library system that provides access
to the collection of the Raboud University Library,
and full-text articles:
http://www.ru.nl/library/vm/ruquest/
Suzan Verberne, May 2014

7. Challenges
1. Mapping the metadata schemes of the four databases onto one set of
metadata categories;
2. Developing the technology that allows the user to search all databases at
once;
3. Designing a user interface that is intuitive for our target group.
Suzan Verberne, May 2014

8. Evaluation: user study
Research questions:
1. How satisfied are target group users with RemBench?
2. Which features are used by the participants and which are not?
3. How efficient are the users in finding answers to Rembrandt research
questions using RemBench?
Suzan Verberne, May 2014

9. User study
Design:
• Participants: 9 students (art history and history, bachelor and master)
• Task: try to find the best possible answer(s) to max. 10 questions related to
Rembrandt, using RemBench
• Observation of user-system interaction
- (Guided) thinking-aloud
- Desktop activity recording
• Post-task questionnaire
Suzan Verberne, May 2014

10. User study
• 61 questions formulated by a Rembrandt expert, e.g.
- Wanneer is de Staalmeesters geschilderd?
- Waar heeft de Nachtwacht (in de loop der tijd) gehangen
(herkomstgeschiedenis)?
- Welke werken van Rembrandt bevinden zich in St. Petersburg?
- Hoe vaak schilderde Rembrandt Christus?
- Is Rembrandts Lezende Vrouw in het Rijksmuseum geschilderd op doek of
paneel?
- Had Rembrandt kleinkinderen; zo ja, waar zijn die geboren?
• Questionnaire per question:
- What was the result of your search?
- How satisfied are you with the answer(s) found? (5-point Likert scale)
- How satisfied are you with the use of RemBench for answering the
question? (5-point Likert scale)
Suzan Verberne, May 2014

11. User study
• The participant is asked to use RemBench exhaustively to find the
answer(s)
• When he/she feels that he tried everything to find the answer(s), he/she
stops his search
• After all questions, he/she is asked to fill in the post-task questionnaire
- Please list the positive aspects of RemBench
- Please list the negative aspects of RemBench
Suzan Verberne, May 2014

12. Example of user interaction
Suzan Verberne, May 2014

13. Quantitative evaluation
# of questions 61
# of questions with at least one user evaluation 54
# of questions with two user evaluations 15
Suzan Verberne, May 2014
Mean difference
between the two
users' evaluations
1.3
1.0

14. Qualitative evaluation
All functionalities are used, except for Refresh (clear all), boolean query options
and fuzzy search.
Positive aspects:
• Clear colours, good font type, nice overview, good layout, easy to use
• Search options are immediately clear
• Searching possible in English and Dutch
• Good filter options (esp. date range and location)
• Works of art, primary and secondary sources combined
• Results ordered by type
• Lots of information
• Reliable sources in the search results
• References to RKD explore with even more information
Suzan Verberne, May 2014

15. Qualitative evaluation
Negative aspects:
• Options missing:
- Specific topics of works of art
- Search for works by one specific artist (Rembrandt)
- Person search should not only cover artists but also family members etc.
• Difficult to find specific answers
• Differences in results between English and Dutch queries
• Sometimes difficult to choose between many search options
• Difficult to get the expected number of results: either too many or too few
• Searching sometimes takes a lot of time
• No explanation on the web site
• Low quality of images
Suzan Verberne, May 2014

16. Recommendations for the final RemBench demonstrator
Suzan Verberne, May 2014
• Add a facet 'Artist/author name' with search field, to make it possible to find
works of 1 specific artist.
• Change the name of the facet 'Person type' to 'Artist/author type‘ and
change the caption ‘Persons’ in the result list to ‘Artists’
• Make it possible to enter specific years in the date filter
• Solve the problem of too many irrelevant results for queries containing the
term Rembrandt
• Make the use of the refresh button more clear
• How to add explanation of use of boolean AND, phrasal queries, and fuzzy
search?

17. Recommendations for usability studies
• In our case, 9 subjects was enough to reach saturation of evaluation points
• Guided thinking-aloud protocol successful. Make extensive notes, no
microphone recordings necessary.
• Screen capture more useful than camera recordings (e.g. with Blueberry
Flashback Express)
• Don’t make the tasks too difficult. Some of our questions where too
complex (esp. “list all...”, “how many...”, requiring exhaustive search)
• Explain elaborately that you are testing the web site, not the subject.
Suzan Verberne, May 2014

18. Try RemBench yourself
• Come to our demo presentation this afternoon
Suzan Verberne, May 2014