DIVE+ thesis defense presentation Minyi Cheng

1. DIVE : User Interaction Design with Digital Humanities Thesis defense of Minyi Cheng + https://uncommoncarib-wpengine.netdna-ssl.com/wp-content/uploads/2016/01/Dive-into-Exuma-e1452889402298-1200x656.jp

2. What are we discussing?

3. What are we discussing? The user experience of an exploratory search engine

4. What is exploratory search?

5. Traditional search / helps only when you know what to search for / assumes you understand a topic / accuracy driven / click-through driven

6. Traditional search VS. Exploratory search / helps only when you know what to search for / assumes you understand a topic / accuracy driven / click-through driven / helps when you don’t know what to search for / helps you understand & deepen in a topic / serendipity driven / focused on engagement

7. Use Case:

8. Use Case: DIVE+

9. - Focuses on supporting Digital Humanities Scholars in their development of research questions and explorations - Access to Integrated Online Multimedia collections using Linked Open Data to integrate metadata of various heritage collections - Interactive Exploration & Discovery in Context linking objects to events and entities building automatic storylines (narratives) Use Case: DIVE+

10. What are events? ● a relationship between an object and event with a historical meaning (Van den Akker et al., 2011) ● Things that happen ● Events are defined by the SEM model: actor, place, time and type (De Boer et al., 2015)

11. What are events? ● a relationship between an object and event with a historical meaning (Van den Akker et al., 2011) ● Things that happen ● Events are defined by the SEM model: actor, place, time and type (De Boer et al., 2015) ● Example: Actor: Mohammed Toha Place: Yogyakarta Time: 19 December 1948 http://www.oceansbridge.co.uk/paintings/museums/rijks-museum/small/The-President-the-Vice-Pesident-and-other-leaders-are-exiled-to-Sumatra-1948-1949-Z Z-Mohammad-Toha-Adimidjojo.1050.jpg

12. What are narratives? ● two or more events that are related with each other ● storyline of events Queen Beatrix

13. What are narratives? ● two or more events that are related with each other ● storyline of events Queen Beatrix narrativenarrative The Coronation Relationship with Claus

14. Problem? http://usabilitygeek.com/wp-content/uploads/2015/04/how-difficult-to-design-user-experience.jpg - Usability of the User Interface of DIVE+ for Digital Humanities - Requires complex interaction and support that is not always intuitive - Not able to efficiently use DIVE+ for their research

15. Research questions “How efficient and effective is the exploration based on ‘events’-enriched collections and ‘narratives’ support for Digital Humanities scholars for their (1) understanding of objects, context and collections, and for their (2) discovery of new insights in the object of study?” - what is the role of events and event types in understanding of objects and their context? - what is the role of narratives for the discovery of new insights, objects, links and other information? - how effective and efficient is the DIVE+ Browser (events & narratives) UI/UX for the typical tasks that Digital Humanities scholars perform during their exploratory search?

16. Humanities Scholars Research stages ● Four stages (Collijn, 2016; Melgar Estrada, 2016) ○ Exploring stage ○ Assembling stage ○ Analysis stage ○ Presenting stage

17. Humanities Scholars Research stages ● Four stages (Collijn, 2016; Melgar Estrada, 2016) ○ Exploring stage ○ Assembling stage ○ Analysis stage ○ Presenting stage

18. Humanities Scholars Research stages ● Four stages (Collijn, 2016; Melgar Estrada, 2016) ○ Exploring stage ○ Assembling stage ○ Analysis stage ○ Presenting stage - idea generation - preliminary research - decide if collection is suitable - information gathering - refine a research question/topic

19. Current interface

20. How did we evaluate?

21. User groups ● 18 Digital humanities scholars (DHS) ○ 6 scholars and 5 students ● 11 Media professionals (MP) ○ 5 broadcasting organisations and 6 libraries and archives ● 16 (Digital) humanities students (HS) ○ students from RUG ● 22 Computer science students (CSS) ○ students from VU HCI course

22. User groups ● 18 Digital humanities scholars (DHS) ○ 6 scholars and 5 students ● 11 Media professionals (MP) ○ 5 broadcasting organisations and 6 libraries and archives ● 16 (Digital) humanities students (HS) ○ students from RUG ● 22 Computer science students (CSS) ○ students from VU HCI course Results taken into account

23. Workshops ● CreateSalon Workshop (1 hour) ○ 18 Digital Humanities Scholars ○ 9 May 2017 @ UvA CreateSalon ● MediaNow Workshop (3 hours) ○ 11 Media professionals ○ 24 May 2017 @ Dutch Institute of Sound and Vision ● Data collection I (online) ○ 16 (digital) humanities students ○ May-June 2017 ● Data collection II (online) ○ 22 computer science students ○ June 2017

24. Evaluation Tools ● Simulated work task situation 1 ● Think-aloud protocols ● Questionnaire 2 ● Focus group 1 https://docs.google.com/document/d/176cwCfo21bZIFCOjvZrdD95_TcVslCxAveffMXLK6GI/edit?usp=sharing 2 https://docs.google.com/forms/d/1sL9uYo_Zl56Nb-XFEzIvwV5c1HFEbFuifm9oXR8h-LU/edit?usp=sharing

25. Reflection Methods ● What went well? ○ Workshops was constructed well due to the formed protocols ○ The participants were pro-active with the simulated work task ○ Through the questionnaire and task, we received a lot of feedback/suggestions

26. Reflection Methods ● What went not so well? ○ Due to time constraints, the think aloud protocols were conducted in a group of users. The conductor walked by with a recorder and users had to think their thoughts out loud. ■ Suggestion to improve: conduct the think aloud protocol individually on each user while performing the task. This will give better results, and the focus can be on one person instead divided over the whole group.

27. Where there limitations? ● Shortcoming of time during workshop ○ users were not able to fully experience DIVE+ ● DIVE+ browser ○ overload → error ○ users were not able to perform task

28. Results3 ● Task performance ● Narratives ● Events ● DIVE+ UI Evaluation ● Comparison with other groups - Digital Humanities Scholars (DHS) - Media professionals (MP) - (Digital) humanities students (HS) - Computer science students (CSS) 3 https://docs.google.com/document/d/1Uq4eGAmAD7oWa3zX6fQPWLEI5yVkhj-1I-Ku3SRo8sk/edit?usp=sharing

29. Results DIVE+ Task performance of DHS ● 5/11 participants found an angle

30. Results DIVE+ Task performance of DHS ● 5/11 participants found an angle ● struggled with interface ● which collections?

31. Results DIVE+ Task performance of DHS “I didn’t know how to tailor my search to the collection (of objects & texts) because I didn’t know what the collection was.” - DHS participant 3 ● 5/11 participants found an angle ● struggled with interface ● which collections?

32. Results DIVE+ Task performance of DHS “I didn’t know how to tailor my search to the collection (of objects & texts) because I didn’t know what the collection was.” - DHS participant 3 ● 5/11 participants found an angle ● struggled with interface ● which collections? ● data limited

33. Results DIVE+ Narratives of DHS

34. Results DIVE+ Narratives of DHS ● did not discover different narratives ● would like to see suggested narratives ● exploration path tool seen as useful ● not clear how the exploration path was constructed ● the path was now seen as a list of search results ● would like to interact more with the path (possibility to re-arrange)

35. Results DIVE+ Narratives of DHS ● did not discover different narratives ● would like to see suggested narratives ● exploration path tool seen as useful ● not clear how the exploration path was constructed ● the path was now seen as a list of search results ● would like to interact more with the path (possibility to re-arrange) ● “ The navigational aspect confused me: I struggled with organising a cohesive research path, and found myself wondering why the things are organised in such an order, instead of looking for more material. “ - DHS participant 11

36. Results DIVE+ Events of DHS

37. Results DIVE+ Events of DHS ● did not notice the event characteristics e.g. person, media object etc. ● did not understand the purpose of events ● did not find the different event characteristics useful ● the event characteristics did not narrow down enough the overload of results ● users need more fine-grained event characteristics e.g. radio, tv or image.

38. Results DIVE+ Events of DHS “For me, the media object was way too general. I’m a media scholar, not a historian, so saying there are 400 media object, includes from photos from a museum to.. It is not enough. I need a more granular selection. [the filters] didn’t eliminate the type of media what’s on offer for example, at least in a rough categorization.” - DHS participant 3 ● did not notice the event characteristics e.g. person, media object etc. ● did not understand the purpose of events ● did not find the different event characteristics useful ● the event characteristics did not narrow down enough the overload of results ● users need more fine-grained event characteristics e.g. radio, tv or image.

39. Results DIVE+ Evaluation of DHS

40. Results DIVE+ Evaluation of DHS ● The DHS think DIVE+ could be useful, but not sure if they would use DIVE ● There was not enough information about the objects e.g. source or date ● Some buttons on the UI unclear e.g. ‘bookmark’ and ‘save to path’ ● confusion about ‘related searches’, it’s experienced as not related at all

41. Results DIVE+ Evaluation of DHS ● The DHS think DIVE+ could be useful, but not sure if they would use DIVE ● There was not enough information about the objects e.g. source or date ● Some buttons on the UI unclear e.g. ‘bookmark’ and ‘save to path’ ● confusion about ‘related searches’, it’s experienced as not related at all “The ‘related entities’ suggestions were not very helpful, and it was not clear how they were related.” - DHS participant 7

42. Results Comparisons between other groups ● Media professionals ● (digital) humanities students ● computer science students ● Purpose of narratives was unclear among MP, DHS, and HS. ● Almost every group would like to see suggested narratives, except the MP ● All groups missed some characteristics of events ● DHS, MP and HS would like to see more fine-grained filtering among the characteristics. ● The usefulness of event characteristics was experienced negative among 3 out of 4 groups → HS did see the usefulness “It’s unclear why search results receive certain tags. Thereby it’s impossible to acknowledge any value to it.” - MP Participant 2

43. Results Differences between other groups ● Media professionals ● (digital) humanities students ● computer science students ● Humanities scholars see potential in DIVE+, other groups not convinced ● Only DHS mentioned to have more interaction with the exploration path ● HS would like to see the exploration path easier or with instructions to use ● MP need more information about relations between queries and results, and why object have certain labels ● MP want to filter between collections, and media objects ● DHS and MP have their own way of doing research.

44. FINDINGS

45. FINDINGS ● More information is needed ○ collections ○ (meta)data ○ topic descriptions

46. FINDINGS ● More information is needed ○ collections ○ (meta)data ○ topic descriptions ● ‘related searches’ is experienced as not related

47. FINDINGS ● More information is needed ○ collections ○ (meta)data ○ topic descriptions ● ‘related searches’ is experienced as not related ● trouble with event characteristics ○ need more fine-grained sorting of media types

48. FINDINGS ● More information is needed ○ collections ○ (meta)data ○ topic descriptions ● ‘related searches’ is experienced as not related ● trouble with event characteristics ○ need more fine-grained sorting of media types ● exploration path is not optimal in use ○ does not result in narratives ○ the purpose is not clear ○ need more interaction with the tool

49. FINDINGS ● More information is needed ○ collections ○ (meta)data ○ topic descriptions ● ‘related searches’ is experienced as not related ● trouble with event characteristics ○ need more fine-grained sorting of media types ● exploration path is not optimal in use ○ does not result in narratives ○ the purpose is not clear ○ need more interaction with the tool ● unclear what some UI buttons meant

50. Humanities Scholars Research stages and purpose ● Four stages (Collijn, 2016; Melgar Estrada, 2016) ○ Exploring stage ○ Assembling stage ○ Analysis stage ○ Presenting stage - idea generation - preliminary research - decide if collection is suitable - information gathering - refine a research question/topic

51. Recommendations

53. Recommendations Finding: more information is needed about which collections there are in DIVE+, so the digital humanities scholars know how to search. Recommendation: include on the start page what the collections are, and what are in those collections (size, type of objects and more)

54. Current interface

55. Recommendations Finding: more information is needed about which collections there are in DIVE+, so the digital humanities scholars know how to search. Recommendation: include on the start page what the collections are, and what are in those collections (size, type of objects and more)

57. Recommendations Finding: ‘related searches’ is experienced as not related, the digital humanities scholars don’t see the relationship between the searches/results. Recommendation: adapt the back-end algorithm such that the related searches are only related to the current query.

59. Recommendations Finding: the digital humanities scholars experience trouble with the event characteristics. They are having trouble understanding it or see the purpose of it. The purpose of the event characteristics was also experienced troubling, as it does not narrow down enough the results. Recommendation: include a more fine-grained filter of the event characteristics e.g. more filters for Media Object (radio, tv, image)

63. Recommendations Finding: the exploration path is not optimal to use. It’s now seen as a search result list because there is no possibility to form an interesting narrative. Recommendation: add more interaction to the exploration path To form a narrative: ○ there should be the possibility to drag and drop interesting queries ○ there should be the possibility to categorizing interesting queries with colors ○ there should be the possibility to re-name queries with the same title

64. Recommendations Finding: the exploration path is not optimal to use. It’s now seen as a search result list because there is no possibility to form an interesting narrative. Recommendation: add more interaction to the exploration path To form a narrative: ○ there should be the possibility to drag and drop interesting queries ○ there should be the possibility to categorizing interesting queries with colors ○ there should be the possibility to re-name queries with the same title

66. Recommendations Finding: It was unclear what some UI buttons meant, some buttons were experienced as the same purpose e.g. ‘add to path’ and ‘bookmark’. Also the events button was seen as an calendar or date button instead of an events button. Recommendation: Put the title of the button in text together with the button. So the purpose of the button is clear for users.

67. Recommendations Finding: It was unclear what some UI buttons meant, some buttons were experienced as the same purpose e.g. ‘add to path’ and ‘bookmark’. Also the events button was seen as an calendar or date button instead of an events button. Recommendation: Put the title of the button in text together with the button. So the purpose of the button is clear for users.

68. Looking back “How efficient and effective is the exploration based on ‘events’-enriched collections and ‘narratives’ support for Digital Humanities scholars for their (1) understanding of objects, context and collections, and for their (2) discovery of new insights in the object of study?” - what is the role of events and event types in understanding of objects and their context? - what is the role of narratives for the discovery of new insights, objects, links and other information? - how effective and efficient is the DIVE+ Browser (events & narratives) UI/UX for the typical tasks that Digital Humanities scholars perform during their exploratory search?

69. Conclusion “How efficient and effective is the exploration based on ‘events’-enriched collections and ‘narratives’ support for Digital Humanities scholars for their (1) understanding of objects, context and collections, and for their (2) discovery of new insights in the object of study?” - what is the role of events and event types in understanding of objects and their context? → The role of events and event types in understanding is minimum, the events and event types is more used to form a narrative or narrow down results - what is the role of narratives for the discovery of new insights, objects, links and other information? → the current narratives support is not optimal, which leads to a minimum discovery of new insights, objects, links and other information - how effective and efficient is the DIVE+ Browser (events & narratives) UI/UX for the typical tasks that Digital Humanities scholars perform during their exploratory search? → the current UI/UX of the DIVE+ browser is not that effective and efficient for the digital humanities scholars to perform their typical tasks.

70. Conclusion “How efficient and effective is the exploration based on ‘events’-enriched collections and ‘narratives’ support for Digital Humanities scholars for their (1) understanding of objects, context and collections, and for their (2) discovery of new insights in the object of study?” ● The current interface has some shortcomings of DIVE+. ● The ‘events’-enriched collection is experienced as an overload of results and the event types does not help in narrowing down this overload. ● The ‘narratives’ support is not optimal to form a narratives. ● With the given recommendations implemented, improvement will take place in the exploration but more user evaluations are needed in the future.

71. Conclusion “How efficient and effective is the exploration based on ‘events’-enriched collections and ‘narratives’ support for Digital Humanities scholars for their (1) understanding of objects, context and collections, and for their (2) discovery of new insights in the object of study?” ● The current interface has some shortcomings of DIVE+. ● The ‘events’-enriched collection is experienced as an overload of results and the event types does not help in narrowing down this overload. ● The ‘narratives’ support is not optimal to form a narratives. ● With the given recommendations implemented, improvement will take place in the exploration but more user evaluations are needed in the future. Thank you!

72. Questions?