Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)

Supporting the Interpretation of
Enriched Audiovisual Sources through
Temporal Content Exploration
Hugo Huurdeman University of Amsterdam
Liliana Melgar Estrada University of Utrecht, Netherlands Institute for Sound and Vision
Roeland Ordelman Netherlands Institute for Sound and Vision, University of Twente
Julia Noordegraaf University of Amsterdam
Introduction
• Increasingly, images and audiovisual media are being used in
humanities research (Clivaz, 2016)
• However, moving images have been called a "blind medium" for
retrieval (Sandom & Enser, 2001), since it is not possible to search their
content in the same way as we search for text
• Need for sequential viewing & annotation
for transcoding or interpreting the audiovisual contents
• This poses problems for unlocking access to large audiovisual archives
in a way that they are most suitable for their users
• Including academic researchers (Tommasi et al., 2014)
• mediasuite.clariah.nl
• The Media Suite provides access to the rich collections of
various large heritage institutions in the Netherlands
(Ordelman et al., 2018a, 2018b)
Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)
Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)
• Resource Viewer: 1st version (2018)
• ReVI Project – Further enhancing user experience in
Resource Viewer of Media Suite (6 month pilot project)
• September 2018 - February 2019
Resource Viewer (ReVI) Project
• How to support the exploration of different types of content metadata of
audiovisual sources, such as segment information or automatic speech
transcripts (ASR)?
• Our hypothesis: providing temporal visualizations of the content
metadata might enable scholars to identify relevant items and segments
• Facilitating the transition between distant and close reading (van der
Molen, Buitinck & Pieters, 2017)
• More 'scalable' approach to "connecting the distant with the close"
(Denbo and Fraistat 2011)
• This led to various mockups, a functional prototype & their evaluation
…ASR?
• ASR: Automated Speech Recognition (or speech-to-text)
• Currently performed on news and current affairs
subcollections of the Netherlands Institute for Sound
and Vision
• Provides new ways to access collections
• Via the directly spoken words
• Beyond access via previously assigned metadata
(which might be incomplete or succint)
Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)
ASR
Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)
Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)
ReVI Timeline Prototype
• Layered approach to presenting and creating annotations and transcripts
• Temporal display of content metadata & temporal tag clouds
• E.g. useful for overviews (Heimerl et al., 2014) and to detect "busts of
attention" (van der Molen & Pieters, 2017)
• Temporal word clouds were created by:
• 1. Dividing an ASR transcript into a number of predefined segments
• 2. Extracting salient (combinations of) words for each segment using TF-IDF
algorithm and Part-of-Speech tagging
Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)
• Evaluation: Formative usability study (Baxter, 2015)
• 5 scholars
• Media Studies (Television / Film Studies / New Media)
• Experience in using Media Suite
• Using simulated work tasks (Borlund, 2003)
• Two NISV videos relevant for participant, exploratory & focused task
• Initial reactions
• "It's really exciting” (P02) - “Well designed prototype with a
lot of potential” (P01)
• "Some sense of episode” (P01)
• “Really liked how the customizable parts make it possible to
create exactly the categories that I need for my approach!”
(P02)
• “Based on the prototype I really believe I am going to be using
it for my research - that: I am very happy about”
• "More agency than in current Resource Viewer” (P01)
Regarding temporal tag clouds
• Useful for exploratory stages
• “Quick way to see prevalence” (P01)
• “the tag clouds invite to look at the program”
• “Word clouds are nice, give an overview” (P02)
• “When I am still looking at the topic across more videos, in
that stage I would use it definitely” (P02)
• …but less useful for focused stages:
• “Just as an exploratory thing - not every name would show
up in the word cloud” (…) “Wouldn’t want any part of the
strategy to really depend on it” (P02)
• “For conspiracy thinkers: you have stay away before you
get the wrong impression” (P01)
Regarding temporal tag clouds
• Useful for exploratory stages
• “Quick way to see prevalence” (P01)
• “the tag clouds invite to look at the program”
• “Word clouds are nice, give an overview” (P02)
• “When I am still looking at the topic across more videos, in
that stage I would use it definitely” (P02)
• …but less useful for focused stages:
• “Just as an exploratory thing - not every name would show
up in the word cloud” (…) “Wouldn’t want any part of the
strategy to really depend on it” (P02)
• “For conspiracy thinkers: you have stay away before you
get the wrong impression” (P01)
“Positive serendipity” vs “Negative serendipity”
Challenges
• Users’ feedback underlined importance of data and tool criticism
approach (Koolen, van Gorp & Ossenbruggen, 2018) to the
Resource Viewer as part of a digital research environment
1. Issue of maintaining and displaying provenance in terms of data,
given the abundance of metadata created in different ways
2. Inherent quality issues of automatic enrichments
3. Importance of transparency in terms of visualization
• E.g., automatic tag clouds showing words side-by-side ⇨
potentially suggesting links that are not present in the content
1. Transparency: add indications of potentially unreliable ASR sentences
Some immediate steps in the prototype:
Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)
2. Transparency: change layout of automatic tag clouds
“Scalable reading” (Denbo & Fraistat, 2011)
shorter timeframe + mouseover info
Conclusion (1)
• The Timeline Prototype created in the ReVI project showed how
temporal visualizations and advanced annotation features
potentially facilitate looking beyond the picture and into the
temporal aspects of audiovisual content
• This type of content exploration can benefit scholars doing
research with audiovisual sources
• e.g., in media studies, oral history, film studies, and other
disciplines which are increasingly using AV media
• The participants’ feedback showed that transparency of tools
and respecting provenance information about the data is of
utmost importance.
Next Steps
• Final outcomes of ReVI project can serve as inspiration
for improving AV-media-based research tools
• In particular in the phase of exploratory search for
corpus building
• E.g., in Autumn 2019, iterative integration of ReVI
prototype elements into CLARIAH Media Suite
Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)
• CLARIAH
• Everyone in CLARIAH WP5 team
• Design thinking sessions & user study participants
• DH Benelux organizers
Acknowledgments
Literature
• Baxter, Kathy. Understanding Your Users (Interactive Technologies). Elsevier Science. Kindle Edition.
• Borlund, P. (2003). The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research, 8(3).
Retrieved from http://www.informationr.net/ir/8-3/paper152.html
• Bron, M., van Gorp, J., & de Rijke, M. (2016). Media studies research in the data-driven age: How research questions evolve. Journal of the Association
for Information Science and Technology, 67(7), 1535–1554. https://doi.org/10.1002/asi.23458
• Clivaz, C. (2016, June). Keynote address. Presented at the AV in DH workshop at the Digital Humanities Conference, Krakow, Poland. Retrieved from
https://avindhsig.wordpress.com/workshop-2016-krakow/accepted-abstracts/keynote-address-dr-claire-clivaz/
• Denbo, S., and Fraistat, N. (2011). “Diggable Data, Scalable Reading and New Humanities Scholarship”. In 2011 Second International Conference on
Culture and Computing, 169–70. https://doi.org/10.1109/Culture-Computing.2011.49
• Huurdeman, H. C., Wilson, M. L., & Kamps, J. (2016). Active and Passive Utility of Search Interface Features in Different Information Seeking Task
Stages. Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, 3–12. https://doi.org/10.1145/2854946.2854957
• Jatowt, A., Kawai, Y., & Tanaka, K. (2011). Page History Explorer: Visualizing and Comparing Page Histories. IEICE TRANSACTIONS on Information
and Systems, 94(3). Retrieved from http://www.dl.kuis.kyoto-u.ac.jp/~adam/ieice11.pdf
• Kaptein, R., Hiemstra, D., & Kamps, J. (2010). How Different Are Language Models and Word Clouds? In C. Gurrin, Y. He, G. Kazai, U. Kruschwitz, S.
Little, T. Roelleke, … K. van Rijsbergen (Eds.), Advances in Information Retrieval (pp. 556–568). Springer Berlin Heidelberg.
• Koolen, M., van Gorp, J., & van Ossenbruggen, J. (2018). Toward a model for digital tool criticism: Reflection as integrative practice. Digital Scholarship
in the Humanities. https://doi.org/10.1093/llc/fqy048
• Melgar, L., Koolen, M., Huurdeman, H., & Blom, J. (2017). A Process Model of Scholarly Media Annotation. In Proceedings of the 2017 Conference on
Conference Human Information Interaction and Retrieval (pp. 305–308). New York, NY, USA: ACM. https://doi.org/10.1145/3020165.3022139
• Ordelman, R. J. F., & Hessen, A. J. van. (2018). Speech Recognition and Scholarly Research: Usability and Sustainability. CLARIN 2018 Annual
Conference, 163–168. Retrieved from https://research.utwente.nl/en/publications/speech-recognition-and-scholarly-research-usability-and-sustainab
• Ordelman, R., Melgar Estrada, L., Martínez Ortiz, C., Noordegraaf, J., & Blom, J. (2018a). Media Suite: Unlocking Archives for Mixed Media Scholarly
Research. In CLARIN Annual Conference. Pisa, Italy. Retrieved from https://www.clarin.eu/clarin-annual-conference-2018-abstracts
• Ordelman, R., Martínez Ortíz, C., Melgar Estrada, L., Koolen, M., Blom, J., Melder, W., de Boer, V., Karavellas, T., Aroyo, L., Poell, T., Karrouche, N.,
Baaren, E., Wassenaar, J., Noordegraaf, J., Inel, O. (2018b). Challenges in Enabling Mixed Media Scholarly Research with Multi-media Data in a
Sustainable Infrastructure. In Digital Humanities Conference. Mexico: ADHO. Retrieved from https://dh2018.adho.org/en/challenges-in-enabling-
mixed-media-scholarly-research-with-multi-media-data-in-a-sustainable-infrastructure/
• Sandom, C., & Enser, P. G. B. (2001). VIRAMI: visual information retrieval for archival moving imagery. Presented at the International Cultural Heritage
Informatics Meeting, Milano, Italy: Archives & Museum Informatics. Retrieved from http://www.archimuse.com/publishing/ichim01_vol1/sandom.pdf
• Sloetjes, H. (2014). ELAN. The Oxford Handbook of Corpus Phonology. https://doi.org/10.1093/oxfordhb/9780199571932.013.019
• Tommasi, T., Aly, R., McGuinness, K., Chatfield, K., Arandjelovic, R., Parkhi, O., … Tuytelaars, T. (2014). Beyond metadata: searching your archive
based on its audio-visual content. Proceedings of the 2014 International Broadcasting Convention. Presented at the IBC, Amsterdam.
https://doi.org/10.1049/ib.2014.0003
• van der Molen, B., & Pieters, T. (2017). Distant and Close Reading of Dutch Drug Debates in Historical Newspapers: Possibilities and Challenges of Big
Data Analysis in Historical Public Debate Research. In A. K. Somani & G. C. Deka (Eds.), Big Data Analytics: Tools and Technology for Effective
Planning. (pp. 373–390). Retrieved from http://public.eblib.com/choice/publicfullrecord.aspx?p=5116734
• van der Molen, B., Buitinck, L., & Pieters, T. (2017). The leveled approach. Using and evaluating text mining tools AVResearcherXL and Texcavator for
historical research on public perceptions of drugs. ArXiv:1701.00487 [Cs]. Retrieved from http://arxiv.org/abs/1701.00487
Supporting the Interpretation of
Enriched Audiovisual Sources through
Temporal Content Exploration
Hugo Huurdeman University of Amsterdam
Liliana Melgar Estrada University of Utrecht, Netherlands Institute for Sound and Vision
Roeland Ordelman Netherlands Institute for Sound and Vision, University of Twente
Julia Noordegraaf University of Amsterdam
1 of 28

Recommended

Recommending Items in Social Tagging Systems Using Tag and Time Information by
Recommending Items in Social Tagging Systems Using Tag and Time InformationRecommending Items in Social Tagging Systems Using Tag and Time Information
Recommending Items in Social Tagging Systems Using Tag and Time InformationChristoph Trattner
10.9K views21 slides
From Search to Predictions in Tagged Information Spaces by
From Search to Predictions in Tagged Information SpacesFrom Search to Predictions in Tagged Information Spaces
From Search to Predictions in Tagged Information SpacesChristoph Trattner
10.3K views72 slides
Visual Navigation Project - Rethinking Digital Access to Library Materials by
Visual Navigation Project - Rethinking Digital Access to Library MaterialsVisual Navigation Project - Rethinking Digital Access to Library Materials
Visual Navigation Project - Rethinking Digital Access to Library MaterialsVisual Navigation Project
135 views56 slides
Cognitive Models in Recommender Systems by
Cognitive Models in Recommender SystemsCognitive Models in Recommender Systems
Cognitive Models in Recommender SystemsChristoph Trattner
9.5K views49 slides
Je t’aime… moi non plus: reporting on the opportunities, expectations and cha... by
Je t’aime… moi non plus: reporting on the opportunities, expectations and cha...Je t’aime… moi non plus: reporting on the opportunities, expectations and cha...
Je t’aime… moi non plus: reporting on the opportunities, expectations and cha...Christoph Trattner
14.7K views33 slides
Linked Data Quality Assessment – daQ and Luzzu by
Linked Data Quality Assessment – daQ and LuzzuLinked Data Quality Assessment – daQ and Luzzu
Linked Data Quality Assessment – daQ and Luzzujerdeb
1.1K views27 slides

More Related Content

Similar to Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)

Impact the UX of Your Website with Contextual Inquiry by
Impact the UX of Your Website with Contextual InquiryImpact the UX of Your Website with Contextual Inquiry
Impact the UX of Your Website with Contextual InquiryRachel Vacek
1.1K views104 slides
Capturing the Behaviors of the Elusive User: Strategies for Library Ethnography by
Capturing the Behaviors of the Elusive User: Strategies for Library EthnographyCapturing the Behaviors of the Elusive User: Strategies for Library Ethnography
Capturing the Behaviors of the Elusive User: Strategies for Library EthnographyOCLC
354 views37 slides
Capturing the Behaviors of the Elusive User: Strategies for Library Ethnography by
Capturing the Behaviors of the Elusive User: Strategies for Library EthnographyCapturing the Behaviors of the Elusive User: Strategies for Library Ethnography
Capturing the Behaviors of the Elusive User: Strategies for Library EthnographyLynn Connaway
355 views37 slides
Chaos&Order: Using visualization as a means to
 explore large heritage collec... by
Chaos&Order: Using visualization as a means to
 explore large heritage collec...Chaos&Order: Using visualization as a means to
 explore large heritage collec...
Chaos&Order: Using visualization as a means to
 explore large heritage collec...TimelessFuture
1.7K views50 slides
Digital History Projects as Boundary Objects by
Digital History Projects as Boundary ObjectsDigital History Projects as Boundary Objects
Digital History Projects as Boundary ObjectsMaxKemman
644 views17 slides
Outcomes Visual Navigation Project by
Outcomes Visual Navigation ProjectOutcomes Visual Navigation Project
Outcomes Visual Navigation ProjectTimelessFuture
548 views96 slides

Similar to Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)(20)

Impact the UX of Your Website with Contextual Inquiry by Rachel Vacek
Impact the UX of Your Website with Contextual InquiryImpact the UX of Your Website with Contextual Inquiry
Impact the UX of Your Website with Contextual Inquiry
Rachel Vacek1.1K views
Capturing the Behaviors of the Elusive User: Strategies for Library Ethnography by OCLC
Capturing the Behaviors of the Elusive User: Strategies for Library EthnographyCapturing the Behaviors of the Elusive User: Strategies for Library Ethnography
Capturing the Behaviors of the Elusive User: Strategies for Library Ethnography
OCLC354 views
Capturing the Behaviors of the Elusive User: Strategies for Library Ethnography by Lynn Connaway
Capturing the Behaviors of the Elusive User: Strategies for Library EthnographyCapturing the Behaviors of the Elusive User: Strategies for Library Ethnography
Capturing the Behaviors of the Elusive User: Strategies for Library Ethnography
Lynn Connaway355 views
Chaos&Order: Using visualization as a means to
 explore large heritage collec... by TimelessFuture
Chaos&Order: Using visualization as a means to
 explore large heritage collec...Chaos&Order: Using visualization as a means to
 explore large heritage collec...
Chaos&Order: Using visualization as a means to
 explore large heritage collec...
TimelessFuture1.7K views
Digital History Projects as Boundary Objects by MaxKemman
Digital History Projects as Boundary ObjectsDigital History Projects as Boundary Objects
Digital History Projects as Boundary Objects
MaxKemman644 views
Outcomes Visual Navigation Project by TimelessFuture
Outcomes Visual Navigation ProjectOutcomes Visual Navigation Project
Outcomes Visual Navigation Project
TimelessFuture548 views
User experience at Imperial: a case study of qualitative approaches to Primo ... by Andrew Preater
User experience at Imperial: a case study of qualitative approaches to Primo ...User experience at Imperial: a case study of qualitative approaches to Primo ...
User experience at Imperial: a case study of qualitative approaches to Primo ...
Andrew Preater1.3K views
Studying information behavior: The Many Faces of Digital Visitors and Residents by Lynn Connaway
Studying information behavior: The Many Faces of Digital Visitors and ResidentsStudying information behavior: The Many Faces of Digital Visitors and Residents
Studying information behavior: The Many Faces of Digital Visitors and Residents
Lynn Connaway363 views
Studying information behavior: The Many Faces of Digital Visitors and Residents by OCLC
Studying information behavior: The Many Faces of Digital Visitors and ResidentsStudying information behavior: The Many Faces of Digital Visitors and Residents
Studying information behavior: The Many Faces of Digital Visitors and Residents
OCLC159 views
Research culture presentation Sept 4, 2013 by Shawna Reibling
Research culture presentation Sept 4, 2013Research culture presentation Sept 4, 2013
Research culture presentation Sept 4, 2013
Shawna Reibling1.3K views
Audiovisual archives and digital humanities by Johan Oomen
Audiovisual archives and digital humanitiesAudiovisual archives and digital humanities
Audiovisual archives and digital humanities
Johan Oomen1.5K views
20190221 Algorithmic transparency and accountability in practice by Brussels Legal Hackers
20190221 Algorithmic transparency and accountability in practice20190221 Algorithmic transparency and accountability in practice
20190221 Algorithmic transparency and accountability in practice
Digital History Projects as Boundary Objects by MaxKemman
Digital History Projects as Boundary ObjectsDigital History Projects as Boundary Objects
Digital History Projects as Boundary Objects
MaxKemman497 views
Conclusions and Learned Lessons - Visual Navigation Project Outcomes - by Visual Navigation Project
Conclusions and Learned Lessons - Visual Navigation Project Outcomes - Conclusions and Learned Lessons - Visual Navigation Project Outcomes -
Conclusions and Learned Lessons - Visual Navigation Project Outcomes -
Digital Humanities Research by elli.m
Digital Humanities ResearchDigital Humanities Research
Digital Humanities Research
elli.m2.5K views
Lessons Learned from a Digital Tool Criticism Workshop by Marijn Koolen
Lessons Learned from a Digital Tool Criticism WorkshopLessons Learned from a Digital Tool Criticism Workshop
Lessons Learned from a Digital Tool Criticism Workshop
Marijn Koolen76 views
Digital Humanities in Practice, DHC 2012 by Monica Bulger
Digital Humanities in Practice, DHC 2012Digital Humanities in Practice, DHC 2012
Digital Humanities in Practice, DHC 2012
Monica Bulger3.3K views
Contextual Inquiry: How Ethnographic Research can Impact the UX of Your Website by Rachel Vacek
Contextual Inquiry: How Ethnographic Research can Impact the UX of Your WebsiteContextual Inquiry: How Ethnographic Research can Impact the UX of Your Website
Contextual Inquiry: How Ethnographic Research can Impact the UX of Your Website
Rachel Vacek4.9K views
Epistemic Encounters: Interdisciplinary collaboration in developing virtual r... by Smiljana Antonijevic
Epistemic Encounters: Interdisciplinary collaboration in developing virtual r...Epistemic Encounters: Interdisciplinary collaboration in developing virtual r...
Epistemic Encounters: Interdisciplinary collaboration in developing virtual r...

More from TimelessFuture

Webmapping: maps for presentation, exploration & analysis by
Webmapping: maps for presentation, exploration & analysisWebmapping: maps for presentation, exploration & analysis
Webmapping: maps for presentation, exploration & analysisTimelessFuture
142 views75 slides
Experiential Interfaces: 

3D reconstructions as entry points for exploration... by
Experiential Interfaces: 

3D reconstructions as entry points for exploration...Experiential Interfaces: 

3D reconstructions as entry points for exploration...
Experiential Interfaces: 

3D reconstructions as entry points for exploration...TimelessFuture
99 views38 slides
Step inside the Image: 

Interpretative Interfaces for 
3D Historical Content by
Step inside the Image: 

Interpretative Interfaces for 
3D Historical ContentStep inside the Image: 

Interpretative Interfaces for 
3D Historical Content
Step inside the Image: 

Interpretative Interfaces for 
3D Historical ContentTimelessFuture
77 views66 slides
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Info... by
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Info...The Multi-Stage Experience: the Simulated Work Task Approach to Studying Info...
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Info...TimelessFuture
408 views82 slides
Op Ontdekkingsreis door het KB Webarchief - Exploratieve Visualisatie in een ... by
Op Ontdekkingsreis door het KB Webarchief - Exploratieve Visualisatie in een ...Op Ontdekkingsreis door het KB Webarchief - Exploratieve Visualisatie in een ...
Op Ontdekkingsreis door het KB Webarchief - Exploratieve Visualisatie in een ...TimelessFuture
488 views22 slides
Visualization Lecture - Clariah Summer School 2018 by
Visualization Lecture - Clariah Summer School 2018Visualization Lecture - Clariah Summer School 2018
Visualization Lecture - Clariah Summer School 2018TimelessFuture
559 views102 slides

More from TimelessFuture(20)

Webmapping: maps for presentation, exploration & analysis by TimelessFuture
Webmapping: maps for presentation, exploration & analysisWebmapping: maps for presentation, exploration & analysis
Webmapping: maps for presentation, exploration & analysis
TimelessFuture142 views
Experiential Interfaces: 

3D reconstructions as entry points for exploration... by TimelessFuture
Experiential Interfaces: 

3D reconstructions as entry points for exploration...Experiential Interfaces: 

3D reconstructions as entry points for exploration...
Experiential Interfaces: 

3D reconstructions as entry points for exploration...
TimelessFuture99 views
Step inside the Image: 

Interpretative Interfaces for 
3D Historical Content by TimelessFuture
Step inside the Image: 

Interpretative Interfaces for 
3D Historical ContentStep inside the Image: 

Interpretative Interfaces for 
3D Historical Content
Step inside the Image: 

Interpretative Interfaces for 
3D Historical Content
TimelessFuture77 views
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Info... by TimelessFuture
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Info...The Multi-Stage Experience: the Simulated Work Task Approach to Studying Info...
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Info...
TimelessFuture408 views
Op Ontdekkingsreis door het KB Webarchief - Exploratieve Visualisatie in een ... by TimelessFuture
Op Ontdekkingsreis door het KB Webarchief - Exploratieve Visualisatie in een ...Op Ontdekkingsreis door het KB Webarchief - Exploratieve Visualisatie in een ...
Op Ontdekkingsreis door het KB Webarchief - Exploratieve Visualisatie in een ...
TimelessFuture488 views
Visualization Lecture - Clariah Summer School 2018 by TimelessFuture
Visualization Lecture - Clariah Summer School 2018Visualization Lecture - Clariah Summer School 2018
Visualization Lecture - Clariah Summer School 2018
TimelessFuture559 views
KNVI 2017: De collectie in een ander licht - Creatieve inzet van nieuwe techn... by TimelessFuture
KNVI 2017: De collectie in een ander licht - Creatieve inzet van nieuwe techn...KNVI 2017: De collectie in een ander licht - Creatieve inzet van nieuwe techn...
KNVI 2017: De collectie in een ander licht - Creatieve inzet van nieuwe techn...
TimelessFuture1.6K views
Workshop: Inspirational Journeys - Challenges and Solutions for Visual Naviga... by TimelessFuture
Workshop: Inspirational Journeys - Challenges and Solutions for Visual Naviga...Workshop: Inspirational Journeys - Challenges and Solutions for Visual Naviga...
Workshop: Inspirational Journeys - Challenges and Solutions for Visual Naviga...
TimelessFuture417 views
“More than Meets the Eye” - Analyzing the Success of User Queries in Oria by TimelessFuture
“More than Meets the Eye” - Analyzing the Success of User Queries in Oria“More than Meets the Eye” - Analyzing the Success of User Queries in Oria
“More than Meets the Eye” - Analyzing the Success of User Queries in Oria
TimelessFuture1.5K views
Not available, or not found? Lessons from user queries in the Oria catalog at... by TimelessFuture
Not available, or not found? Lessons from user queries in the Oria catalog at...Not available, or not found? Lessons from user queries in the Oria catalog at...
Not available, or not found? Lessons from user queries in the Oria catalog at...
TimelessFuture767 views
Webarchief & Wetenschap (Dutch) by TimelessFuture
Webarchief & Wetenschap (Dutch)Webarchief & Wetenschap (Dutch)
Webarchief & Wetenschap (Dutch)
TimelessFuture444 views
From Exploration to Construction
 - How to Support the Complex Dynamics of In... by TimelessFuture
From Exploration to Construction
 - How to Support the Complex Dynamics of In...From Exploration to Construction
 - How to Support the Complex Dynamics of In...
From Exploration to Construction
 - How to Support the Complex Dynamics of In...
TimelessFuture390 views
Towards Multidimensional Web Archive Access (IIPC 2016) by TimelessFuture
Towards Multidimensional Web Archive Access (IIPC 2016)Towards Multidimensional Web Archive Access (IIPC 2016)
Towards Multidimensional Web Archive Access (IIPC 2016)
TimelessFuture992 views
Active & Passive Utility of Search Interface Features in different Informatio... by TimelessFuture
Active & Passive Utility of Search Interface Features in different Informatio...Active & Passive Utility of Search Interface Features in different Informatio...
Active & Passive Utility of Search Interface Features in different Informatio...
TimelessFuture784 views
Supporting the Process - Adapting Search Systems To Search Stages (ECIL15) by TimelessFuture
Supporting the Process - Adapting Search Systems To Search Stages (ECIL15)Supporting the Process - Adapting Search Systems To Search Stages (ECIL15)
Supporting the Process - Adapting Search Systems To Search Stages (ECIL15)
TimelessFuture737 views
The Value of Multistage Search Systems for Book Search by TimelessFuture
The Value of Multistage Search Systems for Book SearchThe Value of Multistage Search Systems for Book Search
The Value of Multistage Search Systems for Book Search
TimelessFuture433 views
Towards Research Engines: Supporting Search Stages in Web Archives (2015) by TimelessFuture
Towards Research Engines: Supporting Search Stages in Web Archives (2015)Towards Research Engines: Supporting Search Stages in Web Archives (2015)
Towards Research Engines: Supporting Search Stages in Web Archives (2015)
TimelessFuture1.8K views
WebART: hoe maak je webarchieven bruikbaar voor de wetenschap? (Dutch) by TimelessFuture
WebART: hoe maak je webarchieven bruikbaar voor de wetenschap? (Dutch)WebART: hoe maak je webarchieven bruikbaar voor de wetenschap? (Dutch)
WebART: hoe maak je webarchieven bruikbaar voor de wetenschap? (Dutch)
TimelessFuture2K views
Finding Pages on the Unarchived Web (DL 2014) by TimelessFuture
Finding Pages on the Unarchived Web (DL 2014)Finding Pages on the Unarchived Web (DL 2014)
Finding Pages on the Unarchived Web (DL 2014)
TimelessFuture2.9K views
From multistage information seeking models to multistage search systems (IIiX... by TimelessFuture
From multistage information seeking models to multistage search systems (IIiX...From multistage information seeking models to multistage search systems (IIiX...
From multistage information seeking models to multistage search systems (IIiX...
TimelessFuture931 views

Recently uploaded

12.5.23 Poverty and Precarity.pptx by
12.5.23 Poverty and Precarity.pptx12.5.23 Poverty and Precarity.pptx
12.5.23 Poverty and Precarity.pptxmary850239
381 views30 slides
Creative Restart 2023: Atila Martins - Craft: A Necessity, Not a Choice by
Creative Restart 2023: Atila Martins - Craft: A Necessity, Not a ChoiceCreative Restart 2023: Atila Martins - Craft: A Necessity, Not a Choice
Creative Restart 2023: Atila Martins - Craft: A Necessity, Not a ChoiceTaste
45 views50 slides
NodeJS and ExpressJS.pdf by
NodeJS and ExpressJS.pdfNodeJS and ExpressJS.pdf
NodeJS and ExpressJS.pdfArthyR3
48 views17 slides
The Future of Micro-credentials: Is Small Really Beautiful? by
The Future of Micro-credentials:  Is Small Really Beautiful?The Future of Micro-credentials:  Is Small Really Beautiful?
The Future of Micro-credentials: Is Small Really Beautiful?Mark Brown
75 views35 slides
JQUERY.pdf by
JQUERY.pdfJQUERY.pdf
JQUERY.pdfArthyR3
105 views22 slides
Gross Anatomy of the Liver by
Gross Anatomy of the LiverGross Anatomy of the Liver
Gross Anatomy of the Liverobaje godwin sunday
77 views12 slides

Recently uploaded(20)

12.5.23 Poverty and Precarity.pptx by mary850239
12.5.23 Poverty and Precarity.pptx12.5.23 Poverty and Precarity.pptx
12.5.23 Poverty and Precarity.pptx
mary850239381 views
Creative Restart 2023: Atila Martins - Craft: A Necessity, Not a Choice by Taste
Creative Restart 2023: Atila Martins - Craft: A Necessity, Not a ChoiceCreative Restart 2023: Atila Martins - Craft: A Necessity, Not a Choice
Creative Restart 2023: Atila Martins - Craft: A Necessity, Not a Choice
Taste45 views
NodeJS and ExpressJS.pdf by ArthyR3
NodeJS and ExpressJS.pdfNodeJS and ExpressJS.pdf
NodeJS and ExpressJS.pdf
ArthyR348 views
The Future of Micro-credentials: Is Small Really Beautiful? by Mark Brown
The Future of Micro-credentials:  Is Small Really Beautiful?The Future of Micro-credentials:  Is Small Really Beautiful?
The Future of Micro-credentials: Is Small Really Beautiful?
Mark Brown75 views
JQUERY.pdf by ArthyR3
JQUERY.pdfJQUERY.pdf
JQUERY.pdf
ArthyR3105 views
CUNY IT Picciano.pptx by apicciano
CUNY IT Picciano.pptxCUNY IT Picciano.pptx
CUNY IT Picciano.pptx
apicciano64 views
11.30.23A Poverty and Inequality in America.pptx by mary850239
11.30.23A Poverty and Inequality in America.pptx11.30.23A Poverty and Inequality in America.pptx
11.30.23A Poverty and Inequality in America.pptx
mary850239130 views
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE... by Nguyen Thanh Tu Collection
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...
INT-244 Topic 6b Confucianism by S Meyer
INT-244 Topic 6b ConfucianismINT-244 Topic 6b Confucianism
INT-244 Topic 6b Confucianism
S Meyer45 views
Class 9 lesson plans by TARIQ KHAN
Class 9 lesson plansClass 9 lesson plans
Class 9 lesson plans
TARIQ KHAN82 views
Career Building in AI - Technologies, Trends and Opportunities by WebStackAcademy
Career Building in AI - Technologies, Trends and OpportunitiesCareer Building in AI - Technologies, Trends and Opportunities
Career Building in AI - Technologies, Trends and Opportunities
WebStackAcademy45 views
Parts of Speech (1).pptx by mhkpreet001
Parts of Speech (1).pptxParts of Speech (1).pptx
Parts of Speech (1).pptx
mhkpreet00146 views
Nelson_RecordStore.pdf by BrynNelson5
Nelson_RecordStore.pdfNelson_RecordStore.pdf
Nelson_RecordStore.pdf
BrynNelson546 views

Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration (DH Benelux) (v2)

  • 1. Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration Hugo Huurdeman University of Amsterdam Liliana Melgar Estrada University of Utrecht, Netherlands Institute for Sound and Vision Roeland Ordelman Netherlands Institute for Sound and Vision, University of Twente Julia Noordegraaf University of Amsterdam
  • 2. Introduction • Increasingly, images and audiovisual media are being used in humanities research (Clivaz, 2016) • However, moving images have been called a "blind medium" for retrieval (Sandom & Enser, 2001), since it is not possible to search their content in the same way as we search for text • Need for sequential viewing & annotation for transcoding or interpreting the audiovisual contents • This poses problems for unlocking access to large audiovisual archives in a way that they are most suitable for their users • Including academic researchers (Tommasi et al., 2014)
  • 3. • mediasuite.clariah.nl • The Media Suite provides access to the rich collections of various large heritage institutions in the Netherlands (Ordelman et al., 2018a, 2018b)
  • 6. • Resource Viewer: 1st version (2018) • ReVI Project – Further enhancing user experience in Resource Viewer of Media Suite (6 month pilot project) • September 2018 - February 2019
  • 7. Resource Viewer (ReVI) Project • How to support the exploration of different types of content metadata of audiovisual sources, such as segment information or automatic speech transcripts (ASR)? • Our hypothesis: providing temporal visualizations of the content metadata might enable scholars to identify relevant items and segments • Facilitating the transition between distant and close reading (van der Molen, Buitinck & Pieters, 2017) • More 'scalable' approach to "connecting the distant with the close" (Denbo and Fraistat 2011) • This led to various mockups, a functional prototype & their evaluation
  • 8. …ASR? • ASR: Automated Speech Recognition (or speech-to-text) • Currently performed on news and current affairs subcollections of the Netherlands Institute for Sound and Vision • Provides new ways to access collections • Via the directly spoken words • Beyond access via previously assigned metadata (which might be incomplete or succint)
  • 10. ASR
  • 13. ReVI Timeline Prototype • Layered approach to presenting and creating annotations and transcripts • Temporal display of content metadata & temporal tag clouds • E.g. useful for overviews (Heimerl et al., 2014) and to detect "busts of attention" (van der Molen & Pieters, 2017) • Temporal word clouds were created by: • 1. Dividing an ASR transcript into a number of predefined segments • 2. Extracting salient (combinations of) words for each segment using TF-IDF algorithm and Part-of-Speech tagging
  • 15. • Evaluation: Formative usability study (Baxter, 2015) • 5 scholars • Media Studies (Television / Film Studies / New Media) • Experience in using Media Suite • Using simulated work tasks (Borlund, 2003) • Two NISV videos relevant for participant, exploratory & focused task
  • 16. • Initial reactions • "It's really exciting” (P02) - “Well designed prototype with a lot of potential” (P01) • "Some sense of episode” (P01) • “Really liked how the customizable parts make it possible to create exactly the categories that I need for my approach!” (P02) • “Based on the prototype I really believe I am going to be using it for my research - that: I am very happy about” • "More agency than in current Resource Viewer” (P01)
  • 17. Regarding temporal tag clouds • Useful for exploratory stages • “Quick way to see prevalence” (P01) • “the tag clouds invite to look at the program” • “Word clouds are nice, give an overview” (P02) • “When I am still looking at the topic across more videos, in that stage I would use it definitely” (P02) • …but less useful for focused stages: • “Just as an exploratory thing - not every name would show up in the word cloud” (…) “Wouldn’t want any part of the strategy to really depend on it” (P02) • “For conspiracy thinkers: you have stay away before you get the wrong impression” (P01)
  • 18. Regarding temporal tag clouds • Useful for exploratory stages • “Quick way to see prevalence” (P01) • “the tag clouds invite to look at the program” • “Word clouds are nice, give an overview” (P02) • “When I am still looking at the topic across more videos, in that stage I would use it definitely” (P02) • …but less useful for focused stages: • “Just as an exploratory thing - not every name would show up in the word cloud” (…) “Wouldn’t want any part of the strategy to really depend on it” (P02) • “For conspiracy thinkers: you have stay away before you get the wrong impression” (P01) “Positive serendipity” vs “Negative serendipity”
  • 19. Challenges • Users’ feedback underlined importance of data and tool criticism approach (Koolen, van Gorp & Ossenbruggen, 2018) to the Resource Viewer as part of a digital research environment 1. Issue of maintaining and displaying provenance in terms of data, given the abundance of metadata created in different ways 2. Inherent quality issues of automatic enrichments 3. Importance of transparency in terms of visualization • E.g., automatic tag clouds showing words side-by-side ⇨ potentially suggesting links that are not present in the content
  • 20. 1. Transparency: add indications of potentially unreliable ASR sentences Some immediate steps in the prototype:
  • 22. 2. Transparency: change layout of automatic tag clouds “Scalable reading” (Denbo & Fraistat, 2011) shorter timeframe + mouseover info
  • 23. Conclusion (1) • The Timeline Prototype created in the ReVI project showed how temporal visualizations and advanced annotation features potentially facilitate looking beyond the picture and into the temporal aspects of audiovisual content • This type of content exploration can benefit scholars doing research with audiovisual sources • e.g., in media studies, oral history, film studies, and other disciplines which are increasingly using AV media • The participants’ feedback showed that transparency of tools and respecting provenance information about the data is of utmost importance.
  • 24. Next Steps • Final outcomes of ReVI project can serve as inspiration for improving AV-media-based research tools • In particular in the phase of exploratory search for corpus building • E.g., in Autumn 2019, iterative integration of ReVI prototype elements into CLARIAH Media Suite
  • 26. • CLARIAH • Everyone in CLARIAH WP5 team • Design thinking sessions & user study participants • DH Benelux organizers Acknowledgments
  • 27. Literature • Baxter, Kathy. Understanding Your Users (Interactive Technologies). Elsevier Science. Kindle Edition. • Borlund, P. (2003). The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research, 8(3). Retrieved from http://www.informationr.net/ir/8-3/paper152.html • Bron, M., van Gorp, J., & de Rijke, M. (2016). Media studies research in the data-driven age: How research questions evolve. Journal of the Association for Information Science and Technology, 67(7), 1535–1554. https://doi.org/10.1002/asi.23458 • Clivaz, C. (2016, June). Keynote address. Presented at the AV in DH workshop at the Digital Humanities Conference, Krakow, Poland. Retrieved from https://avindhsig.wordpress.com/workshop-2016-krakow/accepted-abstracts/keynote-address-dr-claire-clivaz/ • Denbo, S., and Fraistat, N. (2011). “Diggable Data, Scalable Reading and New Humanities Scholarship”. In 2011 Second International Conference on Culture and Computing, 169–70. https://doi.org/10.1109/Culture-Computing.2011.49 • Huurdeman, H. C., Wilson, M. L., & Kamps, J. (2016). Active and Passive Utility of Search Interface Features in Different Information Seeking Task Stages. Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, 3–12. https://doi.org/10.1145/2854946.2854957 • Jatowt, A., Kawai, Y., & Tanaka, K. (2011). Page History Explorer: Visualizing and Comparing Page Histories. IEICE TRANSACTIONS on Information and Systems, 94(3). Retrieved from http://www.dl.kuis.kyoto-u.ac.jp/~adam/ieice11.pdf • Kaptein, R., Hiemstra, D., & Kamps, J. (2010). How Different Are Language Models and Word Clouds? In C. Gurrin, Y. He, G. Kazai, U. Kruschwitz, S. Little, T. Roelleke, … K. van Rijsbergen (Eds.), Advances in Information Retrieval (pp. 556–568). Springer Berlin Heidelberg. • Koolen, M., van Gorp, J., & van Ossenbruggen, J. (2018). Toward a model for digital tool criticism: Reflection as integrative practice. Digital Scholarship in the Humanities. https://doi.org/10.1093/llc/fqy048 • Melgar, L., Koolen, M., Huurdeman, H., & Blom, J. (2017). A Process Model of Scholarly Media Annotation. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (pp. 305–308). New York, NY, USA: ACM. https://doi.org/10.1145/3020165.3022139 • Ordelman, R. J. F., & Hessen, A. J. van. (2018). Speech Recognition and Scholarly Research: Usability and Sustainability. CLARIN 2018 Annual Conference, 163–168. Retrieved from https://research.utwente.nl/en/publications/speech-recognition-and-scholarly-research-usability-and-sustainab • Ordelman, R., Melgar Estrada, L., Martínez Ortiz, C., Noordegraaf, J., & Blom, J. (2018a). Media Suite: Unlocking Archives for Mixed Media Scholarly Research. In CLARIN Annual Conference. Pisa, Italy. Retrieved from https://www.clarin.eu/clarin-annual-conference-2018-abstracts • Ordelman, R., Martínez Ortíz, C., Melgar Estrada, L., Koolen, M., Blom, J., Melder, W., de Boer, V., Karavellas, T., Aroyo, L., Poell, T., Karrouche, N., Baaren, E., Wassenaar, J., Noordegraaf, J., Inel, O. (2018b). Challenges in Enabling Mixed Media Scholarly Research with Multi-media Data in a Sustainable Infrastructure. In Digital Humanities Conference. Mexico: ADHO. Retrieved from https://dh2018.adho.org/en/challenges-in-enabling- mixed-media-scholarly-research-with-multi-media-data-in-a-sustainable-infrastructure/ • Sandom, C., & Enser, P. G. B. (2001). VIRAMI: visual information retrieval for archival moving imagery. Presented at the International Cultural Heritage Informatics Meeting, Milano, Italy: Archives & Museum Informatics. Retrieved from http://www.archimuse.com/publishing/ichim01_vol1/sandom.pdf • Sloetjes, H. (2014). ELAN. The Oxford Handbook of Corpus Phonology. https://doi.org/10.1093/oxfordhb/9780199571932.013.019 • Tommasi, T., Aly, R., McGuinness, K., Chatfield, K., Arandjelovic, R., Parkhi, O., … Tuytelaars, T. (2014). Beyond metadata: searching your archive based on its audio-visual content. Proceedings of the 2014 International Broadcasting Convention. Presented at the IBC, Amsterdam. https://doi.org/10.1049/ib.2014.0003 • van der Molen, B., & Pieters, T. (2017). Distant and Close Reading of Dutch Drug Debates in Historical Newspapers: Possibilities and Challenges of Big Data Analysis in Historical Public Debate Research. In A. K. Somani & G. C. Deka (Eds.), Big Data Analytics: Tools and Technology for Effective Planning. (pp. 373–390). Retrieved from http://public.eblib.com/choice/publicfullrecord.aspx?p=5116734 • van der Molen, B., Buitinck, L., & Pieters, T. (2017). The leveled approach. Using and evaluating text mining tools AVResearcherXL and Texcavator for historical research on public perceptions of drugs. ArXiv:1701.00487 [Cs]. Retrieved from http://arxiv.org/abs/1701.00487
  • 28. Supporting the Interpretation of Enriched Audiovisual Sources through Temporal Content Exploration Hugo Huurdeman University of Amsterdam Liliana Melgar Estrada University of Utrecht, Netherlands Institute for Sound and Vision Roeland Ordelman Netherlands Institute for Sound and Vision, University of Twente Julia Noordegraaf University of Amsterdam