This document discusses indexing and searching noisy data. Part I provides examples of noisy data from spelling corrections, speech recognition, machine translation and optical character recognition where information is lost. Part II discusses emerging scenarios for scholarly use of noisy metadata including discovery and exploration of large digital collections. Part III argues that metadata mining can enhance accessibility and understanding by integrating different annotation types from manual, automatic and community sources.

1. Indexing and searching
of noisy data
Franciska de Jong
University of Twente Erasmus University
cluster Human Media Interaction Erasmus Studio for e-research
Enschede, The Netherlands Rotterdam, The Netherlands
http://hmi.ewi.utwente.nl/~fdejong
IMPACT Closing Event - The Hague 1

2. Overview
Part I: Noisy data analysis – other examples
Part II: Emerging scenarios of scholarly use
Part III: From noisy (meta)data towards
metadata mining
IMPACT Closing Event - The Hague 2

3. Noisy Channel for Spelling Correction
J&M Figure 5.23
noise: limitations in spelling skills

4. Noisy Channel for Speech Recognition
J&M Figure 9.2
noise: limitations in sound captured

5. Noisy Channel for Machine Translation
J&M Figure 25.15
noise: loss of information through translation

6. Noisy Channel for OCR
J&M Figure 5.23
noise:
loss of information through typesetting/handwriting

7. Decoding spoken audio
• Audio modelling: collect data on the ground
truth for audio segments
• Language modelling: collect data on co-
occurrence s of words
• 100 hours of speech,
• Text data (500 M words)
There is no data like more data
IMPACT Closing Event - The Hague 7

8. After decoding
• multiple hypotheses with varying probabilities
of being correct
• selection from n-best list: errors unavoidable
• post-editing can be an option, but never
without extra costs
– time (editors), money (editing platform)
– complexity of workflow
IMPACT Closing Event - The Hague 8

9. Impact of noise on access tasks
• Content/metadata with a certain amount of
errors
• Search with reduced accuracy:
– missed hits (false negatives)
– incorrect hits (false positive; ‘noise’)
• Noisy data less suited for presentation layer
– pdf versus ascii
– original audio versus transcript; alternatives: word
clouds, related content
IMPACT Closing Event - The Hague 9

10. Access to interviews: transcript generation
metadata multimedia
interview
archive
speech/
speaker speech
non-speech result
detection recognition
detection presentation
automatic speech transcription
users:
transcripts with time stamps search general public,
and semantic annotations engine archivists,
researchers
query
summarization text mining tagging
automatic metadata extraction

11. Optimization Strategies (1)
• Error correction: post-editing, better
recognition
• Improved recognition
– typically effective for core collections (WER below
20%)
– less effective for the long tail
Case: interviews with Willem Frederik Hermans
• With models for news: 81% WER
• Aim: reduction to around 60%
IMPACT Closing Event - The Hague 11

12. Optimization Strategies (2)
• Dedicated /task-specific evaluation
– for seach applications errors in function words are
less critical than errors in e.g. names of persons
and locations
• Dedicated weigthing schemes for search tasks
– assign confidences scores to fragments found and
rerank search results accordingly
IMPACT Closing Event - The Hague 12

13. Access to interviews: support for users
metadata multimedia
interview
archive
speech/
speaker speech
non-speech result
detection recognition
detection presentation
automatic speech transcription
users:
transcripts with time stamps search general public,
and semantic annotations engine archivists,
researchers
query
summarization text mining tagging
automatic metadata extraction

14. • Part II: Emerging scenarios of scholarly use
IMPACT Closing Event - The Hague 14

15. DLs and knowledge discovery
• Focus of attention for analysis is no longer the
document alone.
• Room for statistical methods to analyse entire
collections, archives, libraries.
• Tools that automatically detect and capture
various semantic layers and feed the patterns
found back into the metadata structures.
• Discovery versus item finding: room for
serendipity and data-driven content
exploration. IMPACT Closing Event - The Hague 15

16. Paradigm evolution
Science Information
examples studies examples
direct obervation interpretation/ decoding of
Experimental texts
work
E = mc2 S → NP VP
Theoretical
a2 + b2 = c2 Principle of
modeling Compositionality
change GIS for visualisation of
Computational mobility patterns
simulation
modeling text-mining: cross-
particle physics, document entity linking for
Data-intensive cultural heritage libraries
astronomy
computing rule-based parsing of large
IMPACT Closing Event - The Hague corpora (typology studies))
16

17. More than search: metadata
extraction
• For large-scale digital (distributed) collections the
potential added value of automatically generated
metadata is becoming more and more apparent.
• Automatic content labeling:
– not just a matter of speeding up the annotation process and
enlarging the scope of analysis, also
– starting point for generating annotation layers at collection
level , and
– basis for link structures for all kinds of semantic aspects of
content, such as chronological trends, topic shifts, style and
authenticity.
– potentially noisy IMPACT Closing Event - The Hague 17

18. “Multi”-issues for DL metadata (1)
• Multi-layer
– beyond tomb stone: content description at
fragment level (full text, full content, etc.)
– free text annotation versus thesaurus-based
labeling
• Multiple media formats
– text, text, text
– spoken audio, video, still images, music, scores,
umerical data, sensor data, sensus data, etc.
IMPACT Closing Event - The Hague 18

19. Multi-issues for DL metadata (2)
• Multiple perspectives
– cover more than local context
– cover more than one domain perspective
– cover more than one language
• Multiple values due to uncertainty
– multiple human annotators
– automatic labeling extracted from potentially
noisy data
– dynamics in collection/context
IMPACT Closing Event - The Hague 19

20. Scholarly use
• Comparative perspective
– Quantitative and qualitative issues
• Need for enhanced content presentation:
– Multiple layers
– Links to context
– Links to related content
• Emerging methodological shift
– Enhanced collection exploration (think of Google
n-grams)
IMPACT Closing Event - The Hague 20

21. Part III
From noisy data/metadata towards metadata
mining
IMPACT Closing Event - The Hague 21

22. Metadata mining: crucial steps
• Treat all annotation types (classical
metadata, automatically extracted
metadata, scholarly annotation, community
tagging) as assets.
• Learn how to integrate the various types and
layers to enhance accessibility and to be able to
exploit the knowledge captured in metadata
– Exploiting manual annotation for machine learning
training
– Detection of collection-level semantic features
– Innovative interface Event - The Hague
IMPACT Closing
and interaction design 22

23. What can metadata mining bring?
• Quality added to metadata for increased accessibility
of content:
– structured search (full text + classification-based)
– navigation across collections, rich presentation layers
• Increased insight in relations between data
collections (across media types, languages, etc.)
• Increased understanding of knowledge production
as captured by metadata and annotation processing
• Support for capturing the essence of association and
analogy.
There is no data like metadata!
IMPACT Closing Event - The Hague 23

24. Issues for metadata models
Old
• annotation interoperability (e.g., metadata
integration for content annotated with coding
tools such as thesauri and ontologies)
New
• how to capture fuzziness and uncertainty coming
from multiple sources and/or statistical
processing
• coding of change over time (e.g., metadata for
the dynamics of temporal and geo-spatial details)
IMPACT Closing Event - The Hague 24

25. Issues for scholarly users
Individual level
• Learn to deal with imperfection
• Understand the limitations of technological
innovation
Community level
• Stay tuned with developers
• Organize methodology teaching
• Study emerging practises
• Share success stories
IMPACT Closing Event - The Hague 25

26. Issues for developers
• Learn about scholarly practises
• Stay tuned with users during the entire
process
• Organize structured feedback loops
• Study best practises
• Share responsibility for centers of expertise
IMPACT Closing Event - The Hague 26

27. Issues for e-humanities
• e-humanities is e-research
• multiple media, multiple patforms
• keep connecting !
IMPACT Closing Event - The Hague 27

28. Contact
• email:
f.m.g.dejong@utwente.nl or
fdejong@ese.eur.nl
• url: http://hmi.ewi.utwente.nl/~fdejong
IMPACT Closing Event - The Hague 28