Approaches to automated metadata extraction : FixRep Project

UKOLN is supported by: Approaches to automated metadata extraction : FixRep Project Emma Tonkin [email_address] www.bath.ac.uk

Wouldn't it be nice if...

...computers could author our metadata for us, thus saving a lot of hassle?

Mechanical metadata extraction vs manual metadata input

But...

Automated tools are fallible

There's never quite enough information available

Templates change, different domains have different standards

In short, computers are often wrong

and so are people

Hybrid approach:

Get what metadata you can

Ask the user to check and clean it if necessary

Philosophy:

If the computer gets it wrong, we can fix it later

The 'half a loaf' hypothesis

Wouldn’t it be nice if…

… computers could fix our metadata for us?

Or, more realistically, help us do this work for ourselves.

All about ‘fixing it later’, doing what we can with what we have

Automated metadata extraction + metadata consistency assessment

Metadata generation, evaluation, characterisation: enabling metadata triage

Challenges in automated metadata extraction

Manual metadata generation

Metadata extraction in brief

Practical use as part of a repository deposit workflow

A user study comparing manual and hybrid input

Towards metadata triage

Whatever can go wrong...

PDFs can be:

Encrypted

Corrupted

Oddly encoded

An image file without embedded text

Occurrence: ~3-6%

Character sets

Ligatures,

Accents,

Symbols - may not always be extractable from PDFs

Image © Daniel Ullrich

Document formats/layouts

Many possible formats

Some formats not widely supported

Document layouts vary widely, esp. by discipline

Challenges in metadata extraction

Whatever can go wrong... (II)

Function following form – interface

Model adapted to suit unique user needs

Data model incompletely supported

Input validation issues

Systematic error; typos; localisation; encoding; etc.

Lots of past work in characterising manual input errors

Image segmentation, templating & OCR

Working from text

There are a number of possible states (ie. title, author, email, affiliation, abstract)

Directed graph with probabilities

Markov chain: for example,

Title Author Email Affil.

Hidden Markov Model

We cannot directly see these states – only the words

But we can gather statistics on the correlation between the words and the underlying states, to inform guesses as to how the data should be segmented

This may be expressed in terms of an HMM

Bayesian statistics used across term appearance

Example parse

Confirmation-Guided Discovery of First-Order Rules, PETER A. FLACH, NICOLAS LACHICHE

Confirmation-Guided Discovery of First-Order Rules, PETER A. FLACH, NICOLAS LACHICHE

...

Confirmation-Guided Discovery of First-Order Rules , PETER A. FLACH, NICOLAS LACHICHE

Self-correcting, to the extent that the knowledge base grows as new papers are added to the collection

Aims

Adaption of existing interfaces

Enhancing rather than rewriting

Cross-platform, accessible interface

Simple reusable REST API, metadata as DC/XML

Sample interfaces

Architecture

Using what we know...

Question:

“ Do people accept ‘hybrid’ interfaces?”

Here’s one we did earlier…

Hypotheses

Correcting extracted metadata is faster than entering or cutting-and-pasting metadata.

The resulting metadata has fewer errors when the user is provided with already extracted metadata to correct.

User satisfaction with a system is higher if it 'tries' to extract metadata, even if it fails.

Measured: speed and accuracy of entering information manually versus hybrid entry, and qualitatively, the user-satisfaction

Results: Timing

Hybrid faster under both conditions

(Summary of median times)‏

Results: Accuracy

Tested against ground-truth

Keyword accuracy: First keyword listed was relevant for 46% of the publications. The top two were relevant in 66%; the top-5 cover 81% of all desired keywords.

Manual metadata accuracy:

Few users use cut and paste

Capitalisation, punctuation frequently differs

Synonyms are accidentally substituted

Hybrid closer to ground-truth, and more complete, but results not clear-cut.

Qualitative results

Most users preferred the hybrid mode

Most perceived it to be faster than manual data entry

Few believed the hybrid approach to be more accurate; in practice, there was no significant difference in quality between hybrid and manual approach

Both were good - quality

Discussion

Results support hypotheses

People prefer the hybrid interface, and found it more satisfying to use

Accessibility issues exist, but can be overcome

The punchline: one subject actually preferred manual entry because the hybrid system filled in metadata fields that he preferred to leave empty – ie. it did more than the subject wanted!

MetRe prototype (2008)