A presentation for the Museum Computer Network conference 2017. Four examples where automated analysis of images and text worked for us, and four where it went wrong, often in an amusing way.
40. Tristan Roddis. Cogapp
Images
• Sourced from Nationalmuseum Sweden
• Using Europeana API for discovery
• 2000 images
• http://labs.cogapp.com/iiif-ml/
Digital agency based in UK. Work internationally. Would love to work for you.
Present some experiments and research we’ve been doing.
Look at three problems that can be solved using automated image analysis: interesting items, color extraction, finding similar images
In all cases: images only. Deliberately not using metadata: manifests only for providing lists of images to analyse.
Present our findings: some positive, full disclosure: some negative
Adrian
We manage the Qatar Digital Library website, that currently has nearly a million scanned pages.
Set ourselves the challenge of automating the process of finding “visually interesting” document pages
Not sure what it is but we know what it’s not (not text, not blank, not bindings...
Over 600 pages
Several dozen manuscripts available, so tens of thousands of images.
And constantly adding more, so not easily achievable by humans.
Impractical
We have some info but not for all, this is on the logical, hard to extract info or missing
Examples
Tristan
First approach we tried was colour analysis
More colours used in illustrations than in plain black script
Imagaa extracts foreground/background colours, “color variance”
Between variance 11 and 17. Hugely mixed.
Gave up: tried a different tack.
Adrian
We manage the Qatar Digital Library website, that currently has nearly a million scanned pages.
Set ourselves the challenge of automating the process of finding “visually interesting” document pages
Not sure what it is but we know what it’s not (not text, not blank, not bindings...
Adrian
Lots of concepts: food, colour, focus ...
Picking up stains
Still not good results
Call this what you want
Tried with one - not very good
Build up two training sets: one with positive results, one with negative results
IIIF Collection picked 10 random archives and then 10 images per archive
Red -> interest
Trained it a couple times (good results)
1 error here updated the set
Example for one archive
And it works
And again
Created a manifest - Mirador
Last point: you can apply this technique of negative/positive training sets to _any_ visual problem. We demonstrated one version of this particular to our collection, but I’m sure you can think of similar questions about your own.
Tristan
Colour extraction is the low-hanging fruit of automated image analysis
Very easy to do via scripts or APIs
“a machine quite literally just said the Met’s collection is full of shit”
Tristan
Colour extraction is the low-hanging fruit of automated image analysis
Very easy to do via scripts or APIs
Paris, Texas vs Paris, France
Adrian
Last problem we looked
Tried two approaches
Collection of algorithms for image processingSkeletonize: each pixel removed if doesn’t break connectivity, then segment colourCensure: feature detector (scale invariant center-surround detector)Daisy: local image descriptor based on gradient orientation histogramsThese are good to find the same image (with noise...)
Mean squared error: square of the diff between px in A and B, sum and divide by nb of pixelsStructural similarity: measure similarity (viewed as a quality measure)
Results as expected
Very different
These 2 are more similar
Then started working with ML
Harder than it looks (ok if you are an ML scientist)
Not quick
Tried something else
Adrian
Last problem we looked
Tried two approaches
Tristan
10 days before conference. Tried different approach.
Tristan: only indexed if confidence value is > 0.75
Elasticsearch with Searchkit interface
Focus on left-hand tags
Different tags
Different tags
Different tags
Adrian
Last problem we looked
Tried two approaches