Applying Computer
Vision to Art History
John Resig - http://ejohn.org/research/
Visiting Researcher, Ritsumeikan University

What “Works” Today
Reading license plates, zip codes, checks

Optical Character
Recognition
• Tesseract
• https://
code.google.com/
p/tesseract-ocr/

What “Works” Today
Face recognition

Face Matching
• OpenBR
• http://openbiometrics.org/

What “Works” Today
Recognition of flat, textured, objects

Computer Vision
• Unsupervised (requires no labeling):
• Comparing an entire image
• Categorizing an image
• Supervised (requires labeling):
• Finding parts of an image
• Finding and categorizing parts of an image

Unsupervised Training
• Requires little-to-no prepping of data
• Can just give the tool a set of images and
have it produce results
• Extremely easy to get started, results aren’t
always as interesting.

Supervised Training
• Need lots of training data
• Needs to be pre-selected/categorized
• Think: Thousands of images.
• If your collection is smaller than this, perhaps
it may not benefit.
• Or you may need crowd sourcing.
• Results can be more interesting:
• “Find all the people in this image”

Image Similarity
• imgSeek (Open Source)
• http://www.imgseek.net/
• TinEye’s MatchEngine
• http://services.tineye.com/MatchEngine
• Both are completely unsupervised. No
training data is required.

imgSeek
• Compares entire
image.
• Finds similar images,
not exact.
• Does not find parts of
an image.
• Color sensitive.

• Compares
portions of
images.
• Finds exact
matches.
• Finds images
inside other
images.
• Color
Ukiyo-e.org (Using MatchEngine)
insensitive.

Anonymous Italian Art (Frick PhotoArchive)
Using MatchEngine

Conservation

Copies

Image Portion
Partial Image vs. Much Larger Image

Image Categorization
• Deep neural networks
• Requires minimal categorization
• Very little user-input required.
• Ersatz
• http://ersatz1.com/

Requires a lot of training
data (thousands of images)
Takes a lot of computers
(Not cheap)
The less categories you
have, the better.

General Computer
Vision
• Ideal for some supervised training problems
• CCV
• http://libccv.org/
• https://github.com/liuliu/ccv
• OpenCV
• http://opencv.org/

Object Detection

Training Caveats
• Requires thousands (if not 10s of
thousands) of images
• Will take at least a week to run on a very
powerful computer
• Does not work with 3D objects

Learn More about
Computer Vision
• Learn more:
• http://cs.brown.edu/courses/csci1430/
• Just published paper on Frick Computer
Vision work:
• http://ejohn.org/research/