Your SlideShare is downloading. ×
0
Photo Archive

Image Analysis
John Resig	

March 2014	

http://ejohn.org/research/
Ukiyo-e.org
Frick Art Reference Library
Photoarchive
• Frick Art Reference Library Photoarchive
established in 1920.	

• Collects phot...
Test case:Anonymous
Italian Art Photographs
• Anonymous Italian Art photographs are
small part (~18,000) of over 1.2 milli...
Digital Anonymous
Italian Art
• Early digitization project.	

• At the time there was no in-house digital lab
at the Frick...
Image Analysis
• I wrote custom software to automate
matches using MatchEngine (a TinEye
service).	

• Extremely effective...
MatchEngine
image1.jpg
image2.jpg
image3.jpg
imageN.jpg
Upload Images to MatchEngine
.
.
.
.
.
.
MatchEngine
image1.jpg
image2.jpg
Retrieve Image Matches from MatchEngine
image9.jpg
Matches
image2.jpg
image1.jpg
Matches...
Archive (e.g. Frick Art Reference Library)
Artwork
Image
Image
Image
Artwork
Image
Image
Image
Artwork
Image
Image
Image
A...
Artwork
Confirming a Known Match
ArtworkArtwork
Discovering a New Match
5%
3%
92%
Artworks with no match Confirms a known match Matches a new work
Frick Artworks That Have A Match
Similar Images
Different photo, same work of art.
Similar Images
Different photo, slightly different cropping.
Similar Images
Different photo, dramatically different lighting.
Alternate Images
Partial Image vs. Full Image
Alternate Images
Partial Image vs. Much Larger Image
Alternate Images
Color vs. Black-and-White
Conservation
Conservation
Repairs and possibly removal of later additions.
Conservation
Conservation
Possibly just a drastic difference in lighting?
Conservation
Analysis even spots dramatic conservation work.
Copies
Copies
Copies
Copies
Copies
Copies
Digitization Errors
Arms with Folded Hands
Castello sforzesco, Milan.
Female Head

Gabinetto disegni e stampe
degli Uffizi,...
Still Life with a Bottle, a Plate, a Mortar and
Pestle...

Virgin Enthroned Nursing Christ, Between
Two Saints

S. Biovena...
2%4%
4%
26%
31%
34%
Similar Image Wrong Image Alternate Image Ambiguous Image
Different Works Conservation
Types of New Ma...
Measuring Image
Analysis Efficacy
Artwork
Confirming a Known Match
Artwork
Impossible to Confirm Match
Artworks With Multiple Photos
47%
33%
20%
Failed Match Successful Match No Possible Matches
# of Match Successes and Failures by Type
0
80
160
240
320
Same Similar Alternate Conservation 3D Negative
Success Failure
Similar Images
Similar Images
Alternate Images
Alternate Images
Likelihood of a match by

% of image overlap
0%
25%
50%
75%
100%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95...
Likelihood of an ‘Alternate Image’
match by % of image overlap
9%
91%
Failure Success
86%
14%
Failure Success
Under 30% Ov...
Conservation
3D Works
3D Works
Negative Images
Overall MatchEngine Quality
62%
38%
Failure Success
88%
12%
Failure Success
All Potential Matches No: Under 30% Overlap, 3...
Frederico Zeri
Foundation
• University of Bologna	

• 290,000 photos of artworks from national
archives, private archives,...
Zeri 15thC Italian Art
• Digitized 15th century Italian art	

• Persumably some overlap with the Frick’s
Anonymous Italian...
Image-to-Artwork Ratio
0
10000
20000
30000
40000
Frick Zeri
17,929
14,284
38,511
18,548
Images Artworks
3%
36%
61%
Artworks with no match Confirms a known match Matches a new work
Zeri Artworks That Have A Match
Image Analysis Matches
0
5500
11000
16500
22000
Frick Zeri
738815
20,051
1,053
Same Artwork Different Artwork
Image Detail
Image Detail
Same Piece, Similar Style
Same Piece, Similar Style
Similar/Copies
What Happens When
We Combine Archives?
• Discover new relationships with the other
archive	

• Discover new relationships ...
Frick Photoarchive
Artwork
Image
Image
Image
Artwork
Image
Image
Image
Zeri Archive
Artwork
Image
Image
Image
Artwork
Imag...
Frick-Zeri Image Matches
18,548

Images
38,511 Images
1,117 Images 1,442 Images
Frick-Zeri Artwork Matches
14,284

Artworks
17,929 Artworks
855 Artworks (6%) 938 Artworks (5%)
MergeVerification
• Previously, 77 artworks were manually
matched by a researcher at the Zeri
foundation	

• Can use to ver...
MergeVerification
16%
84%
Same Artwork Failed to Match
Lighting Differences
Tiny Details
Merging Works!
• 84% success confirming known
relationships (with a small sample)	

• Only 6% of computer-discovered matche...
Graph Analysis
Frick 420
420
Zeri 15836
42091
42090
Frick 417
417
42092
?
Finding new Links
• Found 14 new Frick links via Zeri	

• Found 236 new Zeri links via Frick
Frick 347
347
Zeri 12227
33526 33527
Frick 348
348
33525 33528
?
Frick 8131
Zeri 23402
57136
Frick 8132
8132a 8132
57129
57134
Zeri 23405
5713057137
57138
8131a 8131
?
Conclusion
• It becomes possible to merge photo archives,
even though they’re in different langauges.	

• New relationship...
Contact
• John Resig	

• jeresig@gmail.com	

• http://ejohn.org/research/	

• http://ukiyo-e.org/
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
Upcoming SlideShare
Loading in...5
×

NYARC 2014: Frick/Zeri Results

743

Published on

Talk given at the 2014 NYARC annual meeting.

Published in: Art & Photos, Business, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
743
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
2
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "NYARC 2014: Frick/Zeri Results"

  1. 1. Photo Archive
 Image Analysis John Resig March 2014 http://ejohn.org/research/
  2. 2. Ukiyo-e.org
  3. 3. Frick Art Reference Library Photoarchive • Frick Art Reference Library Photoarchive established in 1920. • Collects photographs of works of art from around the world, including sponsoring own photography. • Traditionally image analysis has not been applied to photo archives. • The Frick Photoarchive was interested in exploring image analysis more after seeing
 Ukiyo-e.org.
  4. 4. Test case:Anonymous Italian Art Photographs • Anonymous Italian Art photographs are small part (~18,000) of over 1.2 million images in the Frick Photoarchive. • Very few color photographs (makes most image analysis difficult, good test case). • Analysis was performed by myself in collaboration with the Frick Photoarchive.
  5. 5. Digital Anonymous Italian Art • Early digitization project. • At the time there was no in-house digital lab at the Frick Photoarchive. • Digitization self-funded and executed by an outside institution. • When new images were added they were not “merged” -- too hard with no attribution.
  6. 6. Image Analysis • I wrote custom software to automate matches using MatchEngine (a TinEye service). • Extremely effective image matching technology. • Matches same images and portions of images, even in black-and-white.
  7. 7. MatchEngine image1.jpg image2.jpg image3.jpg imageN.jpg Upload Images to MatchEngine . . . . . .
  8. 8. MatchEngine image1.jpg image2.jpg Retrieve Image Matches from MatchEngine image9.jpg Matches image2.jpg image1.jpg Matches Match Database
  9. 9. Archive (e.g. Frick Art Reference Library) Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image
  10. 10. Artwork Confirming a Known Match
  11. 11. ArtworkArtwork Discovering a New Match
  12. 12. 5% 3% 92% Artworks with no match Confirms a known match Matches a new work Frick Artworks That Have A Match
  13. 13. Similar Images Different photo, same work of art.
  14. 14. Similar Images Different photo, slightly different cropping.
  15. 15. Similar Images Different photo, dramatically different lighting.
  16. 16. Alternate Images Partial Image vs. Full Image
  17. 17. Alternate Images Partial Image vs. Much Larger Image
  18. 18. Alternate Images Color vs. Black-and-White
  19. 19. Conservation
  20. 20. Conservation Repairs and possibly removal of later additions.
  21. 21. Conservation
  22. 22. Conservation Possibly just a drastic difference in lighting?
  23. 23. Conservation Analysis even spots dramatic conservation work.
  24. 24. Copies
  25. 25. Copies
  26. 26. Copies
  27. 27. Copies
  28. 28. Copies
  29. 29. Copies
  30. 30. Digitization Errors Arms with Folded Hands Castello sforzesco, Milan. Female Head
 Gabinetto disegni e stampe degli Uffizi, Florence. Work doesn’t match description, wrong cataloging.
  31. 31. Still Life with a Bottle, a Plate, a Mortar and Pestle...
 Virgin Enthroned Nursing Christ, Between Two Saints
 S. Biovenale, Orvieto. Work doesn’t match description, wrong cataloging.
  32. 32. 2%4% 4% 26% 31% 34% Similar Image Wrong Image Alternate Image Ambiguous Image Different Works Conservation Types of New Matches Discovered
  33. 33. Measuring Image Analysis Efficacy
  34. 34. Artwork Confirming a Known Match
  35. 35. Artwork Impossible to Confirm Match
  36. 36. Artworks With Multiple Photos 47% 33% 20% Failed Match Successful Match No Possible Matches
  37. 37. # of Match Successes and Failures by Type 0 80 160 240 320 Same Similar Alternate Conservation 3D Negative Success Failure
  38. 38. Similar Images
  39. 39. Similar Images
  40. 40. Alternate Images
  41. 41. Alternate Images
  42. 42. Likelihood of a match by
 % of image overlap 0% 25% 50% 75% 100% 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 Failed Alternate Matches Successful Alternate Matches Successful Other Matches % of Image Overlapping
  43. 43. Likelihood of an ‘Alternate Image’ match by % of image overlap 9% 91% Failure Success 86% 14% Failure Success Under 30% Overlap Over 30% Overlap
  44. 44. Conservation
  45. 45. 3D Works
  46. 46. 3D Works
  47. 47. Negative Images
  48. 48. Overall MatchEngine Quality 62% 38% Failure Success 88% 12% Failure Success All Potential Matches No: Under 30% Overlap, 3D, or Negative
  49. 49. Frederico Zeri Foundation • University of Bologna • 290,000 photos of artworks from national archives, private archives, museums, etc.
  50. 50. Zeri 15thC Italian Art • Digitized 15th century Italian art • Persumably some overlap with the Frick’s Anonymous Italian Art archive
  51. 51. Image-to-Artwork Ratio 0 10000 20000 30000 40000 Frick Zeri 17,929 14,284 38,511 18,548 Images Artworks
  52. 52. 3% 36% 61% Artworks with no match Confirms a known match Matches a new work Zeri Artworks That Have A Match
  53. 53. Image Analysis Matches 0 5500 11000 16500 22000 Frick Zeri 738815 20,051 1,053 Same Artwork Different Artwork
  54. 54. Image Detail
  55. 55. Image Detail
  56. 56. Same Piece, Similar Style
  57. 57. Same Piece, Similar Style
  58. 58. Similar/Copies
  59. 59. What Happens When We Combine Archives? • Discover new relationships with the other archive • Discover new relationships inside our own archive
  60. 60. Frick Photoarchive Artwork Image Image Image Artwork Image Image Image Zeri Archive Artwork Image Image Image Artwork Image Image Image
  61. 61. Frick-Zeri Image Matches 18,548
 Images 38,511 Images 1,117 Images 1,442 Images
  62. 62. Frick-Zeri Artwork Matches 14,284
 Artworks 17,929 Artworks 855 Artworks (6%) 938 Artworks (5%)
  63. 63. MergeVerification • Previously, 77 artworks were manually matched by a researcher at the Zeri foundation • Can use to verify the quality of the matches
  64. 64. MergeVerification 16% 84% Same Artwork Failed to Match
  65. 65. Lighting Differences
  66. 66. Tiny Details
  67. 67. Merging Works! • 84% success confirming known relationships (with a small sample) • Only 6% of computer-discovered matches were also found by a human. • (55 human matches vs. 938 computer matches)
  68. 68. Graph Analysis
  69. 69. Frick 420 420 Zeri 15836 42091 42090 Frick 417 417 42092 ?
  70. 70. Finding new Links • Found 14 new Frick links via Zeri • Found 236 new Zeri links via Frick
  71. 71. Frick 347 347 Zeri 12227 33526 33527 Frick 348 348 33525 33528 ?
  72. 72. Frick 8131 Zeri 23402 57136 Frick 8132 8132a 8132 57129 57134 Zeri 23405 5713057137 57138 8131a 8131 ?
  73. 73. Conclusion • It becomes possible to merge photo archives, even though they’re in different langauges. • New relationships are discovered in the process. • The synergy of combining multiple archives reveals even more information. • All of this is completely automated!
  74. 74. Contact • John Resig • jeresig@gmail.com • http://ejohn.org/research/ • http://ukiyo-e.org/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×