Download to read offline
Uploaded byjeresig
NYARC 2014: Frick/Zeri Results
The document summarizes research using image analysis to match photographs across multiple art archives, including the Frick Art Reference Library Photoarchive and the Frederico Zeri Foundation archive. Running image matching software on over 1.2 million Frick photos and 290,000 Zeri photos revealed new connections within and between the archives. Combining the archives through automated matching discovered over 1,000 new artwork connections that had not previously been found by human researchers. The analysis demonstrates the potential of merging photo archives through computational methods to reveal new relationships and insights.
More Related Content
Similar to NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
- 1. Photo Archive Image Analysis JohnResig March 2014 http://ejohn.org/research/
- 2.
- 12. Frick Art ReferenceLibrary Photoarchive • Frick Art Reference Library Photoarchive established in 1920. • Collects photographs of works of art from around the world, including sponsoring own photography. • Traditionally image analysis has not been applied to photo archives. • The Frick Photoarchive was interested in exploring image analysis more after seeing Ukiyo-e.org.
- 13. Test case:Anonymous Italian ArtPhotographs • Anonymous Italian Art photographs are small part (~18,000) of over 1.2 million images in the Frick Photoarchive. • Very few color photographs (makes most image analysis difficult, good test case). • Analysis was performed by myself in collaboration with the Frick Photoarchive.
- 14. Digital Anonymous Italian Art •Early digitization project. • At the time there was no in-house digital lab at the Frick Photoarchive. • Digitization self-funded and executed by an outside institution. • When new images were added they were not “merged” -- too hard with no attribution.
- 15. Image Analysis • Iwrote custom software to automate matches using MatchEngine (a TinEye service). • Extremely effective image matching technology. • Matches same images and portions of images, even in black-and-white.
- 17. MatchEngine image1.jpg image2.jpg image3.jpg imageN.jpg Upload Images toMatchEngine . . . . . .
- 18. MatchEngine image1.jpg image2.jpg Retrieve Image Matchesfrom MatchEngine image9.jpg Matches image2.jpg image1.jpg Matches Match Database
- 19. Archive (e.g. FrickArt Reference Library) Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image
- 20.
- 21.
- 22. 5% 3% 92% Artworks with nomatch Confirms a known match Matches a new work Frick Artworks That Have A Match
- 23. Similar Images Different photo,same work of art.
- 24. Similar Images Different photo,slightly different cropping.
- 25. Similar Images Different photo,dramatically different lighting.
- 26. Alternate Images Partial Imagevs. Full Image
- 27. Alternate Images Partial Imagevs. Much Larger Image
- 28. Alternate Images Color vs.Black-and-White
- 29.
- 30. Conservation Repairs and possiblyremoval of later additions.
- 31.
- 32. Conservation Possibly just adrastic difference in lighting?
- 33. Conservation Analysis even spotsdramatic conservation work.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40. Digitization Errors Arms withFolded Hands Castello sforzesco, Milan. Female Head Gabinetto disegni e stampe degli Uffizi, Florence. Work doesn’t match description, wrong cataloging.
- 41. Still Life witha Bottle, a Plate, a Mortar and Pestle... Virgin Enthroned Nursing Christ, Between Two Saints S. Biovenale, Orvieto. Work doesn’t match description, wrong cataloging.
- 42. 2%4% 4% 26% 31% 34% Similar Image WrongImage Alternate Image Ambiguous Image Different Works Conservation Types of New Matches Discovered
- 43.
- 44.
- 45.
- 46. Artworks With MultiplePhotos 47% 33% 20% Failed Match Successful Match No Possible Matches
- 47. # of MatchSuccesses and Failures by Type 0 80 160 240 320 Same Similar Alternate Conservation 3D Negative Success Failure
- 48.
- 49.
- 50.
- 51.
- 52. Likelihood of amatch by % of image overlap 0% 25% 50% 75% 100% 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 Failed Alternate Matches Successful Alternate Matches Successful Other Matches % of Image Overlapping
- 53. Likelihood of an‘Alternate Image’ match by % of image overlap 9% 91% Failure Success 86% 14% Failure Success Under 30% Overlap Over 30% Overlap
- 54.
- 55.
- 56.
- 57.
- 58. Overall MatchEngine Quality 62% 38% FailureSuccess 88% 12% Failure Success All Potential Matches No: Under 30% Overlap, 3D, or Negative
- 59. Frederico Zeri Foundation • Universityof Bologna • 290,000 photos of artworks from national archives, private archives, museums, etc.
- 60. Zeri 15thC ItalianArt • Digitized 15th century Italian art • Persumably some overlap with the Frick’s Anonymous Italian Art archive
- 61.
- 62. 3% 36% 61% Artworks with nomatch Confirms a known match Matches a new work Zeri Artworks That Have A Match
- 63. Image Analysis Matches 0 5500 11000 16500 22000 FrickZeri 738815 20,051 1,053 Same Artwork Different Artwork
- 64.
- 65.
- 66. Same Piece, SimilarStyle
- 67. Same Piece, SimilarStyle
- 68.
- 70. What Happens When WeCombine Archives? • Discover new relationships with the other archive • Discover new relationships inside our own archive
- 71.
- 72. Frick-Zeri Image Matches 18,548 Images 38,511Images 1,117 Images 1,442 Images
- 73. Frick-Zeri Artwork Matches 14,284 Artworks 17,929Artworks 855 Artworks (6%) 938 Artworks (5%)
- 74. MergeVerification • Previously, 77artworks were manually matched by a researcher at the Zeri foundation • Can use to verify the quality of the matches
- 75.
- 76.
- 77.
- 78. Merging Works! • 84%success confirming known relationships (with a small sample) • Only 6% of computer-discovered matches were also found by a human. • (55 human matches vs. 938 computer matches)
- 79.
- 80. Frick 420 420 Zeri 15836 42091 42090 Frick417 417 42092 ?
- 82. Finding new Links •Found 14 new Frick links via Zeri • Found 236 new Zeri links via Frick
- 83. Frick 347 347 Zeri 12227 3352633527 Frick 348 348 33525 33528 ?
- 85. Frick 8131 Zeri 23402 57136 Frick8132 8132a 8132 57129 57134 Zeri 23405 5713057137 57138 8131a 8131 ?
- 88. Conclusion • It becomespossible to merge photo archives, even though they’re in different langauges. • New relationships are discovered in the process. • The synergy of combining multiple archives reveals even more information. • All of this is completely automated!
- 89. Contact • John Resig •jeresig@gmail.com • http://ejohn.org/research/ • http://ukiyo-e.org/