NYARC 2014: Frick/Zeri Results

  • 523 views
Uploaded on

Talk given at the 2014 NYARC annual meeting.

Talk given at the 2014 NYARC annual meeting.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
523
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
2
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Photo Archive
 Image Analysis John Resig March 2014 http://ejohn.org/research/
  • 2. Ukiyo-e.org
  • 3. Frick Art Reference Library Photoarchive • Frick Art Reference Library Photoarchive established in 1920. • Collects photographs of works of art from around the world, including sponsoring own photography. • Traditionally image analysis has not been applied to photo archives. • The Frick Photoarchive was interested in exploring image analysis more after seeing
 Ukiyo-e.org.
  • 4. Test case:Anonymous Italian Art Photographs • Anonymous Italian Art photographs are small part (~18,000) of over 1.2 million images in the Frick Photoarchive. • Very few color photographs (makes most image analysis difficult, good test case). • Analysis was performed by myself in collaboration with the Frick Photoarchive.
  • 5. Digital Anonymous Italian Art • Early digitization project. • At the time there was no in-house digital lab at the Frick Photoarchive. • Digitization self-funded and executed by an outside institution. • When new images were added they were not “merged” -- too hard with no attribution.
  • 6. Image Analysis • I wrote custom software to automate matches using MatchEngine (a TinEye service). • Extremely effective image matching technology. • Matches same images and portions of images, even in black-and-white.
  • 7. MatchEngine image1.jpg image2.jpg image3.jpg imageN.jpg Upload Images to MatchEngine . . . . . .
  • 8. MatchEngine image1.jpg image2.jpg Retrieve Image Matches from MatchEngine image9.jpg Matches image2.jpg image1.jpg Matches Match Database
  • 9. Archive (e.g. Frick Art Reference Library) Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image Artwork Image Image Image
  • 10. Artwork Confirming a Known Match
  • 11. ArtworkArtwork Discovering a New Match
  • 12. 5% 3% 92% Artworks with no match Confirms a known match Matches a new work Frick Artworks That Have A Match
  • 13. Similar Images Different photo, same work of art.
  • 14. Similar Images Different photo, slightly different cropping.
  • 15. Similar Images Different photo, dramatically different lighting.
  • 16. Alternate Images Partial Image vs. Full Image
  • 17. Alternate Images Partial Image vs. Much Larger Image
  • 18. Alternate Images Color vs. Black-and-White
  • 19. Conservation
  • 20. Conservation Repairs and possibly removal of later additions.
  • 21. Conservation
  • 22. Conservation Possibly just a drastic difference in lighting?
  • 23. Conservation Analysis even spots dramatic conservation work.
  • 24. Copies
  • 25. Copies
  • 26. Copies
  • 27. Copies
  • 28. Copies
  • 29. Copies
  • 30. Digitization Errors Arms with Folded Hands Castello sforzesco, Milan. Female Head
 Gabinetto disegni e stampe degli Uffizi, Florence. Work doesn’t match description, wrong cataloging.
  • 31. Still Life with a Bottle, a Plate, a Mortar and Pestle...
 Virgin Enthroned Nursing Christ, Between Two Saints
 S. Biovenale, Orvieto. Work doesn’t match description, wrong cataloging.
  • 32. 2%4% 4% 26% 31% 34% Similar Image Wrong Image Alternate Image Ambiguous Image Different Works Conservation Types of New Matches Discovered
  • 33. Measuring Image Analysis Efficacy
  • 34. Artwork Confirming a Known Match
  • 35. Artwork Impossible to Confirm Match
  • 36. Artworks With Multiple Photos 47% 33% 20% Failed Match Successful Match No Possible Matches
  • 37. # of Match Successes and Failures by Type 0 80 160 240 320 Same Similar Alternate Conservation 3D Negative Success Failure
  • 38. Similar Images
  • 39. Similar Images
  • 40. Alternate Images
  • 41. Alternate Images
  • 42. Likelihood of a match by
 % of image overlap 0% 25% 50% 75% 100% 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 Failed Alternate Matches Successful Alternate Matches Successful Other Matches % of Image Overlapping
  • 43. Likelihood of an ‘Alternate Image’ match by % of image overlap 9% 91% Failure Success 86% 14% Failure Success Under 30% Overlap Over 30% Overlap
  • 44. Conservation
  • 45. 3D Works
  • 46. 3D Works
  • 47. Negative Images
  • 48. Overall MatchEngine Quality 62% 38% Failure Success 88% 12% Failure Success All Potential Matches No: Under 30% Overlap, 3D, or Negative
  • 49. Frederico Zeri Foundation • University of Bologna • 290,000 photos of artworks from national archives, private archives, museums, etc.
  • 50. Zeri 15thC Italian Art • Digitized 15th century Italian art • Persumably some overlap with the Frick’s Anonymous Italian Art archive
  • 51. Image-to-Artwork Ratio 0 10000 20000 30000 40000 Frick Zeri 17,929 14,284 38,511 18,548 Images Artworks
  • 52. 3% 36% 61% Artworks with no match Confirms a known match Matches a new work Zeri Artworks That Have A Match
  • 53. Image Analysis Matches 0 5500 11000 16500 22000 Frick Zeri 738815 20,051 1,053 Same Artwork Different Artwork
  • 54. Image Detail
  • 55. Image Detail
  • 56. Same Piece, Similar Style
  • 57. Same Piece, Similar Style
  • 58. Similar/Copies
  • 59. What Happens When We Combine Archives? • Discover new relationships with the other archive • Discover new relationships inside our own archive
  • 60. Frick Photoarchive Artwork Image Image Image Artwork Image Image Image Zeri Archive Artwork Image Image Image Artwork Image Image Image
  • 61. Frick-Zeri Image Matches 18,548
 Images 38,511 Images 1,117 Images 1,442 Images
  • 62. Frick-Zeri Artwork Matches 14,284
 Artworks 17,929 Artworks 855 Artworks (6%) 938 Artworks (5%)
  • 63. MergeVerification • Previously, 77 artworks were manually matched by a researcher at the Zeri foundation • Can use to verify the quality of the matches
  • 64. MergeVerification 16% 84% Same Artwork Failed to Match
  • 65. Lighting Differences
  • 66. Tiny Details
  • 67. Merging Works! • 84% success confirming known relationships (with a small sample) • Only 6% of computer-discovered matches were also found by a human. • (55 human matches vs. 938 computer matches)
  • 68. Graph Analysis
  • 69. Frick 420 420 Zeri 15836 42091 42090 Frick 417 417 42092 ?
  • 70. Finding new Links • Found 14 new Frick links via Zeri • Found 236 new Zeri links via Frick
  • 71. Frick 347 347 Zeri 12227 33526 33527 Frick 348 348 33525 33528 ?
  • 72. Frick 8131 Zeri 23402 57136 Frick 8132 8132a 8132 57129 57134 Zeri 23405 5713057137 57138 8131a 8131 ?
  • 73. Conclusion • It becomes possible to merge photo archives, even though they’re in different langauges. • New relationships are discovered in the process. • The synergy of combining multiple archives reveals even more information. • All of this is completely automated!
  • 74. Contact • John Resig • jeresig@gmail.com • http://ejohn.org/research/ • http://ukiyo-e.org/