John Resig discusses four problems that digital librarians face: 1) searching by image without exact matches, 2) disagreements on names and titles, 3) poor quality images, and 4) educating users. He provides examples and potential solutions for each problem, including image search tools, handling alternate names and formats, improving images through cropping and computer vision, and using question/answer interfaces to educate users. Resig advocates for open source tools and collaborative solutions to advance access to knowledge.

1. Accidentally
Becoming
A
Digital Librarian
John Resig

2. Providing Access to Knowledge.
Educating and Empowering Others.

3. 1. Solve your own problems

4. 2. Adapt and solve other’s
problems

5. Talking about 4 problems

6. Chazen Museum: 1980.2386

7. Problem 1:
Searching by Image

12. imgSeek
• Compares entire image.
• Finds similar images, not
exact.
• Does not find parts of
an image.
• Color sensitive.
• Open Source

13. https://services.tineye.com/MatchEngine

14. http://pastec.io/

17. Problem 2: No one agrees on
names/titles

18. Japanese Names
• Utagawa Hiroshige
• Ando Hiroshige
• Andō Hiroshige
• Hiroshige
• 歌川広重
• 広重

25. Similar Images
Different photo, same work of art.

26. Similar Images
Different photo, slightly different cropping.

27. Alternate Images
Partial Image vs. Much Larger Image

28. Alternate Images
Color vs. Black-and-White

29. Conservation

30. Conservation
Repairs and possibly removal of later additions.

31. Conservation/Destruction
Analysis even spots dramatic conservation work.

32. Copies

33. Copies

34. Copies

35. Copies

37. Fondazione Zeri

38. PHAROS: An International Consortium of Photo Archives
• Bibliotheca Hertziana, Rome (1,065,000)
• Bildarchiv Foto Marburg, Germany (2,000,000)
• Courtauld Institute of Art, London (4,173,500)
• Fondazione Federico Zeri, Bologna (290,000)
• Frick Art Reference Library, New York (1,346,000)
• Getty Research Institute, Los Angeles (2,086,000)
• Villa I Tatti, Florence (239,000)
• Institut National d’Histoire de l’Art, Paris (750,000)
• Kunsthistorisches Institut, Florence (650,000)
• National Gallery of Art, Washington (7,600,000)
• Paul Mellon Centre, London (185,000)
• Rijksbureau, The Hague (7,000,000
• Warburg Institute, London (3,500,000)
• Yale Center for British Art, New Haven (132,000)

40. PHAROS Images: Coming June 2016

41. Problem 3: Bad Images

44. Idyll: Offline Image Cropping
• Crop and annotate images offline and on a
mobile device.
• Saves the selections back to a server.

48. ComputerVision
• Unsupervised (requires no labeling):
• Comparing an entire image
• Categorizing an image
• Supervised (requires labeling):
• Finding parts of an image
• Finding and categorizing parts of an image

49. Unsupervised Training
• Requires little-to-no prepping of data
• Can just give the tool a set of images and
have it produce results
• Extremely easy to get started, results aren’t
always as interesting.
• Unsupervised: MatchEngine, PasteC

50. Supervised Training
• Need lots of training data
• Needs to be pre-selected/categorized
• Think:Thousands of images.
• If your collection is smaller than this, perhaps
it may not benefit.
• Or you may need crowd sourcing.
• Results can be more interesting:
• “Find all the people in this image”

51. General Computer
Vision
• Ideal for some supervised training problems
• CCV
• http://libccv.org/
• https://github.com/liuliu/ccv
• OpenCV
• http://opencv.org/

52. Object Detection

57. Problem 4: Education

64. Does your person have a bald spot at the top of their head?

65. Does your person have a *hidden* bald spot at the top of their head?

66. Does your person have a *hidden* bald spot at the top of their head?

67. Does your person have a cloth covering the front of their head?

68. Does your person have a comb in their hair?

72. • http://ejohn.org/research/
• http://ukiyo-e.org/
• https://github.com/jeresig