This presentation was provided by Gerald Benoit of Simmons College during the NISO webinar, Enabling Discovery and Retrieval of Non-Traditional and Granular Content, held on June 7, 2017

1. projects and impacts
G Benoît
June 7, 2017
1
Visual only retrieval

2. Visual Only Retrieval
• Introduction
• Search (traditional and visual)
• Projects
• Conclusions
2
A draft version of notes is available here.

3. Introduction
• Many attempts to improve image retrieval
• Automatic identification (“blobs”)
• “Traditional” descriptors
• Human-added metadata
• Metadata extracted from image files
3

4. Introduction
• Algorithmic approaches use techniques, similar to k-
nearest neighbor, to compare density of color or scan
for primitive shapes (such as the shape of an animal)
• Other techniques for automatic identification use the
Golden Mean (1:1.14) and lattices of triangles
• Others require control over image production (light,
object size, etc.) [e.g., Global Memory Net]
4

5. “Traditional” - text
• Using one language (text) to find another (visual) is
useful but might we do more? How would this affect
end-user behavior, information system architecture,
and the future of retrieval as the volume, variability, and
variety of files increase?
• Controlled Vocabularies and Tech
• E.g., Getty, ULAN, TGN, AAT;
• File storage and standards: e.g., VRA-4; .json, sql;
tags for original content; XML
5

6. “Traditional”
• Consequence is possible query reformation
• Number and types of queries limited usually to
standardized descriptors or professional practice
• Silo-ization of data: requires additional tech to
crosswalk between descriptors (Vocabulary
Coordinaty System, [Pratt, 2008]).
• Raises question about how end-users, cataloguers,
and computer scientists perceive images …
6

7. All-Visual?
• Text-oriented searching for images is dependent upon
the quality and type of metadata; equally dependent
upon the end-user’s knowledge of the terms used in
cataloguing
• Other efforts:
• clipart.com
• Google Images
• ARTStor, etc.
• Locally-created collections
7
blaue-reiter.purzuit.com

8. All Visual - Tech
• An all visual approach to retrieval changes the dynamic of searching and the
end-users’ behaviors
• Role of interpretation
• Impact of knowledge of the content of the visuals
• Visual literacy
• Greatly simplified by more powerful, easy-to-use tools (HTML5, CSS3, JS,
SQL, XML parsers, etc.)
• Visuals combined with elementary tech yields both a novel experience as well
as a familiar one, combining mouse events with the sense of a light table…
• Much greater shift to end-users’ cognition, meaning construction; retrieval
set membership [more false hits]
8

9. Project
• Create an all visual retrieval
• 5000+ images from Boston Public Library
• Backend: .txt, mysql; Apache webserver; PHP
• Frontend: randomized presentation of images to
end-users …
• Create click-thru record for each subject
• Shifts in options between users suggests a
different trigger or reaction to the input … [the
data aren’t finished being analyzed…]
9

10. Project
• 3 groups of self-selected volunteers [15/group]
• Librarians, Artists, Students (UG/G - not in the arts),
Administrators
• Defined their interests of the metadata (such as color
model, when was the image used in class, emotional
and symbolic language and their own concept tags,
integration with other OPACs, design their own
interface)
10

11. Project
• 20 pre-defined collections [American Indians, Ships,
Famous people, Travel Posters, etc.]
• Randomly presented images to end-users; can “flip the
card” to see details [metadata, file info, culture, genre,
subject tracings, use-history, user-created tags, “why
this is important” wikipedia-type text]
• Users can opt to follow a hyperlink (back of card) or
continue with images. Click behaviors captured in log.
• Example: “I want blue things.”
11

12. Example
12
Lincoln, presidents, Civil War, 19th cent fashion, history
of photography, military uniforms, sepia prints, b&w;
history of beards (grin)

13. Example
13
Which view is more helpful in finding “William”?

14. Project: How would people react in a visual-only retrieval system?
In the absence of such IR systems, not much is known about how
users will interact with a visuals-only retrieval system.
(a) how users interact with graphic-only retrieval for exploring
traditional and non-traditional access points and
(b) how the affective component impacts the use of such systems.
Findings based on the study will help shed light on research based
on visual information systems and user behavior when interacting
with such systems. The findings will be useful both in designing
systems that respond to user needs, and add to prior research in
information seeking and retrieval.

15. Project - first interface
15

16. Project - 2nd interface
16
Added build-and-name our own
collections; share with others.
Can create several collections at
once (serendipitous finding of
ideas other than the original
motivation;
Search by density of R G B

17. Group Desired features and/or concerns
Students (UG/G) Add and search by own tags Use history
“Wikipedia” type
expert text “why
Artists Color models Image data (file type, no of pixels)
build own
collections
Librarians Integrate with existing OPACs
Create new service for patrons
(community informatics)
Why use this
when we pay
Administrators Who’s going to maintain it? Is is scaleable?

18. Conclusions
Makes sense to pursue visual only because the tremendous rise in the use of visual devices
- iPhones, pads, etc., as well as increased familiarity with creating images on one’s own.
Cognitive models of Info seeking behavior are well-known and recognize
a. serendipitous discovery
b.impact of aesthetics and design in non-linear processing
c. supports end-user division/classification of data into clusters that aren’t otherwise
possible
Greater need, then, for controlled vocabularies/metadata and the end-users’ own tags to
establish meaning: a seemingly heterogeneous set of images becomes one based on
individually tailorable reasons
Encourages participation because of a “sense of ownership” by the end user in creating
sets and sense of the institution’s value in providing this service
As a conversation starter; similar to Herr’s application of information visualizations and chat
options to study the reactions of the visualization and the end-users’ exchanges about the
visuals
18

19. Conclusions - Research
• IRO - allows asking questions about intergroup
behavior and differences exerted in the context of
creating a set - new understanding of ISB
• “Generative Metaphor” - creating subsets on one’s
own; investigating otherwise impossible combinations;
new expressions to capture the set
• Integrate into metadata standards
• Click thru data - can be used predictively; to control
user choices in real-time - teaching tool
19

20. Conclusions - Research
• Same “language” of search and display - greater
interactivity between user and data - suggestive of
Interactive Information Visualization
• Greater exploration of the data - can drag-and-drop
images to test clusters when making sense - from
heterogeneous sets to a unified one that the end-user
owns, understands - has value.
• How deep to go in adding more metadata?
20

21. Conclusions - Research
• Train end-users and librarians in Visual Literacy
• Too specific metadata limits use to experts and
programmers; end-user provided “loose” tags reflects real-
time language; evidence to map between CVs
• Affects information system architecture in general - retrieval
algorithms, set combinatorics, db design - and supports
non-traditional terms, such as symbolic, emotional, …
“memories” [one user’s observation]
• Questions of how people (PIM?) classify; what and why;
move to 3D?
21

22. Finally …
• Visual only retrieval system is really in its infancy but it is
worth pursuing
• It may be that visual-only retrieval becomes perhaps
more “intuitive” - connecting to the end-user not unlike
early Modernists’ desire to “speak directly to the mind.”
• Thank you! Questions?
• Gerald Benoît, Ph.D., Associate Professor, Computer-
and Information Science, Simmons College, 300 The
Fenway, Boston, MA 02115 USA benoit@simmons.edu
22