The document discusses using machine learning and artificial intelligence to generate tags for artworks in the collection of the Metropolitan Museum of Art. The goals are to increase user engagement, improve search and discovery, and make the collection more accessible. An overview is provided of the human tagging process and resulting stats. Challenges of AI tagging including accuracy, subjectivity, relevance and lack of context are discussed. Opportunities for using generated tags to improve search and discoverability are explored.

1. MCN 201901
Tags, Art, and AI. Oh My.
Jennie Choi, The Metropolitan Museum of Art
Elena Villaespesa, Pratt Institute (@elenustika)
Andrew Lih, Wikimedia (@fuzheado)

2. Goals
• Increase user engagement
• Improve search and discovery of the collection
• Make collection accessible to the widest possible audience
• Explore using tags as training data for AI models

3. • Taxonomy drafted
• Outside vendor selected
• Vendor team trained
• Single judgements
• Weekly calls and data review
• Tags imported into collections
management system
• Ongoing review
Human Tagging Process

4. Tagging project stats:
● 1,000 total unique tags
● 233,000 objects tagged
Top tags:
● Men 63,000
● Women 38,000
● Portraits 35,000
● Flowers 20,000
Fun Facts:
● Female Nudes 3,000
● Male Nudes 1,700
● Dogs 3,000
● Cats 600

5. 5
Tag Distribution

6. Completeness
• Circus
• Tigers
• Acrobats
• Dogs
• Monkeys
• Horses
• Elephants
• Snakes
• Men
• Women

7. • Women
• Horses

8. Accuracy
Mihrab Crucifixion Lighting

9. Subjectivity

10. Relevance

11. It’s complicated…

12. Boy with Blond Hair
ca. 1840–50
1973.323.5
Madame Georges Charpentier (Marguérite-Louise Lemonnier, 1848–1904)
and Her Children, Georgette-Berthe (1872–1945) and Paul-Émile-Charles
(1875–1895)
Auguste Renoir, French, Limoges 1841–1919 Cagnes-sur-Mer
1878
07.122
John Yellow Flower, No. 40, collector card from the
American Indian Series (D6), issued by the Kelley
Baking Company to promote Kelley's Bread
Issued by Kelley Baking Company
1940
63.350.307.6.27

13. Kaggle Competition
https://www.kaggle.com/c/imet-2019-fgvc6

14. “Kernels Only” Competition

15. Tensor Flow Hub- public model repository

16. AI Challenges
• Lack of Developer Resources
• Imperfect Training Data
o Subjectivity
o Completeness
o Accuracy
o Relevance
• Not Enough Training Data (we only have 600 cats…)
• No Right Answers for Tagging Art
• Bias

17. Human vs. Machine
AI-assisted tagging for artworks
Elena Villaespesa (@elenustika)
Seth Crider (@SethCrider2)
Pratt Institute

18. 1,414 objects with images on the public domain
The Met - Highlights

19. Tags usage (long tail and top tags)
Human tags Google Amazon

20. Google
(918)
Human tags
(537)
Amazon
(733)
26
Unique tags
286 20
There is a small number of tags that
are applied both by the museum
and these algorithms
Note: exact tags, singular vs plural
12
Human - tag of sentiments, actions,
what is depicted
Machine - art form, material, color,
art movements,

21. Accuracy
Diaper bag Birthday cake Skateboard
Accuracy is one of the primary challenges of these tags.

22. Accuracy
Weapon 3D modeling Modern art
Accuracy is one of the primary challenges of these tags.

23. Accuracy and confidence score Person 99.873642
Human 99.873642
Clothing 98.1922913
Apparel 98.1922913
Transportation 87.5043945
Boat 87.5043945
Vehicle 87.5043945
Airfield 71.124527
Airport 71.124527
Tire 70.2007675
Pants 65.6449432
Watercraft 61.6692047
Vessel 61.6692047
Hat 61.4706268
Machine 59.0481873
Spoke 59.0481873
Shorts 56.8293343
Coat 55.262558
Overcoat 55.262558
Wheel 55.0610924
Alloy Wheel 55.0610924

24. Subjectivity: Medium and art period
Human Forests, Landscapes, Oaks
Google Nature, Tree, Leaf, Snapshot, Branch,
Monochrome, Woody plant, Rock, Stock
photography, Organism
Amazon Nature, Outdoors, Landscape, Weather, Tree,
Plant, Scenery, Rug, Snow, Art, Painting,
Vegetation, Winter, Ice, Land, Woodland,
Forest
Human Hieroglyphs
Google Hose
Amazon Pendant
Scarab of the Storehouse
Overseer Wah
(Egypt)
Oak Tree and
Rocks, Forest of
Fontainebleau by
Gustave Le Gray

25. Completeness
Human Bears, Centaurs, Deer, Men, Hunting, Satyrs, Dogs, Forests,
Lions
Google Painting, Art, Visual arts, Mythology, Stock photography,
Modern art
Amazon Art, Painting
Human Interiors, Girls, Men, Women,
Smoking, Dogs
Google Visual arts, Art, Painting
Amazon Human, Person, Art, Painting

26. Relevance
Clothing
Forehead
Art

27. Context
Human George Washington, Men, Portraits
Google Portrait, Self-portrait, Gentleman, Lady,
Painting, Art, Barrister, Elder
Amazon Painting, Art, Person, Human
George Washington
By Gilbert Stuart
The context of historical or political figures is not captured by the machine tags.

28. Context
Cleopatra
By William Wetmore Story
Human Cleopatra, Women
Google Sculpture, Statue, Classical sculpture,
Figurine, Stone carving, Art, Carving,
Monument, Marble, Mythology
Amazon Art, Sculpture, Statue, Figurine, Person,
Human, Archaeology

29. The usage of gender-related tags (Female, Lady, Gettleman, Man…) is low and neutral tags such as figure,
person or human are used.
Gender
Human Apples, Male Nudes
Google Sculpture, Bronze sculpture, Statue, Art,
Standing, Figurine, Classical sculpture,
Metal, Bronze, Human
Amazon Sculpture, Art, Statue, Person, Human,
Torso, Bronze, Coat, Apparel, Clothing,
Overcoat, Suit, Tire, Figurine
Paris
By Antico (Pier Jacopo Alari Bonacolsi)

30. Tags to improve the Online Collection UX

31. Online Collection - User feedback
I was looking for reference photos
for an 18th century japanese
bedroom. It would help if all the
subject matter of ancient pictures
was hashtagged in a way that I
could advance search, along with
time period.
I would love more tags on
historical pieces so they are easier
to search. Then, I could come
directly to The Met site instead of
searching through Pinterest.
Have a search function for subject
matter
Link with the keyword with other
collection of the MET, like word
cartonnage and after link to
picture of egyptian cartonnage
mummy, and after restoration of
the cartonnage and after
pigmentation use to do the paint
etc etc..
Narrowing down search results
using the filters is not always
effective; I wish there were more
specific categories that could be
browsed more easily, like
"Spanish painting," etc. --
perhaps through tags or
something similar?
Organize the art works by
theme/subject in addition to
country/region i.e. nature,
abstraction, religion, political
themes. In this way, I could tie the
art works to subject specific
curriculum such as history,
geography, government, etc.

32. How can tags improve the Search user experience?
● Improve discoverability
of object that do not
have these keywords on
the title or object
description
● Respond to user needs
and current searches on
the Online Collection
● Potentially machine
tags complement the
taxonomy with a variety
of keywords

33. Search Analytics
6% website users use the
search functionality
10% Online Collection
users
3.3M searches
875K keywords
Source: Google Analytics (Oct 2018 - Sep 2019)

34. Search - Birds

35. Search - Dance

36. Search volume and top
searches
Human Tags 208K
Google Vision 205K
Amazon 198K
Painting
Guitar
Fashion
Portrait
Dress
Landscape
Sculpture
Map
Ceramic
Mask
Flower
Costume
Photography
Still life
Drawing
Sword
Cat
Music
Textile
Dog
Guitars
Portraits
Landscapes
Maps
Masks
Flowers
Costumes
Painting
Armor
Sculpture
Jewelry
Still Life
Japanese
Swords
Cats
Music
Buddha
Dogs
Cloisters
Musical Instruments
Painting
Guitar
Fashion
Portrait
Dress
Landscape
Sculpture
Map
Mask
Flower
Costume
Armor
Jewelry
Photography
Drawing
Sword
Cat
Buddha
Dog
Corset
Human Google Amazon

37. ● Accuracy and lack of context are the major challenges of using these technologies
● Usage of image recognition can generate labels that may increase the diversity of the
terms used to tag the collection
● These tags can significantly increase the discoverability of the collection artworks via
search, navigation, SEO
Further analysis:
● Analysis of the impact on search analytics (e.g. search exit rate)
● Include only tags with high levels of confidence
● Collect and analyse tags from computer vision tools (e.g. Clarifai, Imagga, Microsoft…)
● Gather user feedback via user testing/eye tracking on the usefulness of these tags
(display info about the source of the tag, usage, etc)
Conclusion

38. Wikipedia articles
Wikimedia environment
50 million pages in 200+ languages
English: 5.9 million articles
Britannica < 500,000
Highly notable topics

39. Wikipedia articles
Wikicommons media files
Wikimedia environment
50 million pages in 200+ languages
English: 5.9 million articles
Britannica < 500,000
Highly notable topics
56 million media files
500+ million views per month
Wide project scope

40. Wikipedia articles
Wikidata items
● Structured database of all notable figures/works
● Language independent, rich metadata
● Supports comprehensive linkages to collections
● Searchable, interactive, scalable
Linked
Open Data
Wikicommons media files
Wikimedia environment – Focus on Wikidata contributions

41. Wikidata
Potential
Try it! w.wiki/BUA
Interconnected knowledge
graph of culture: art,
fashion, literature

42. 1 - AI machine learning
Met subject keywords used to train machine learning model
Use image classifier to predict labels for other artworks
Training takes hours, but predictions are fast (multiple per second)
Create Wikidata Game to help assess predicted labels and add to Wikidata

46. Wikidata Game using Met-trained machine learning engine - link

47. Depiction information added to Wikidata watcher - link

48. Met AI experiments - Met blog post
"...even such a high
measure of confidence
becomes useless if one
cannot sift the incorrect
classifications from the
correct ones. This is
where the Wikimedia
community comes in."

49. Results of Wikidata Game - Depicts
Focused on 2D artworks such as paintings
More than 7,000 judgments via the game resulting in ~5,000 edits
Depiction topics - tree, boat, flower, horse, soldier, house, ship
landscape painting features performed well
Gender determination, cats, and dogs not so well
Wikimedia Commons putting resources into similar ML capabilities

50. Depiction judgments
One judgment = one live edit to Wikidata
Recruiting and retaining a user much more expensive than undoing
vandalism
Users can inspect and patrol edits of bad faith editors (and block them)
For AI, Wikimedia editors are perhaps the best humans-in-the-loop

51. 2 - Status - Live SPARQL dashboards of Met collections
Most commonly
depicted themes
In Met artworks
(partial, Jan 2019)

52. 2 - Status - Live SPARQL dashboards of Met collections
Most commonly
depicted themes
In Met artworks
(partial, Nov 2019)

53. SXSW presentation
https://panelpicker.sxsw.com
/vote/100500
March 2020

54. Future work
Feed judgments back into ML model to refine the neural net
Perform training for specific artwork types and domains -
paintings vs sculpture vs costumes/fashion

55. Future work
ML image classification as a "suggestion module" for other tools
Example: Wiki Art Depiction Explorer (Knight Foundation-funded project)
https://art.wikidata.link
Suggest Met AI-generated tags

56. Wiki Art Depiction Explorer - https://art.wikidata.link/property/P195

57. Wiki Art Depiction Explorer - https://art.wikidata.link/browse?P195=Q160236

58. WADE - https://art.wikidata.link/item/Q19911950

59. WADE possible interface - suggestion from Met ML model
Automatically
generated tags

60. Conclusions
Promising exploratory work combines best of both worlds:
scale of ML/AI operations + expertise of the best volunteer community
Caveats:
● Are we reproducing systemic/historical biases in the ML models?
● Incorporating better metadata and vocabularies for non-Western art