Successfully reported this slideshow.
Your SlideShare is downloading. ×

Tags, Art, and AI. Oh My.

Check these out next

41 of 60 Ad
41 of 60 Ad

More Related Content

Similar to Tags, Art, and AI. Oh My. (20)


Tags, Art, and AI. Oh My.

  1. 1. MCN 201901 Tags, Art, and AI. Oh My. Jennie Choi, The Metropolitan Museum of Art Elena Villaespesa, Pratt Institute (@elenustika) Andrew Lih, Wikimedia (@fuzheado)
  2. 2. Goals • Increase user engagement • Improve search and discovery of the collection • Make collection accessible to the widest possible audience • Explore using tags as training data for AI models
  3. 3. • Taxonomy drafted • Outside vendor selected • Vendor team trained • Single judgements • Weekly calls and data review • Tags imported into collections management system • Ongoing review Human Tagging Process
  4. 4. Tagging project stats: ● 1,000 total unique tags ● 233,000 objects tagged Top tags: ● Men 63,000 ● Women 38,000 ● Portraits 35,000 ● Flowers 20,000 Fun Facts: ● Female Nudes 3,000 ● Male Nudes 1,700 ● Dogs 3,000 ● Cats 600
  5. 5. 5 Tag Distribution
  6. 6. Completeness • Circus • Tigers • Acrobats • Dogs • Monkeys • Horses • Elephants • Snakes • Men • Women
  7. 7. • Women • Horses
  8. 8. Accuracy Mihrab Crucifixion Lighting
  9. 9. Subjectivity
  10. 10. Relevance
  11. 11. It’s complicated…
  12. 12. Boy with Blond Hair ca. 1840–50 1973.323.5 Madame Georges Charpentier (Marguérite-Louise Lemonnier, 1848–1904) and Her Children, Georgette-Berthe (1872–1945) and Paul-Émile-Charles (1875–1895) Auguste Renoir, French, Limoges 1841–1919 Cagnes-sur-Mer 1878 07.122 John Yellow Flower, No. 40, collector card from the American Indian Series (D6), issued by the Kelley Baking Company to promote Kelley's Bread Issued by Kelley Baking Company 1940 63.350.307.6.27
  13. 13. Kaggle Competition
  14. 14. “Kernels Only” Competition
  15. 15. Tensor Flow Hub- public model repository
  16. 16. AI Challenges • Lack of Developer Resources • Imperfect Training Data o Subjectivity o Completeness o Accuracy o Relevance • Not Enough Training Data (we only have 600 cats…) • No Right Answers for Tagging Art • Bias
  17. 17. Human vs. Machine AI-assisted tagging for artworks Elena Villaespesa (@elenustika) Seth Crider (@SethCrider2) Pratt Institute
  18. 18. 1,414 objects with images on the public domain The Met - Highlights
  19. 19. Tags usage (long tail and top tags) Human tags Google Amazon
  20. 20. Google (918) Human tags (537) Amazon (733) 26 Unique tags 286 20 There is a small number of tags that are applied both by the museum and these algorithms Note: exact tags, singular vs plural 12 Human - tag of sentiments, actions, what is depicted Machine - art form, material, color, art movements,
  21. 21. Accuracy Diaper bag Birthday cake Skateboard Accuracy is one of the primary challenges of these tags.
  22. 22. Accuracy Weapon 3D modeling Modern art Accuracy is one of the primary challenges of these tags.
  23. 23. Accuracy and confidence score Person 99.873642 Human 99.873642 Clothing 98.1922913 Apparel 98.1922913 Transportation 87.5043945 Boat 87.5043945 Vehicle 87.5043945 Airfield 71.124527 Airport 71.124527 Tire 70.2007675 Pants 65.6449432 Watercraft 61.6692047 Vessel 61.6692047 Hat 61.4706268 Machine 59.0481873 Spoke 59.0481873 Shorts 56.8293343 Coat 55.262558 Overcoat 55.262558 Wheel 55.0610924 Alloy Wheel 55.0610924
  24. 24. Subjectivity: Medium and art period Human Forests, Landscapes, Oaks Google Nature, Tree, Leaf, Snapshot, Branch, Monochrome, Woody plant, Rock, Stock photography, Organism Amazon Nature, Outdoors, Landscape, Weather, Tree, Plant, Scenery, Rug, Snow, Art, Painting, Vegetation, Winter, Ice, Land, Woodland, Forest Human Hieroglyphs Google Hose Amazon Pendant Scarab of the Storehouse Overseer Wah (Egypt) Oak Tree and Rocks, Forest of Fontainebleau by Gustave Le Gray
  25. 25. Completeness Human Bears, Centaurs, Deer, Men, Hunting, Satyrs, Dogs, Forests, Lions Google Painting, Art, Visual arts, Mythology, Stock photography, Modern art Amazon Art, Painting Human Interiors, Girls, Men, Women, Smoking, Dogs Google Visual arts, Art, Painting Amazon Human, Person, Art, Painting
  26. 26. Relevance Clothing Forehead Art
  27. 27. Context Human George Washington, Men, Portraits Google Portrait, Self-portrait, Gentleman, Lady, Painting, Art, Barrister, Elder Amazon Painting, Art, Person, Human George Washington By Gilbert Stuart The context of historical or political figures is not captured by the machine tags.
  28. 28. Context Cleopatra By William Wetmore Story Human Cleopatra, Women Google Sculpture, Statue, Classical sculpture, Figurine, Stone carving, Art, Carving, Monument, Marble, Mythology Amazon Art, Sculpture, Statue, Figurine, Person, Human, Archaeology
  29. 29. The usage of gender-related tags (Female, Lady, Gettleman, Man…) is low and neutral tags such as figure, person or human are used. Gender Human Apples, Male Nudes Google Sculpture, Bronze sculpture, Statue, Art, Standing, Figurine, Classical sculpture, Metal, Bronze, Human Amazon Sculpture, Art, Statue, Person, Human, Torso, Bronze, Coat, Apparel, Clothing, Overcoat, Suit, Tire, Figurine Paris By Antico (Pier Jacopo Alari Bonacolsi)
  30. 30. Tags to improve the Online Collection UX
  31. 31. Online Collection - User feedback I was looking for reference photos for an 18th century japanese bedroom. It would help if all the subject matter of ancient pictures was hashtagged in a way that I could advance search, along with time period. I would love more tags on historical pieces so they are easier to search. Then, I could come directly to The Met site instead of searching through Pinterest. Have a search function for subject matter Link with the keyword with other collection of the MET, like word cartonnage and after link to picture of egyptian cartonnage mummy, and after restoration of the cartonnage and after pigmentation use to do the paint etc etc.. Narrowing down search results using the filters is not always effective; I wish there were more specific categories that could be browsed more easily, like "Spanish painting," etc. -- perhaps through tags or something similar? Organize the art works by theme/subject in addition to country/region i.e. nature, abstraction, religion, political themes. In this way, I could tie the art works to subject specific curriculum such as history, geography, government, etc.
  32. 32. How can tags improve the Search user experience? ● Improve discoverability of object that do not have these keywords on the title or object description ● Respond to user needs and current searches on the Online Collection ● Potentially machine tags complement the taxonomy with a variety of keywords
  33. 33. Search Analytics 6% website users use the search functionality 10% Online Collection users 3.3M searches 875K keywords Source: Google Analytics (Oct 2018 - Sep 2019)
  34. 34. Search - Birds
  35. 35. Search - Dance
  36. 36. Search volume and top searches Human Tags 208K Google Vision 205K Amazon 198K Painting Guitar Fashion Portrait Dress Landscape Sculpture Map Ceramic Mask Flower Costume Photography Still life Drawing Sword Cat Music Textile Dog Guitars Portraits Landscapes Maps Masks Flowers Costumes Painting Armor Sculpture Jewelry Still Life Japanese Swords Cats Music Buddha Dogs Cloisters Musical Instruments Painting Guitar Fashion Portrait Dress Landscape Sculpture Map Mask Flower Costume Armor Jewelry Photography Drawing Sword Cat Buddha Dog Corset Human Google Amazon
  37. 37. ● Accuracy and lack of context are the major challenges of using these technologies ● Usage of image recognition can generate labels that may increase the diversity of the terms used to tag the collection ● These tags can significantly increase the discoverability of the collection artworks via search, navigation, SEO Further analysis: ● Analysis of the impact on search analytics (e.g. search exit rate) ● Include only tags with high levels of confidence ● Collect and analyse tags from computer vision tools (e.g. Clarifai, Imagga, Microsoft…) ● Gather user feedback via user testing/eye tracking on the usefulness of these tags (display info about the source of the tag, usage, etc) Conclusion
  38. 38. Wikipedia articles Wikimedia environment 50 million pages in 200+ languages English: 5.9 million articles Britannica < 500,000 Highly notable topics
  39. 39. Wikipedia articles Wikicommons media files Wikimedia environment 50 million pages in 200+ languages English: 5.9 million articles Britannica < 500,000 Highly notable topics 56 million media files 500+ million views per month Wide project scope
  40. 40. Wikipedia articles Wikidata items ● Structured database of all notable figures/works ● Language independent, rich metadata ● Supports comprehensive linkages to collections ● Searchable, interactive, scalable Linked Open Data Wikicommons media files Wikimedia environment – Focus on Wikidata contributions
  41. 41. Wikidata Potential Try it! Interconnected knowledge graph of culture: art, fashion, literature
  42. 42. 1 - AI machine learning Met subject keywords used to train machine learning model Use image classifier to predict labels for other artworks Training takes hours, but predictions are fast (multiple per second) Create Wikidata Game to help assess predicted labels and add to Wikidata
  43. 43. Wikidata Game using Met-trained machine learning engine - link
  44. 44. Depiction information added to Wikidata watcher - link
  45. 45. Met AI experiments - Met blog post "...even such a high measure of confidence becomes useless if one cannot sift the incorrect classifications from the correct ones. This is where the Wikimedia community comes in."
  46. 46. Results of Wikidata Game - Depicts Focused on 2D artworks such as paintings More than 7,000 judgments via the game resulting in ~5,000 edits Depiction topics - tree, boat, flower, horse, soldier, house, ship landscape painting features performed well Gender determination, cats, and dogs not so well Wikimedia Commons putting resources into similar ML capabilities
  47. 47. Depiction judgments One judgment = one live edit to Wikidata Recruiting and retaining a user much more expensive than undoing vandalism Users can inspect and patrol edits of bad faith editors (and block them) For AI, Wikimedia editors are perhaps the best humans-in-the-loop
  48. 48. 2 - Status - Live SPARQL dashboards of Met collections Most commonly depicted themes In Met artworks (partial, Jan 2019)
  49. 49. 2 - Status - Live SPARQL dashboards of Met collections Most commonly depicted themes In Met artworks (partial, Nov 2019)
  50. 50. SXSW presentation /vote/100500 March 2020
  51. 51. Future work Feed judgments back into ML model to refine the neural net Perform training for specific artwork types and domains - paintings vs sculpture vs costumes/fashion
  52. 52. Future work ML image classification as a "suggestion module" for other tools Example: Wiki Art Depiction Explorer (Knight Foundation-funded project) Suggest Met AI-generated tags
  53. 53. Wiki Art Depiction Explorer -
  54. 54. Wiki Art Depiction Explorer -
  55. 55. WADE -
  56. 56. WADE possible interface - suggestion from Met ML model Automatically generated tags
  57. 57. Conclusions Promising exploratory work combines best of both worlds: scale of ML/AI operations + expertise of the best volunteer community Caveats: ● Are we reproducing systemic/historical biases in the ML models? ● Incorporating better metadata and vocabularies for non-Western art