Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Warren Hayes Ai Symposium - working draft

1,777 views

Published on

A Lightweight Ontology for Describing Images
(working draft)

Published in: Technology
  • Be the first to comment

Warren Hayes Ai Symposium - working draft

  1. 1. A Lightweight Ontology for Linking Images Pat Hayes, Margaret Warren © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  2. 2. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  3. 3. what we are/and are not doing • images themselves are our main topic, rather than as illustrations of other topics, linking information to and from them and linking the images themselves • Interested in using linked data to represent the kinds of things artists themselves say about their work (or the person doing the markup) • What we don’t do: expect people to agree on artistic interpretations or emotional content of images • ignore our paper in the proceedings – it was a working draft and we have made significant changes. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  4. 4. foaf:depiction © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  5. 5. Snapshot, line drawings, news photos, paintings, fine art © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  6. 6. Image identity • By Image, we mean a digital image, typically one that can have associated EXIF data. • So, other kinds of "images" (drawings, paintings, non-digital photographs, stained glass, etc..) have to be classified as Works. • A work may be a physical thing with dimensions, a location, a time and place of creation, etc. that is separate from those same properties in an image; a digital image can be a work. • Basic relation between Work and Image is facsimile, a kind of 'visually same As' relation. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  7. 7. Broader notion of depicts All images depict the person: McKenzie Oerting © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  8. 8. Depiction can include imaginary, biblical, mythological notions © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  9. 9. What is the subject? © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  10. 10. shows, depicts, subject, conveys Shows: trees, grass, house, window, ventilator Shows: head Depicts: particular type of car, Depicts: hands each of the people Conveys: grief, sadness Subject: group looking at a car © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  11. 11. Do all images depict something? Depict diamonds? Shapes? Or Nothing… © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  12. 12. Pictorial elements Basic graphical 'unit' of depiction is a pictorial element. This includes 'boxes' but also lines, regions, areas, patches of color, texture, etc.. “ elements of art - The basic components used by the artist when producing works of art. Those elements are color, value, line, shape, form, texture, and space. The elements of art are among the literal qualities found in any artwork. (Artlex, 1996-2010) Pictorial elements are related by inclusion: 3/23/2010 © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
  13. 13. Depicts vs. looksLike Pictorial Elements: Shape of thing that looks like a female torso Shape of thing that looks like a representation of a brain area that looks like the torso is sitting astride a seat of some sort Title: Surfing With Psyche Depicts: egg, a test tube, a photographic transparent layer, a piece of jewelry © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  14. 14. Pictorial Elements can be 'empty' A reclining nude is depicted by white space defined by an outline. The white space is a pictorial element. Cataloging pictorial elements is a large task. As a beginning, we distinguish visible/implied and line/area. Here, the nude is depicted by an implied area. All the visible elements are red lines. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  15. 15. backgrounds hasPictorialBackground hasDepictedBackground all white, not necessarily sky, clouds, not necessary able to define with a region to define with a region © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  16. 16. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  17. 17. Angels in the clouds depicts: Blue Angels ,jets, planes, formation conveys: military power? subject: specific formation? hasDepictedBackground: sky, clouds depicts: angel shows: clouds region contains a shape (PictorialElement) that looksLike an Angel (iconic) depicts: clouds © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  18. 18. The ontology so far © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  19. 19. future work © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  20. 20. http://www.purl.org/net/mw http://www.carmapro.com/mw/mw.rdf © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  21. 21. Extra slides follow © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  22. 22. Looking like an archetype • Some shapes are best described by what they look like iconically. • If we can use RDFS or OWL Full, this could be described as 'looking like' a class; but this will break OWL-DL reasoners. • Or, we could treat a SKOS concept as an archetype, but this is not (yet) a widely accepted convention. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  23. 23. visiblePictorialElements looksLike iconic heart hasColor Red looksLike iconic dancing figures hasColor yellow orange red hasPictorialBackground: white © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  24. 24. What's in an image? • We conventionally say that depicted things are "in" an image. • Is this a philosophical mistake? Maybe. But rather than try to correct it, the ontology tries to avoid it. • Basic methodological rule for linked data: don't try to repair "bad" intuitions. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  25. 25. shows: person, depicts: face in hands shows: broken headlight, turn signal subject: crying conveys: grief or despair, etc. depicts: front fender of school bus subject: conveys: beauty in decay, decline of education © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  26. 26. Can any Thing be depicted? Do we need a class of 'depictable things'? Things depicted include (keywords): • the car (specific type) • people (specific people) • a group of 7 people • a building Subject: group studying a car? • trees one of the people? the car itself? • brown grass © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  27. 27. Adult, Adults Only, Agriculture, Beautiful People, Beauty, Beauty In Nature, Biting, Branch, Color Image, Concepts, Custard Apple, Desire, Enjoyment, Food, Freshness, Fruit, Green, Gripping, Growth, Harvesting, Healthy Eating, Healthy Lifestyle, Horizontal, Human Face, Human Hand, Human Mouth, Hungry, Leaf, Lifestyles, Meal, Nutrient, One Person, One Woman Only, One Young Woman Only, Only Women, Open, Opening, Organic, People, Photography, Picking, Plant, Southeast Asian Ethnicity, The Human Body, Tree, Vegetarian Food, Women, Young Adult © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  28. 28. What can be depicted? • FOAF: People. • DBPedia: People, places, events,… • Umbel (Cyc): VisibleThings • Visual Resources Association: myths, legends, intentions, … • We provide a different property conveys for the relationship between an image and "un-depictable" things such as mood, idea, intention, emotion, etc.. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  29. 29. http://www.gettyimages.com/Search/Search.aspx?query=z.i.H4 sIAAAAAAAEAO29B2AcSZYlJi9tynt_SvVK1- B0oQiAYBMk2JBAEOzBiM3mkuwdaUcjKasqgcplVmVdZhZAz O2dvPfee-- 999577733ujudTif33_8_XGZkAWz2zkrayZ4hgKrIHz9- fB8_In7dfLn91etf49f4NX6PX_dskV3kvyb9mtD_f9N5ns3S9fKqz larYnnxa_7f9PwaG59f0_xM6qpqX2Z1tmjMZ78W_r- 78xvSj1_v3evr5vSdeePX9P7mltm5Bai_4- VfOzuf2g78P35tfLDD8Bete3PR_rr6-2_xa3DX9tc99-s99-t99-s- __rrUB-lhWf_Qne_Lv1RLyw24Z-_Pj7axV-_Hn8- sTA6f_Nr5yEU- ydD2bFQzjtQ3N_8Wm1e6_zJUPYslLoDxf1NBM0tDPMH_s_U 3QV1iQL5yr4f_PVr55PGfmP- 4BnJXY_e779W7pp7v__azWplP8cfBqPfyr54OXUvXk5_bf39t8 TfO_a3ffvbLn77dara4zb7F381yf2vzF94fu3Z1dL8_muaP8BSv0 6z9kYb_PVr55nDz_-Dfl_5X9g_ft0mgBb--etM83futeCvX_vds- dOGugP- 0VWe2h7f_xGzbRa5U_Wy1npUT_41AD8ccB5sHMfv_86xGpr2 zz46_8Bs9pv0zoEAAA. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  30. 30. Linked Image Data • Linked data… • Newest manifestation of the semantic web: use RDF to create a web of semantic links between data items. • Meanings of links are described by lightweight (small, intuitive, useful) ontologies. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  31. 31. Linked data, not data silos © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  32. 32. Linked data is link + description • RDF descriptions use URIs as names to refer to things; linking means using those URIs to also connect to the described thing, or a source of more information about it. • This can be smoothly integrated into HTML using RDFa. <foaf:knows> <foaf:Person rdf:about="http://danbri.org/foaf.rdf#danbri"> <foaf:mbox_sha1sum>362ce75324396f0aa2d3e5f1246f40bf3bb44401</foaf:mbox_sha1sum> <foaf:name>Dan Brickley</foaf:name> <rdfs:seeAlso rdf:resource="http://danbri.org/foaf.rdf"/> </foaf:Person> </foaf:knows> <foaf:based_near rdf:resource="http://dbpedia.org/resource/Milton_Keynes"/> <PersonalProfileDocument rdf:about=""> <cc:license rdf:resource="http://creativecommons.org/licenses/by-nc/3.0/"/> <dc:title>Tim Berners-Lee's FOAF file</dc:title> <maker rdf:resource="http://www.w3.org/People/Berners-Lee/card#i"/> <primaryTopic rdf:resource="http://www.w3.org/People/Berners-Lee/card#i"/> </PersonalProfileDocument> © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  33. 33. Linked data is link + description • Description terms defined by lightweight ontologies. • Designing lightweight ontologies is an art, not a science. • Often, the formal ontologies are quite minimal, eg FOAF refuses to say anything more than the class People is all people, and Agent is "things that make stuff happen". Still, it does specify that People is an rdfs:subClass of Agent. In practice, meaning is specified by use (in the actual links). • It is considered good form to re-use existing relations, classes and properties whenever possible. But this can set up some tension with the meaning-as-use observation. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  34. 34. Linked data is here, now Collectively, the data sets consist of over 13.1 billion RDF triples (dataset statistics), 3/23/2010 © which2010 Margaret Warren,around 142 million RDF links (linksset statistics) (November 2009). 2009 - are interlinked by Pat Hayes, CARMA Productions
  35. 35. Linked Image data? • There are many images in existing linked data, but almost always as illustrations of things (people, places, buildings..), as in FOAF. • We want to describe images as objects in their own right. Images are our topic. So we need an image-describing vocabulary with its own lightweight ontology. Earlier work on image description all focuses either on the things depicted, or on the cultural setting of the images, or on the curation and cataloging of museum collections. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  36. 36. Very lightweight image ontology Questions one might ask…. 1. Can any Thing be depicted? Do we need a class of 'depictable things'? 2. Do all images depict something? 3. What exactly is depiction anyway? 4. What exactly is an Image? (Is it a file type, for example? Are all images digital images?) etc… © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  37. 37. • Dense lines of particular colors, splatters, all pictorial elements • No distinct background, nothing depicted © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  38. 38. Pictorial elements Basic graphical 'unit' of depiction is a pictorial element. This includes 'boxes' but also lines, regions, areas, patches of color, texture, etc.. “ elements of art - The basic components used by the artist when producing works of art. Those elements are color, value, line, shape, form, texture, and space. The elements of art are among the literal qualities found in any artwork. (Artlex, 1996-2010) Pictorial elements are related by inclusion: 3/23/2010 © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
  39. 39. Ontology can be molded to fit A depiction of the torso of a female mannequin AND/OR A Pictorial Element that looks like a female torso © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  40. 40. Depicting vs. looking like • Simple depiction (as in FOAF) means something like "is a picture of". Our depiction is wider in scope, and does not imply visual similarity. An Xray image is a depiction, for example. So we also need the notion of looking like. • This is also important for characterizing pictorial elements. looksLike Depicted female Depicted female looksLike iconic female torso, x-ray torso female torso* © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions torso 3/23/2010
  41. 41. Images are pictorial elements. If an image can depict, then a PART of an image can depict. if a whole image can depict, so can part of an image. © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  42. 42. Entire image is a pictorial element Entire image is a pictorial element that looks like a female torso that depicts a female torso © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  43. 43. Depiction is in the eye of the … depicts Marie-Therese Walter depicts female torso © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010
  44. 44. Defining classes of image • Can now define for example 'line drawing': an image all of whose component visible pictorial elements is a line. <owl:Class rdf:ID="LineDrawing"> <owl:intersectionOf rdf:parseType="Collection"> <owl:Class rdf:about="#Image" /> <owl:Restriction> <owl:onProperty rdf:resource="#hasVisualPart" /> <owl:allValuesFrom> <owl:unionOf rdf:parseType="Collection"> <owl:Class rdf:about="#ImplicitPictorialElement" /> <owl:Class rdf:about="#LinearPictorialElement" /> </owl:unionOf> </owl:allValuesFrom> </owl:Restriction> </owl:intersectionOf> </owl:Class> © 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions 3/23/2010

×