Semantics in Visual Perception: Methodological Remarks, Issues and Perspectives


Published on

Published in: Education, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Semantics in Visual Perception: Methodological Remarks, Issues and Perspectives

  1. 1. Semantics in Visual Perception: Methodological Remarks, Issues and Perspectives Jurgis Skilters Center for Cognitive Sciences and Semantics University of Latvia, Latvia Thanks to my collaborators and colleagues: Baingio Pinna (Sassari, Italy),4-5.11.2010, Aarhus, Jānis Pencis (CCSS, Riga) Methods in Language&Cognition 1Denmark
  2. 2. The aim of my talk• to provide some methodological preliminaries and some parameters for research on the interaction between visual perception and semantics.4-5.11.2010, Aarhus, Methods in Language&Cognition 2Denmark
  3. 3. interference effects between language and visual perception:• Language is an access to human cognition.• Visual perception is another access to human cognition• ....4-5.11.2010, Aarhus, Methods in Language&Cognition 3Denmark
  4. 4. interference effects between language and visual perception:• we do have representations of linguistic stimuli• we do have certain representations of visual stimuli.• are they the same representations?• What does happen when we look at visual stimuli and when we process (either produce or comprehend) language?4-5.11.2010, Aarhus, Methods in Language&Cognition 4Denmark
  5. 5. Outline:1. Visual perception and the role of psychophysics2. How does semantics arise in perception?3. Invariants4. Affordances5. Events6. Experience7. Objects: the basic principles of their perception8. Perceived causality, motion perception9. Semantics in Vision: Multimodality effects10. Remarks on spatial cognition11. Methods from semantics and perception (brief overview)12. Conclusion: Constraints4- Methods in Language&Cognition 55.11.2010, Aarhus, Den
  6. 6. 1. Visual perception and the role of psychophysics4-5.11.2010, Aarhus, Methods in Language&Cognition 6Denmark
  7. 7. • The methods we chose depend on our research motivation.• Relationships between stimuli and physiological response Φ• Relationships between stimuli and perceptual reaction ψ• linking relationship (L) between perceptual and physiological response.4- Methods in Language&Cognition 75.11.2010, Aarhus, Den
  8. 8. Cp. Goldstein, 2005 Stimuli Φ ψPhysiological reaction L perceptual reaction 4-5.11.2010, Aarhus, Methods in Language&Cognition 8 Denmark
  9. 9. Stimuli Φ ψPhysiological reaction L perceptual reaction 4-5.11.2010, Aarhus, Methods in Language&Cognition 9 Denmark
  10. 10. • How are relations ψ studied and explored?• methods formulated by Fechner (1860) to determine thresholds,• methods for magnitude estimation (above the threshold formulated by Stevens (1961)),• phenomenological observations (Katz, 1935, Metzger 2006/1936);• also identification measures, recognition, reaction time research…. Also overlapping with Φ4-5.11.2010, Aarhus, Methods in Language&Cognition 10Denmark
  11. 11. • there is no direct correspondence between stimuli and perceptual result: – different stimuli can cause the same perception and (physically) the same stimulus in different contexts can cause different perceptual results. – we can perceive something without the sensorial basis for it, • e.g., we can perceive movement / causality / motion where there is no movement at the level of physical stimuli.And this has nothing to do with semantics but with the principles of our visual perception.4-5.11.2010, Aarhus, Methods in Language&Cognition 11Denmark
  12. 12. the binding problem:• how is a unitary perceptual experience generated that combines qualities such as color, shape, location, orientation (Goldstein, 2005, 12)?4-5.11.2010, Aarhus, Methods in Language&Cognition 12Denmark
  13. 13. Most of the models are hierarchical:• top level encodes the whole / figural representation;• subsequent / lower levels encode parts and relations between them (Palmer, 1975, 1977, 1978, Carlson- Radvansky & Irwin, 1995)• 1. Grouping• 2. shape assignment• 3. meaning assignment• there are also parallel models assuming that object memories critically determine object recognition and also grouping and shape assignment factors.• NB: grouping and shape assignment processes function without attention, intention and awareness.4-5.11.2010, Aarhus, Methods in Language&Cognition 13Denmark
  14. 14. a. Classical serial hierarchical theory (Marr) (picture from Peterson, 2005)4-5.11.2010, Aarhus, Methods in Language&Cognition 14Denmark
  15. 15. b. interactive hierarchy model (picture from Peterson, 2005)4-5.11.2010, Aarhus, Methods in Language&Cognition 15Denmark
  16. 16. c. Parallel interactive model (picture from Peterson, 2005)4-5.11.2010, Aarhus, Methods in Language&Cognition 16Denmark
  17. 17. experiencestimuli Grouping Shape meaning assignment situation 4-5.11.2010, Aarhus, Methods in Language&Cognition 17 Denmark
  18. 18. 2. How does semantics arise in perception?4-5.11.2010, Aarhus, Methods in Language&Cognition 18Denmark
  19. 19. 1. visual field is segmented, the components are grouped together;2. shape assignment takes place3. semantics is assigned.• And finally….. the result is lexicalized.• NB all these processes are interactive and there is not one way of bottom-up or top-down determination.• NB we „translate‟ continuous perceptual material into discontinuous language. As soon as we lexicalize we make the continuous discontinuous.4-5.11.2010, Aarhus, Methods in Language&Cognition 19Denmark
  20. 20. Gouping ShapeConcrete stimuli assignment Concepts language Meaning assignment meanings Visual processing Continuous Discontinuous / 4-5.11.2010, Aarhus, Methods in Language&Cognition discrete 20 Denmark
  21. 21. GroupingRows vs. columns and the rectangle illusion.4-5.11.2010, Aarhus, Methods in Language&Cognition 21Denmark
  22. 22. The role of directional symmetry in forming the shape The inverted rod and frame illusion4-5.11.2010, Aarhus, Methods in Language&Cognition 22Denmark
  23. 23. The directional symmetry influences the shape The pointing illusion.4-5.11.2010, Aarhus, Methods in Language&Cognition 23Denmark
  24. 24. • Further, we assign border / contour to the figural regions. We discriminate figure from ground.4-5.11.2010, Aarhus, Methods in Language&Cognition 24Denmark
  25. 25. From grouping to shape4-5.11.2010, Aarhus, Methods in Language&Cognition 25Denmark
  26. 26. • And finally we assign meaning to what we see. – We simply do not stop the perceptual activity after grouping and shape-building4-5.11.2010, Aarhus, Methods in Language&Cognition 26Denmark
  27. 27. Form of meaning: More complex resultsThe illusion of meaning: A square showing different kinds of happenings4-5.11.2010, Aarhus, Methods in Language&Cognition 27Denmark
  28. 28. 3. Invariants4-5.11.2010, Aarhus, Methods in Language&Cognition 28Denmark
  29. 29. There is consensus among vision scientists:• at least some structures remain invariant across changes (cp. Carlson-Radvansky & Irwin, 1995, Pinna, Tanca, & Skilters, 2010).• Every type belongs to a different level of perceptual processing and every level contains invariants of a corresponding type. – According to certain invariants from type „grouping‟ the perceptual material (from zero-level objects) is transformed to higher level and processed according to the type „shape‟ and, finally, to meaning.• when one unit (structural invariant in our case) determines another, the one must belong to the higher level (type) than the other (cp. Jeffrey 1981).4-5.11.2010, Aarhus, Methods in Language&Cognition 29Denmark
  30. 30. • invariants itself are not objects of the particular levels; they are principles that structure the organization of particular visual material.4-5.11.2010, Aarhus, Methods in Language&Cognition 30Denmark
  31. 31. • Every level of perception contains structural invariants determining the processing of lower level.• The third-level invariants – meaning – are the strongest• the structural invariants at the level of meaning assignment are the strongest• dissimilarities and multiplicities in lower level structures become units and their dependencies become stronger with every higher level of processing.• strength of functional dependencies increases from lower to higher level4-5.11.2010, Aarhus, Methods in Language&Cognition 31Denmark
  32. 32. 4. Affordances4-5.11.2010, Aarhus, Methods in Language&Cognition 32Denmark
  33. 33. • meaning as an affordance-sensitive structure.• affordances are dispositional properties of objects and events.• „Affordances are the acts or behaviors permitted by objects, places, and events.“ (Michaels & Carello, 1981, 42).• affordances differs in different organisms4-5.11.2010, Aarhus, Methods in Language&Cognition 33Denmark
  34. 34. • two mutually connected aspects: action system and environment. – „The action system (effectivity structure) and the environment (affordance structure) are in a relationship of mutual constraint.“ (Michaels & Carello, 1981, 54, cp. also. Shaw, Turvey, & Mace, 1982, 209)• Affordances are processed to a large part un- intentionally: objects potentiate a range of actions associated to them but irrespective of intentions (Ellis &Tucker, 2000).• the affordances change from one moment or situation to another. – more permanent and more time-dependant („episodic“, cp. Glenberg et al. 2009) affordances can be gradually distinguished.4- Methods in Language&Cognition 345.11.2010, Aarhus, Den
  35. 35. • Meaning “consists of the set of actions the individual can undertake in that situation. The set of actions is determined by the goal-directed mesh of affordances.” (Glenberg et. al., 2007, 223). – In a particular situation an individual modifies the meaning according to his/her experience and bodily configuration.• Affordances constrain meaning but affordances are constrained by the actual situation and experiential factors of the agent as well.• the situation determines meaning; differences in affordances result in differences in meaning.4-5.11.2010, Aarhus, Methods in Language&Cognition 35Denmark
  36. 36. 5. Events4-5.11.2010, Aarhus, Methods in Language&Cognition 36Denmark
  37. 37. Preliminary• Have to be distinguished:• optic flow: when observer moves through environment, where entire field of view is determining,• and• perception of object motion: looking at a point in visual field, where only limited segments of visual field are co-involved (cp. Shiffrar, 2005).4-5.11.2010, Aarhus, Methods in Language&Cognition 37Denmark
  38. 38. • Visual perception takes place according to events:• Events are time-space-linked structures• “If events are the significant units of the world, the world must be described in a way that preserves their integrity. The world must be described in terms of both time and space.” (Michaels & Carello, 1981, 10)4-5.11.2010, Aarhus, Methods in Language&Cognition 38Denmark
  39. 39. • Events are cognitively perceived as goal- oriented patronymic hierarchies (Zacks, Tversky, & Iyer, 2001)• and later lexicalized in object/individual- predicate (object-action) structure (guiding hypothesis of Pencis, Pinna, & Skilters, 2010).4-5.11.2010, Aarhus, Methods in Language&Cognition 39Denmark
  40. 40. • events as segments of time at a given location, perceived as having a beginning and an end, cp. Zacks, Tversky, & Iyer, 2001; – based on partonymic hierarchies.• Even material presented in pictures is perceived in time and according to hierarchic event- structure. – recognition of intentional action on objects is needed (Tversky, Zacks, & Hard, 2008, 461).4-5.11.2010, Aarhus, Methods in Language&Cognition 40Denmark
  41. 41. Correlation of three processes(1) decomposition of visual material in spatial parts(2) decomposition of events in temporal parts(3) lexicalization.functional dependencies and relations are generated.hierarchical bias hypothesis: people are spontaneously disposed to actively encode ongoing activity in terms of a hierarchical part-whole structure; Zacks, Tversky, & Iyer, 20014-5.11.2010, Aarhus, Methods in Language&Cognition 41Denmark
  42. 42. • Goal/sub-goal relationship• a single goal in sense of motion, causality is assigned to a visual representation even in case of minimal stimuli (even in case where there is no objective goal or motion or causality at all).• In a narrative we can observe hierarchic event perception more explicitly (cp. also Abbott, Black, & Smith, 1985, Tversky, Zacks, & Hard, 2008). Faster narrative production is if there is one higher level event (Foss & Bower, 1986, Zacks, Tversky, & Iyer, 2001)• parts as visual segments and parts as sub-goals.• parts are functionally different• fast inference from visual appearance to function• link between perception and function is always present even in case of minimal stimuli – (for the perception-to-function hypothesis cp. Tversky, Zacks, & Hard, 2008)4-5.11.2010, Aarhus, Methods in Language&Cognition 42Denmark
  43. 43. • Objects are integral and crucial to perceive actions and vice versa (Baldwin & Baird, 1999)• actions and objects are comprehended together (Zacks, Tversky, & Iyer, 2001);• we always perceive object-activity- constellations.• actions and objects are mutually constraining (Tversky, Zacks, & Hard, 2008) – Object-action matrices are occurring in events and allow only certain combinations of substituted instances.4-5.11.2010, Aarhus, Methods in Language&Cognition 43Denmark
  44. 44. 4-5.11.2010, Aarhus, Methods in Language&Cognition 44Denmark
  45. 45. 6. Experience4-5.11.2010, Aarhus, Methods in Language&Cognition 45Denmark
  46. 46. (a) interrelations between visual perception and experience vary according to the material under consideration(b) and more crucially – experience effects are not atomistic / summative, simplistic but rather holistic.the question is not whether but rather how experience influences visual perception.• Meaning assignment as resonation of visual stimulus with experiential structures.4-5.11.2010, Aarhus, Methods in Language&Cognition 46Denmark
  47. 47. 7. Objects: the basic principles of their perception4-5.11.2010, Aarhus, Methods in Language&Cognition 47Denmark
  48. 48. • what counts as an object?• Conceptions of object recognition1. we recognize objects by components. I.e., there are some core elements of visual objects (geons) which are basic for building together any complex object. – research tradition by Biederman, Marr – One consequence from Recognition by Components Theory is that object recognition is viewpoint invariant. It might be true but if this is the case we would perhaps see no difference between pigs and dogs.2. Multiple Views Theory – tradition of Buelthoff, Edelman and Tarr• there are multiple two-dimensional views that enable recognition of object. Object recognition is view- dependant because objects observed in new views have to undergo some time-consuming process before they are matched to stored views and recognized.4-5.11.2010, Aarhus, Methods in Language&Cognition 48Denmark
  49. 49. 8. Perceived causality, motion perception4-5.11.2010, Aarhus, Methods in Language&Cognition 49Denmark
  50. 50. • Even when there is not causality at the level of stimuli there is causality at the level of perception.• Integration of motion• there is a tendency to assign intentionality to „moving‟ objects even if they are simple geometric stimuli. (For a single stimulus approach cp. Pinna & Skilters, for a classic study of moving geometric objects cp. Heider & Simmel, 1944.)4-5.11.2010, Aarhus, Methods in Language&Cognition 50Denmark
  51. 51. • In case of animated objects• Heider, F. & Simmel, M. (1944). An experimental study of apparent behaviour. American Journal of Psychology, 13, 1944.• http://www.all-about- or• Ker6PMtM4-5.11.2010, Aarhus, Methods in Language&Cognition 51Denmark
  52. 52. Unanimated / single stimuli4-5.11.2010, Aarhus, Methods in Language&Cognition 52Denmark
  53. 53. 4- Methods in Language&Cognition 535.11.2010, Aarhus, Den
  54. 54. 4-5.11.2010, Aarhus, Methods in Language&Cognition 54Denmark
  55. 55. 4-5.11.2010, Aarhus, Methods in Language&Cognition 55Denmark
  56. 56. • One object according to the reports of naïve perceivers is causing changes in another object.• Our memory for the spatial location of an object is biased towards objects motion even if there is no motion or even if the object is static or geometric (cp. Shiffrar, 2005, 257),• moreover, the motion assigned to non-animated objects is frequently like human-movement.4-5.11.2010, Aarhus, Methods in Language&Cognition 56Denmark
  57. 57. 9. Semantics in Vision: Multimodality effects4-5.11.2010, Aarhus, Methods in Language&Cognition 57Denmark
  58. 58. • Vision is influenced by other expressions of cognition – Vestibular system – Motor system – Auditory system Also purely visual stimulus may induce some e.g. quasi-auditory perception in simulating collision4-5.11.2010, Aarhus, Methods in Language&Cognition 58Denmark
  59. 59. 4-5.11.2010, Aarhus, Methods in Language&Cognition 59Denmark
  60. 60. 4-5.11.2010, Aarhus, Methods in Language&Cognition 60Denmark
  61. 61. cognition Visual Vestibular Motor Auditoryperception system system system4- Methods in Language&Cognition 615.11.2010, Aarhus, Den
  62. 62. cognition Visual Vestibular Motor Auditoryperception system system system4-5.11.2010, Aarhus, Methods in Language&Cognition 62Denmark
  63. 63. 10. Remarks on spatial cognition4- Methods in Language&Cognition 635.11.2010, Aarhus, Den
  64. 64. • language provides strong constraints – selectivity and enrichment – modulating the expression of spatial cognition1. Language with the basic components – symbolic units – noun and verbs and configurations of these give rise to semantic and syntactic functions of agent, patient, subject, object.2. In contrast spatial system contain primitives such as shapes, objects, locations, landmarks, geometric layouts, angles and directions – represented in different spatial reference systems• (Landau, Dessalegn, & Goldberg, 2010)4-5.11.2010, Aarhus, Methods in Language&Cognition 64Denmark
  65. 65. 11. Minimal semantics generated in visual perception4-5.11.2010, Aarhus, Methods in Language&Cognition 65Denmark
  66. 66. • parallelism between visual and language processing• First, we have to build a complementary methodology, using both psychophysical methods and semantic methods.• Second – what to start with?• Primitive perceptual semantics. Very rudimentary stimuli can generate a lot of semantics4-5.11.2010, Aarhus, Methods in Language&Cognition 66Denmark
  67. 67. Object and predicate / thing and action structure4-5.11.2010, Aarhus, Methods in Language&Cognition 67Denmark
  68. 68. action-resonance effects- we see not separate objects but action- resonated objects.- we do recognize events (instead of separate objects) as soon as we assign meaning (even is case of relatively minimal and poor stimuli).4-5.11.2010, Aarhus, Methods in Language&Cognition 68Denmark
  69. 69. 11. Methods from semantics Different biased and non-biased tasks (e.g., sentence completion tasks, priming experiments, self-report measures). (Cp. Hasson, U. & Giora, R. 2007).4-5.11.2010, Aarhus, Methods in Language&Cognition 69Denmark
  70. 70. • Lexical decision tasks are used to determine whether a letter string on the screen makes a word.• prime and target.4-5.11.2010, Aarhus, Methods in Language&Cognition 70Denmark
  71. 71. faster Related prime target Unrelated control target prime4-5.11.2010, Aarhus, Methods in Language&Cognition 71Denmark
  72. 72. • Memory measures• learning stage and a test stage• free recall method.• old / new recognition task• Self report measures: Listing features. The task is to write down properties that capture the meaning of an expression (or visual stimulus).4-5.11.2010, Aarhus, Methods in Language&Cognition 72Denmark
  73. 73. • Effects of comprehension on subsequent tasks. Priming paradigms. How comprehension of one stimulus affects comprehension of another stimuli.• word / sentence fragment completion task• Negative priming (Christie & Klein, 2008, May, Kane, & Hasher, 1995, MacLeod, Chiappe, & Fox, 2002)• Different modifications: some distractor determines the target and decreases activation time.• Unconscious / masked priming (Dehaene, S., Naccache, L., Le ClecH, G.., Koechlin, E., Mueller, M., Dehaene-Lambertz, G., van de Moortele, P. F., & LeBihan, D., 1998)4-5.11.2010, Aarhus, Methods in Language&Cognition 73Denmark
  74. 74. Perception4-5.11.2010, Aarhus, Methods in Language&Cognition 74Denmark
  75. 75. • Central assumption: stimulus and sensory are not directly corresponding and proportional4-5.11.2010, Aarhus, Methods in Language&Cognition 75Denmark
  76. 76. Classically: Threshold determination• Method of average error• Method of minimal changes• Constant stimuli method4-5.11.2010, Aarhus, Methods in Language&Cognition 76Denmark
  77. 77. Classically: Magnitude estimation• Magnitude estimation with standard:• A stimulus is presented (standard) and a number is assigned to it (modulus, e.g., 30).• The following stimuli are evaluated by subjects in that they report the intensity relative to the standard in assigning the stimuli numerical evaluation. The ratio between numerical estimated and sensations.• Magnitude estimation without standard:• Subjects can choose their own standard (any number) and assign it to the first stimulus. All following are assigned by subject; preserving the ratio between sensations and numbers4-5.11.2010, Aarhus, Methods in Language&Cognition 77Denmark
  78. 78. • Cross-modality matching:• Indirect methods to differentiate between stimuli• Direct scaling methods• Methods for stimulus comparison• Some remarks regarding the use of semantic differential in4-5.11.2010, Aarhus, Methods in Language&Cognition 78Denmark
  79. 79. • research on attitudes:• Semantic differential is a fast way how to measure attitudes. (The crucial point is that no more than attitudes can be measured in using semantic differential methodology.)• Visual categorization and the use of quantitative methods as complementary tools in research of semantics in visual perception.• basic level prominence effects (Rosch, 1978, Tversky, Zacks, & Hard, 2008).4- Methods in Language&Cognition 795.11.2010, Aarhus, Den
  80. 80. 12. Conclusion: Constraints4-5.11.2010, Aarhus, Methods in Language&Cognition 80Denmark
  81. 81. • Lexicalization itself / dinscontiuty generation• Selectivity: hierarchic – both in object encoding and in encoding objects in events• Attention• How much can be actually perceived at a moment – Saccades and fixations are mediated and correlated with transsacadic memory (some information retained from one eye fixation is invoked in processing that occurs during the next fixation).• Invariants• Experiential constraints• Situational constraints4- Methods in Language&Cognition 815.11.2010, Aarhus, Den