Analysing Multimodal Data


Published on

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Analysing Multimodal Data

  1. 1. Cardiff Goldsmiths workshop Developing Capacity in Multimodal Research 4th -5th July 2013 Analysing multimodal data Bella Dicks
  2. 2. What are multimodal data? • meaning-making ‘beyond’ verbal language • research investigates the ‘multiplicity of how people communicate’ • how talk, facial/bodily movements and elements of material surroundings combine together in the production of meaning (Bezemer and Mavers, 2011: 192)
  3. 3. Epistemological commitments • 3 paradigms for multimodal data: – communication (social semiotics) – action (ethnomethodology) – experience (sensory ethnography) • Cross-cutting approaches: – Multimodal ethnography – Multimodal discourse analysis
  4. 4. Sensory ethnography • Key term: senses, perceptions • ‘experience’ is prime way of knowing the social world • Objective: understand how whole, sensing self is route into the experiential • methodology depends on a high degree of subjectivity and reflexivity: – Video tours – Sensory interviews • Video used to represent ‘place-making encounters’ between researcher, participants and the material/sensory environment Sarah Pink’s 2009 Doing Sensory Ethnography
  5. 5. Ethnomethodological approach • social action is prime means of knowing social world: • Key terms: modes, sequences, accomplishment • Objective: How social actors accomplish everyday practices through practical actions and reasoning – analyse how meaning emerges ‘from within the course of action’ • Methodology: in-situ, close-up observation of naturally- unfolding action • Video: camera tightly trained on action sequences • Transcripts: video, action, talk: disassembled • Close analysis: turn-taking, action as accomplishment of social practices (e.g. rebukes); gaze-direction; orientations to objects & ‘physical ecology’ etc. Heath and Hindmarsh’s 2010 Video in Qualitative Research
  6. 6. Social semiotics • Key terms: modes, semiotic resources, meaning-potential – Actors’ active choice of signifiers to communicate for specific social purposes – choices of mode affects kind of knowledge produced – Relate this to changes in wider communicational landscapes &‘knowledge-domains’ (discourse) • objective: documenting and inventorying diverse uses of distinct semiotic resources/modes • Methodology: embodied interactions; semiotic inventories (room layouts, magazines, texts, artefacts) • Video: represent use of semiotic resources in ‘sign-making practices’ • Analysis: interactional analysis; textual analysis Kress and Van Leeuwen’s 2001 Multimodal Discourse
  7. 7. Research frameworks • non-verbal domains of meaning are approached as: – resources (originating in the ‘interaction order’ (ethnomethodology) or ‘sign-making order’ (semiotics), respectively). – perceptions, originating in the self’s subjective being-in-the-world (sensory ethnography). ‘looking, touching and listening’ v. ‘seeing, feeling and hearing’
  8. 8. Example: analysis of interaction in science discovery • Take a social semiotic/interactional/sensory approach: • social interaction (actors + physical ecology) can be approached through analysis of embodied actions, talk, use of objects, gaze- directions, gestures, etc. • environments can be ‘read’ as multimodal orchestrations/assemblages (what are the semiotic resources afforded by physical setting and how do they interact?) • senses: think about textures as ‘feeling’; sounds as ‘hearing’; movement as ‘routes’; ‘followings’; ‘chains of excitement’…..
  9. 9. embodied interaction See also: Heath, C. and vom Lehn, D. (2008) ‘Configuring ‘Interactivity’: Enhancing Engagement in Science Centres and Museums’, Social Studies of Science 38(1): 63-91 Dicks, B. (2013) Interacting with …. what? Exploring children’s social and sensory practices in a science discovery centre. Education and Ethnography. forthcoming
  10. 10. environment: material modalities of ‘discovery’ in two learning spaces Exhibits Hall Science Theatre all the modes employed contribute to meaning…
  11. 11. methodology • social interaction – exhibits as ‘sites of engagement’: for interactional talk, accent (prosody), gesture, facial expression, gaze- direction, body posture, position & movement – but also dress, hairstyle, body decoration • communicative environments – interior layouts, wall displays, instructional texts, exhibits (colour, texture, shape, position, opaqueness/light, weight, movement, sound, etc), – also related texts & discourses (marketing, discourses of science, ‘hands-on learning’ etc) • senses – think about how e.g. water feels on skin; listen to how senses are verbalised, registered in face/body
  12. 12. findings • Centre promoted as a place for ‘learning through doing’ • But children primarily engaged in – a) sensory pleasure-taking and – b) playing out shifting relations of affect and sociality. • discovery centres are highly social spaces: such that twin dimensions of the social and the sensory govern children’s responses to exhibits, eclipsing thoughts of science • live demonstrations (science theatre) more controlled and school-like, more sensorially restricted: seemed to serve as a more effective channel of scientific communication • old-style live demonstrations are more effective than technologically interactive exhibits in foregrounding scientific concepts