This document provides an overview of different media and modes that can be used for multimodal interactional analysis, including sound, video, photography, and field notes. It discusses analyzing these different media sources to understand how technologies influence relationships and communication. Key things to look for include reciprocity, rituals, defining physical space, and divided consciousness. The document also provides guidance on developing multimodal themes by identifying significant moments across media sources and relating them to literature on multimodal communication. Standard annotation conventions are suggested for transcription.

1. Multimodal Interactional Analysis

2. Capturing Different Media
1. Sound
2. Video
3. Photo
4. Observation (Field Notes)

3. Media Modes
Sound Speech, music, laughter, background noise
Video Lighting Ambient lighting in
homes, daylight
Physical presence Posture, facial and bodily
gestures, eye contact,
pointing
Production elements On screen effects,
framing of shots, cutting
between scenes, etc.
Physical objects Placing objects in field of
view, gesturing with
objects,
Photography Similar to video but without the movement/time
aspects
Observation Spoken language, physical behaviours, social
(field notes) dynamics, printed text/signs, etc.

4. Multimodal Activity
Observe and record examples of how technologies
influence the way that people relate to one another
Use one of the following media:
o Sound Recording
o Observation (Field Notes)
l Still Photos
l Video Recording

5. Things to look out for?
 Reciprocity in communication
 coordination of turn taking in conversation
 ‘gift’ exchange
 Ritualised behaviour
 openings and closings
 Defining the ‘meaning’ of physical space
 What behaviours are allowed/expected/norms
 Reinforcement of these norms
 Divided Consciousness/Absent Presence
 Dealing with immediate and distant social contact
simultaneously

6. Feedback Session
 What modes have been captured?
 What different stories do they tell?
 Were there any issues with collecting/using particular
media?

7. Developing Multimodal Themes
Watch the videos/listen to recordings/view images
/ Identify themes
◦ Observe and identify significant moments that highlight multimodal
interaction as an aspect of communication – note the time
◦ Refer to existing literature on multimodal aspects of communication to
check what you see and hear – for example Goffman (1959)
3. Watch the videos
3 Describe the multimodal aspects of the videos
t Roughly transcribe the video data to clarify significant moments in
communication and identify commonalities across videos
d Watch the videos
e Final transcription in a suitable form to explain themes and
subthemes
◦ Include standard annotation conventions

8. Multimodal transcription of video data
(using my own coding scheme)

9. Multimodal Transcription
www.sigridnorris.com

10. Multimodal transcription – from
conversational analysis:
 (n) pause noted in seconds
 = joining of words
 : an extension of the preceding syllable;
 ; rising or falling intonation for subsequent utterance
 underlined text indicates spoken with particular vocal
energy
 - a sudden cut off to an utterance;
 ? a rising tone
 . a falling tone
 , a shifting continuous tone bridging utterances.

11. Multimodal Description

13. References
Clark, H. H. (1992). Arenas of language use. Chicago: The
University of Chicago Press.
Clark, H. H., & Schaefer, E. F. (1989). Contributing to discourse.
Cognitive Science, 13, 259–294
Kendon, A., Harris, R. M., & Ritchie Key, M. (1975). Organization
of behavior in faceto-face interaction. Paris: Mouton Publishers.
Kendon, A. (1990). Conducting interaction: Patterns of behavior in
focused encounters.Cambridge: Cambridge University Press.
Goffman, E. (1963). Behavior in public places: Notes on the social
organization of gatherings. New York: The Free Press.
Ten Have, P. (1999). Doing conversation analysis. A practical guide.
London: Sage Publications.