- This document discusses multimodal social semiotics and its applications for analyzing communication and learning with digital technology. - Multimodal social semiotics examines how people use different modes like words, images, sounds, gestures to make meaning, and how available semiotic resources shape potential meanings. - Interaction analysis using video data and transcription can study how learners orchestrate modes and embodied interactions with technology. - Case studies of learner-generated content and full-body interactive environments can explore what learners express through their selection and combination of available multimodal resources.

1. Multimodal Methods for Analyzing
Communication and Learning with Digital
Technology

2. Summary
● Multimodal Social Semiotics & Meaning Making
● Multimodal Social Semiotics: some definitions
● Multimodal Social Semiotics & Research Methods : Interaction analysis
● Multimodal Social Semiotics & Research Methods : video analysis & transcripts
● Applications:
○ Learner / User generated content
○ Embodiment & Technology
■ Embodiment & Technology: a case study

3. Social Semiotics
Semiotics:
● Study of signs and signification processes
● Interpretative semiotics (double formation of meaning / Semiosis:
processes of signification and interpretation (Eco))
Differences between traditional semiotics and social semiotics
● Arbitrariness (signifier and signified):
- traditional: arbitrary rules that can not be modified (De Saussure)
- social: the "codes" of language and communication are formed by social
processes (= social dimension of meaning) :
→ rules made by people that can be modified (through institutional or
symbolic power)
→ motivated sign: instead of fixed and unchanging "codes", signs are
considered to be resources that people use and adapt to make meaning
accordingly to the interests of the producer/interpreter and the
characteristics of the objects

4. MEANING
Multimodality
Multimodality & Social Semiotics:
People use differents modes to make meaning.
Going being linguistics approach → framework
for the analysis of visual, aural, embodied and
spatial aspects of the interaction and the
environment
● Each mode has different affordances that
are the product of the social and historical
use. The form should be particularly apt for
conveying a certain meaning
● People orchestrate meaning through their
selection and configuration of modes
meaning
potential of
an artifact
social &
cultural
environment
resources,
interests and
knowledge
of the person

5. Multimodal Social Semiotics: some
definitions
Mode: a set of resources for making meaning
and organizing principles, a channel of
communication
Semiotic resources (Halliday) : a resource for
making meanings → action, material and
artefacts that we use to communicate (difference
with sign). They have semiotic potential and
semiotic affordances. Ej.: written text, image,
walking

6. Multimodal Social Semiotics: some
definitions
Semiotic potential: each semiotic resource has semiotic
potentials (rather than a specific meaning) that are
culturally and historically shaped
Affordance: material, social and cultural (not just
perception!)
Semiotic affordance: each semiotic resource has
semiotic potentials that are new meanings that have not
been recognized yet and are latent in it
Modal affordance: potentialities and constraints of
different modes; what is possible to express and
communicate easily with the resources of a mode
and what is less straightforward or even
impossible. They are historically and socio-culturally
shaped.

7. Multimodal Social Semiotics: some
definitions
Motivated sign: agency of the sign maker, draws the
attention on the interests and intentions that motivate a
person’s choice of a given semiotic resources
multimodal ensemble: bring together different mode
and look at how different mode bring different meaning
simultaneity: all the semiotic resources mutually
elaborate each other. Each one by itself is incomplete.
together they create a whole that is bigger than its part
coherence & cohesion: how different modes work
together to produce coherence and make a complete
“text” (are the used modes coherents between them or
they contradict?) / cohesion: devices to construct
coherence
Van Leeuwen, Theo. Introducing social semiotics. Psychology Press, 2005.
Jewitt, C (2012) Multimodal methods for researching digital technologies. In: The Sage Handbook of Digital Technology Research. Sage, London.

8. Application in our research
● Design: theoretical perspective

9. Design: theoretical perspective
Full Body Interaction Environment as Multimodal
Ensemble
● what modes are present?
- which are their semiotic resources? semiotic
potentials and modal affordances? what are the
potentials and constraints of Full Body Interaction?
What is possible to communicate and express easily
with this mode? What is less straightforward or even
impossible? (inventories, socio-cultural references)
● How the different modes relate together?
Multimodal
Ensemble
Semiotic
Resources
Semiotic
potentials
Modal
affordances

10. Multimodal Social Semiotics & Research
Methods

11. Applications: Interaction analysis
HISTORY:
‘30/’60: Interaction Sociology (Goffmann,
Garfinkel) → bottom- up approach to study
society
‘60: Conversation Analysis
‘70: Conversation Analysis & Embodiment
(Goodwin)
‘90: Workplaces studies (Goodwin, Mondada)
- role of body and action in achieving
collaboration
- embodiment in conversations
CONCEPTS:
- Adjacency pair (invitation to the other to respond) & paired
action (adjacency but through action)
- Reading other bodies (Hindmarsh and Pilnick, 2002): people
read each other bodies; you see somebody doing something and
if you know the context you will know what to do next and what is
expected from you (ej. study in anesthesia: they don’t speak so
much but they coordinate each other through action & body)

12. Multimodal Social Semiotics & Research Methods :
video analysis & transcripts
From video to data:
- at the end of the day do a little
description of each video to
describe what is going on
- vignette, summary
- coding categories
- multimodal transcription
- visualization
Multimodal transcription :
- Define a methodological
framework
- Define the purpose and focus
of the transcript (ej. timescale,
modes)
- Design the transcript
Multimodal Transcription: the quality of transcript is not how much it is a
representation of the reality but the point to which it allows research your topic
- the modes selected for transcription depend on the focus of the research

13. Multimodal Social Semiotics & Research Methods :
video analysis & transcripts
Bezemer, J. (2012). How to transcribe multimodal interaction? In Texts, Images and Interaction: A Reader in Multimodality. Mouton de Gruyter.
Mavers, D. (2012). Transcribing video. NCRM working paper.

14. Applications: Learner / User generated content
→ Motivated Sign / Form as meaning
● What kind of semiotic resources do
they use to communicate something?
Which cultural references it relate to?
● What are the resources available?
Which one are selected and how they
are re-shape? (which other selection
could have been possible)
● How the different resources are
orchestrated between them?
● Is it coherent? How this coherence have
been constructed?
● What she wants to express with it?

15. Applications: Embodiment & Technology
multimodality / ethnography / psychology
DESIGN
● understand the potential that different modes can bring to interaction
(modal affordance; which mode can be apt for this purpose?)
● possibilities of identifying novel semiotic resources
● ...
ANALYSIS
● emphasis on situated meaning making: how people make meaning
in a specific situation
● analyze which modes are used in the environment and which
resources are available
● examine and compare the effect of using different modes in learning
context
● ...

16. Embodiment & Technology: case study
goal:
METHOD:
1) Logging Video Data: descriptive account (time, content, events, modal
configuration): vignette, sketches, screen-capture, maps..
2) Raw analysis: watch the video in different way (sound, slow/fast motion) →
start to generate criteria for sampling
3) Sampling: select episodes / phenomena / etc. // category definition
4) Transcription
5) Modal analysis:
- break down the modes (ej: position, gaze, hand-action)
- look for intermodal relation (ej: pace)
6) Incorporate the context into the analysis
Light Table

17. Application in our research

18. Application in our research
● Design: participatory design
● Analysis

19. Design: participatory design - children
production
● What kind of semiotic resources do they use to
communicate something? Which cultural references it
relate to?
● What are the resources available? Which one are
selected and how they are re-shape? (which other
selection could have been possible)
● How the different resources are orchestrated between
them?
● What she wants to express with it?
Semiotic
resources
Semiotic
potentials
Motivated Sign

20. Analysis
- Methodological framework
● situated meaning making
● which modes are used in the environment and
which resources are available
● examine and compare the effect of using different
modes
- Go beyond discourse analysis
- Sense-making through the body and the
space