A presentation on Multimodal Discourse Analysis

1. Multimodal Discourse
analysis
BY
ZOOBIA ABBAS

2. Definition
Multimodal discourse analysis considers how texts draw on modes of communication such as
pictures, film, video, images and sound in combination with words to make meaning. It has
examined print genres as well as genres such as web pages, film and television programmes. It
considers how multimodal texts are designed and how semiotic tools such as colour, framing,
focus and positioning of elements contribute to the making of meaning in these texts.

3. Origin
Much of the work in multimodal discourse analysis draws from Halliday’s ( 1978 , 1989a ) social
semiotic approach to language, a view that considers language as one among a number of
semiotic resources (such as gesture, images and music) that people use to communicate, or
make meaning, with each other. Language, in this view, cannot be considered in isolation from
meaning but needs to be considered within the sociocultural context in which it occurs.
Multimodal discourse analysis, thus, aims to describe the socially situated semiotic resources
that we draw on for communication.

4. 3- types of social meanings
Halliday (2009a) describes three types of social meanings, or functions that are drawn on
simultaneously in the use of language. These are:
1. ideational (what the text is about),
2. Multimodal interpersonal (relations between participants)
3. and textual meanings (how the message is organized).
In multimodal texts these meanings are realized visually in how the image conveys aspects of
the real world (the ideational, or representational meaning of the image), how the images
engage with the viewer (the interpersonal, or modal meaning of the image) and how the
elements in an image are arranged to archive its intention or effect (the textual, or
compositional meaning of the image) (de Silva Joyce and Gaudin 2007)

5. Exemplary works
Examples of multimodal discourse analysis that are influenced by this view include:
Kress and van Leeuwen’s ( 2006 ) Reading images: The grammar of visual design , Kress’ ( 2010 )
Multimodality: A social semiotic approach to contemporary communication ,
van Leeuwen’s ( 2005a ) Introducing social semiotics
and O’Halloran’s ( 2004a ) Multimodal discourse analysis and Painter,
Martin and Unsworth’s ( 2012 ) Reading visual narratives .
Machin and van Leeuwen ( 2007 ) discuss multimodality in relation to global media discourse,
Bednarek and Martin ( 2010 ) discuss systemic functional perspectives on multimodality and
Bednarek and Caple ( 2012 ) examine multimodality and news discourse.

6. 4 theoretical assumptions in multimodal
discourse analysis
Jewitt ( 2009a ) describes four theoretical assumptions that underlie multimodal discourse analysis
1. The first is that language is part of an ensemble of modes, each of which has equal potential to
contribute to meaning. Images, gaze and posture, thus, do not just support meaning, they each
contribute to meaning.
2. The second is that each mode of communication realizes different meanings and that looking at
language as the principal (or sole) medium of communication only reveals a partial view of what is
being communicated.
3. The third assumption is that people select from and configure these various modes in order to make
meaning and that the interaction between these modes and the distribution of meanings between
them are part of the production of meaning.
4. The fourth assumption is that meanings that are made by the use of multimodal resources are, like
language, social. These meanings, further, are shaped by the norms, rules and social conventions for
the genre that are current at the particular time, in the particular context.

7. Multimodal approach (explanation)
When people communicate with each other, thus, it is seldom done by one means of
communication alone, that is, language. They most typically draw on a number of modes
simultaneously, such as images, gesture, gaze and posture – as well as language. Each of these
modes has particular affordances (Gibson 1977 ) within the particular context; that is, what it is
possible to represent and express through a particular mode (Kress 1993 ). This meaning
potential (van Leeuwen ( 2005a )) of the particular mode is ‘shaped by how mode has been
used, what it has been repeatedly used to mean and do, and the social conventions that informs
its use in context’ .

8. Images
Images have been given special attention in much of the work in multimodal discourse analysis.
These images may include photographs, diagrams, maps or cartoons. Images, can be seen as
having a grammar of their own.
For example, the social relationship between an image and its viewer is strongly influenced by
whether the subject in the image establishes eye contact with the viewer or does not.
Each of these possibilities could be seen as an example of mood , where eye contact could
perhaps suggests a demand, whereas no eye contact might suggest an offer

9. Point of view in an image
The point of view , or perspective, of the image is also relevant. For example, a horizontal image
suggests involvement as the viewer is on the same level as the subject of the image. A high
angle shot might suggest superiority and a low angle shot may suggest respect. Other meanings
are conveyed through the distance of a shot (close vs. medium vs. long), the lighting, colour and
focus of the shot and the extent to which the image in the shot aims to reflect reality, or not

10. Kress’ Concept
Kress ( 2010 ) in Multimodality: A social semiotic approach to contemporary communication
provides a social-semiotic theory of multimodality.
Among other things, a social semiotic theory of multimodality asks:
• What meaning is being made in a text?
• How is meaning being made in the text?
• What resources have been drawn on to make the meaning in the text?
• In what social environment is the meaning being made?
• Whose interest and agency is at work in the making of the meaning?
It does this by considering modes such as writing, images, colour and facial expressions and the relation
between them. It considers which mode is foregrounded, which mode carries major informational weight
and which mode has what function in the text

11. Genre, speech acts and multimodality
Van Leeuwen ( 2005a , 2005b ) discusses speech acts and genre in relation to multimodality,
using these two notions to capture the ‘how’ (vs. the ‘what’) of multimodal communication.
A key point he draws from speech act theory is how a speech act is both an illocutionary act
(what the speech act is aiming to do) and a perlocutionary act (the effect it has on the thoughts
and actions of people).
An advertisement, thus, may aim to persuade a person to buy a particular product (the
illocutionary act). If the person is convinced by the advertisement and buys the product, this is
the effect, or perlocutionary force , of the advertisement.
Texts, thus, draw on a range of modalities to create a perlocutionary effect. It is not just the
words of the advertisement that persuade a person to buy the product. It is through the use of
linguistic and visual resources in combination with other non-linguistic and contextual factors
that this occurs

12. Multimodality and global media
discourse
Machin and van Leeuwen ( 2007 ) discuss genre and multimodality in relation to global media discourse. Taking
the example of advertisements in Cosmopolitan magazine, they show how their representations aim to create a
certain perlocutionary effect on readers; that is, to use or purchase a particular service, or product.
◦ Using the example of an advertisement for a Health Diet Clinic, they show how the advertisement employs a
‘problemsolution’ rhetorical structure to do this. The problem presented in the advertisement is skin problems. The
solutions are in the Diet Clinic’s guidance and advice and the purchasing of a particular product, the Diet Clinic’s Skin
Care System. The problems are expressed by use of strategies such as an image of blemished skin with accompanying
texts such as ‘Do you suffer from skin problems?’ The solution is a picture of the product for sale. The ‘result’ is a
picture of unblemished skin. Both words and images, thus, interact in the advertisements to construct their
particular message.

13. A genre and multimodality framework
Bateman ( 2008 ) discusses genre in relation to written genres saying that nowadays:
◦ “Text is just one strand in a complex presentational form that seamlessly incorporates visual aspects ‘around’, and sometimes even
instead of, the text itself. “
Genre plays a central role to account for the range of possibilities for how multimodal texts are realized and the ways in
which we can interact with them. He proposes a genre and multimodality framework that provides several layers of
description for multimodal texts. These are the content structure , the genre structure , the rhetorical structure , the
linguistic structure , the layout structure and the navigation structure of the text. Each of these operates within the
constraints of the physical nature of the text being produced (e.g. paper or screen size), constraints arising from production
technologies (e.g. page limits, colour, size of graphics, deadlines) and consumption constraints (e.g. time, place, and
manner of obtaining and consuming the document, the ease with which the text can be read).

14. Multimodality and newspaper genres
Newspapers often draw on the strength of visual images in image-nuclear news stories to capture and retain the
interest of their readers (Caple 2009 ). In the majority of the texts Caple collected she found the headings that
accompanied the images were an idiomatic expression that fitted with the subject matter of the photograph.
For example, in an image nuclear news story about drought in China there was an image of dry, cracked earth which
had central prominence in the photo with a row of people in the background carrying work tools
The newspaper is assuming knowledge on the part of the reader of the cultural allusions of the idioms, which in
turn, enables the newspaper to express cultural and social solidarity with the readers’ . The three components of
this text are the Heading which appears above the text, the Image which has central place and the Caption which
appears in a smaller font beside the image.

15. Film and television genre
Iedema (2001 ) provides a framework for the analysis of films and television by drawing on work
in film theory and genre theory. The levels of analysis he proposes are frame , shot , scene ,
sequence (from film theory), generic stage and work as a whole (from genre theory). These
levels are summarized in Table 8.2 .

16. Multimodal in films and television genres

17. summary
This presentation provided an overview of multimodal discourse analysis, an approach
which considers how texts that employ more than one mode of presentation, such as words
and graphics, make meaning. It has provided the theoretical background to multimodal
discourse analysis as well as given examples of analyses that have been carried out from this
perspective. It has also discussed relations between genre, speech acts and multimodality.
A genre and multimodality framework has then been presented as a proposal for examining
layers of meaning in multimodal texts.