Creativity & multimodality: an analytical framework for creativity in multimodal texts © Locky LAW 2017. Please do not copy or quote without permission.

1. Creativity & multimodality: an analytical
framework for creativity in multimodal texts
1
Locky, LawLocky Law
PhD candidate
The Hong Kong Polytechnic University
Lx3h@yahoo.com
JPSS Symposium 2017
7-8th April, 2016 (Fri-Sat)

2. Content
 Introduction
 Creativity, Cognition and Language
 Analytical framework for creativity in multimodal
texts
 Endo-referenced & Exo-referenced
 Explicit/Known & Implicit/Assumed
 Sample Analysis
 Gorilla Koko and Sign Language
 MTV and Song
 Graphics and Digital Arts
 Key References 2

3. Creativity and
Cognition
Boden (2009, pp. 24-25) defines the three types of creativity as follows:
 “Combinational creativity produces unfamiliar combinations of familiar
ideas, and it works by making associations between ideas that were
previously only indirectly linked…
 Exploratory creativity rests on some culturally accepted style of thinking,
or “conceptual space”... The space is defined (and constrained) by a set
of generative rules. Usually, these rules are largely, or even wholly,
implicit…the person moves through the space, exploring it to find out
what’s there…
 In transformational creativity, the space or style itself is transformed by
altering (or dropping) one or more of its defining dimensions. As a result,
ideas can now be generated that simply could not have been generated
before the change.”
 Boden (2009, p. 25) admits that “there is no clear-cut distinction between
exploratory and transformational creativity”.
3

4. Creativity and
Language
not dissimilar from Carter (2004)’s linguistic classification of creativity.
 pattern-forming creativity: “creativity via conformity to language
rules rather than breaking them, creating convergence, symmetry and
greater mutuality between interlocutors”, which is practically the
linguistic counterpart of combinational creativity,
 pattern-reforming creativity: “creativity by displacement of fixedness,
reforming and reshaping patterns of language” (Vo & Carter, 2010, p.
303),
4

5. Cognition
Language
Creativity
5
Creativity
Cognition Language
Boden (1994,1998, 2004, 2009)
Halliday and Matthiessen (1999 [2006])
Carter (2004); Vo & Carter (2010)

6. Inspired by the concept of ‘given’ and ‘new’ from Halliday’s (1967)
information status, i.e. information recoverability from preceding discourse.
 Endo-referenced creativity is one that is made reference to a source
which is recoverable within the preceding or same ‘text’,
 Exo-referenced creativity is one that is made reference to a source
which is unrecoverable within the preceding or same ‘text’, which means
that such reference is made to an external source.
6
Analytical Framework for
Creativity in Multimodal Texts
Endo-referenced & Exo-referenced

7.  When creativity is created by explicitly showing the formula for creation,
then the instructions of creativity construction is ‘known’;
 otherwise, when creativity is created without explicitly showing the
formula of creation, then the instruction of creativity construction is
‘assumed’
 because it is assumed by the creator that the target of an instance of
creativity has the level of competence to comprehend or decipher the formula
of creativity creation without explicitly showing the steps involved in the
creation process.
7
Explicit/Known & Implicit/Assumed

8. 8
Types of
creativity
Formula of
construction
Reference style
Exo-referenced Endo-referenced
Pattern-
forming
Implicit
Direct use / quoting of
external resources such as famous
lines, quotes, speeches, quotes,
sayings, idioms, metaphor, song
lyrics, classic paintings, movie
scenes and dialogues without
explicit citation of the source and
explicitly showing the formula of
repetition (Assumed).
Repeating / playing along with
existing resource / someone’s
creation to the user or witnesses of
such use of it without explicitly
showing the formula of repetition
(Assumed).
Explicit
Direct use / quoting of
external resources such as famous
lines, quotes, speeches, quotes,
sayings, idioms, metaphor, song
lyrics, classic paintings, movie
scenes and dialogues by explicit
citation of the source by explicitly
showing the formula for repetition
(Known).
Repeating / playing along with
existing resource / someone’s
creation to the user or witnesses of
such use of it by explicitly showing
the formula of repetition (Known).
Pattern-
reforming
Implicit
Direct creation of New /
neologism without explicit
citation / indication of the source
and explicitly showing the formula
for creation (Assumed).
Direct creation of New /
neologism using existing resources
without explicitly showing the
formula for creation (Assumed)
Explicit
Creation of New / neologism
by explicit citation / indication of
the source and by explicitly
showing the formula for creation
(Known).
Creation of New / neologism
using existing resources and by
explicitly showing the formula for
creation (Known).

9.  A Conversation With Koko (1999)
9
Sample Analysis
Gorilla Koko and Sign Language

10.  humans create or construct portmanteau neologisms in
English in which two or more words are blended
together using parts of each word,
 Koko compounds signs in order to invent new words to
represent meanings which are absent from her
vocabulary.
10

11. 11
Endo-referenced
(from existing lexicon)

12. MTV and Song
12

13. [Intro]
P-P-A-P
[Verse 1]
I have a pen, I have an apple
Uh! Apple pen!
[Verse 2]
I have a pen, I have pineapple
Uh! Pineapple pen!
[Verse 3]
Apple pen, Pineapple pen
Uh! Pen-Pineapple-Apple-Pen
Pen-Pineapple-Apple-Pen
13
Endo-referenced
Endo-referenced
Endo-referenced

14. 14
Is this ‘creative’?
Graphics and Digital Arts

15. 15
Is this ‘creative’?

16. Example
Symbol
type
Symbol
Creativity
Type
Formula of
construction
Reference
style
Source -- -- --
Unit
pattern-
reforming
Implicit
exo-
referenced
Logo
pattern-
forming
Implicit
endo-
referenced
16

17. Conclusion
 It is hoped that this paper has provided a new perspective for
researchers of linguistics in the analysis of multimodality and has
assisted studies in realization of meanings in modes and in
intersemiotics between modes.
 For future research, it will be exciting to see how this proposed
analytical framework for creativity in multimodal texts can be
used to explain creativity in other forms of the arts, such as
culinary arts, photography, cinematography, music, sculpting,
classical paintings and social media “re-creativity”.
17

18. 18

19. Key References
 Boden, M. A. (1998). Creativity and artificial intelligence. Artificial Intelligence,
103, 347-356.
 Boden, M. A. (2004). The Creative Mind: Myths and Mechanisms (2nd ed.).
London: Routledge.
 Boden, M. A. (2009). Computer Models of Creativity. AI Magazine, 30(3), 23-
34.
 Carter, R. (1999). Common language: corpus, creativity and cognition.
Language and Literature, 8(3), 195-216.
 Carter, R. (2004). Language and Creativity: The Art of Common Talk. London:
Routledge.
 Gardner, H. (2008). Art, mind, and brain: A cognitive approach to creativity.
Madison, Wisconsin: Basic Books.
 Halliday, M. (1967). Notes on Transitivity and Theme in English: Part 2.
Journal of Linguistics, 3(2), 199-244.
 Halliday, M. (1993). Towards a Language-Based Theory of Learning.
Linguistics and Education, 5(2), 93-116.
 Halliday, M., & Matthiessen, C. M. (1999 [2006]). Construing Experience
Through Meaning: A Language-based Approach to Cognition. London, New
York: Continuum.
19

20. ComputationaL
creativity
 Elgammal and Saleh (2015) unintentionally merges Boden
and Carter.
 Use computer algorithm to rank 1,710 images of art work
from 1412-1996 according to their quantifiable creativity
score.
 H-creativity in paintings along the historical timeline
 “to be creative it is not enough to be novel, it has to be
influential as well (some others have to imitate it)”
(Elgammal & Saleh, 2015, p. 41) in line with Carter’s
hypothesis of creativity in common talk
[See
http://www.huffingtonpost.com/2015/07/06/creativity-
algorithm_n_7700002.html]
20