Keynote on the cognitive demand of the multimodal media object of videogames. Presented at "DigiLitEY Training School 2016", in the Institute of Education, University of Minho. 7 june 2016.

1. Videogames and Multimodal Literacy
Nelson Zagalo, University of Minho
http://nelsonzagalo.googlepages.com
http://virtual-illusion.blogspot.com
Institute of Education, University of Minho 7 June 2016
DigiLitEY Training School 2016

2. …been doing
Authored and edited books on:
Storytelling, Aesthetic Emotions, Virtual Worlds, Videogames and Creativity

3. Research in the domain of Human–Computer Interaction, within a Communication
Cognitive approach.
In specific, Interaction Design, and in concrete, Engagement.
…domains

4. a) Syntax
Code, symbolic objects of expression.
b) Semantic
Meaning, interpretation of the symbols and its interrelations.
c) Pragmatic
Context, understanding the relation between participants.
Multimodal literacy: communication
For the multimodal literacy purpose, problems arise when
different objects of expression (modes) are used to support
the syntax, affecting semantics but mainly pragmatics.
Levels of the communication process (Watzlawick, 1967).

5. Communication can occur between humans, between humans and
machines, between humans mediated by technologies, and even between
humans mediated by artifacts (like videogames).
Defining pragmatics

6. The human communication occurs through body and can be
externalized to objects of expression (media), which add new modes to
human expression, each making use of different codes.
Modes
verbal (voice expression),
non verbal (body language),
sound (object produced),
picture (drawing, photo…),
text (visual symbols),
animation (moving pictures),
audiovisual (moving pictures with sound),
interaction (action simulation).
Syntax
Each mode develops its own conventions, languages, creating what we
call Media Arts (painting, sculpture, literature…), which we use to
express ourselves and communicate.
Multimodal literacy: modes

7. Analogue
Sculpture
Painting
Literature
Music
Theatre
Dance
Fashion
Industrial
Cinema
Photography
Animation
Television
Video
Comics
Graphic design
Digital
Videogame
Virtual Reality
3D Animation
Generative Music
Physic
Computation
Programming
Interaction Design
Multimodal literacy: the media arts
Integration of modes

8. Multimodal arts: theater, film, and videogames
Videogames are an integration of previous media arts, like film,
and even theater.
Videogame: Cinema + Computer Processing
new expression: Interactivity (user’s performance)
Cinema: Theater + Photography
new expression: Montage (sequence of photos)
Theater: Sculpture + Painting + Literature + Music + Dance + Fashion
new expression: Drama (actor’s performance)

9. Multimodal path: from text to videogame
Through literature the
main goal of the writer
is to tell the narrative
as detailed as
possible in order to
develop a strong
“fabula”, or mental
story, in the receptor’s
mind.
Film doesn’t need to spend time telling details
because they are shown. The story world is
given already built to the receptor, purposing
direct perception of the visual world that
enhances sensorial emotion and learning.
Coppola, The Godfather, 1972Puzo, The Godfather, 1969

10. Story is no more an act of “telling”, neither “showing” but an integrated set
of active participation, of “doing”. Interactive media open a new space
(virtual) for the mediation of knowledge. The story containing learning
messages will evolve only through the actions and decisions of the
participator.
Videogames go beyond representation to become simulation.
EA, The Godfather: The Game, 2006
Do Something Do Something
Multimodal path: from text to videogame

11. Representation is made of conventions of symbols, different from reality,
involve strong need of interpretation. More demanding in abstract
reasoning, cultural background, and imagination activation.
Simulations are illusions of the real, purposing a sort of direct access to the
real without its insecurities. Permits learning through infinite repetition, with
little change, modelling for a fixed mental model.
Representation vs. Simulation

12. Acting upon the real, upon objects, simulations, makes learning become a
task memorized through somatic actions, using body perception instead of
mere mental activity, configuring an optimized situated cognition
environment.
One of the greatest examples of the gains of simulations have been flight
simulators, which were able to lower drastically accidents.
The simulation gain

13. In the 1960’s, Seymour Pappert (MIT) following his studies with children,
purposed a sort of visual programming language, LOGO, which he defined
as "body-syntonic reasoning”, because students could write code putting
themselves in the place of the turtle, mentally simulating, doing the
reasoning and predicting based in the turtle’s position and movement.
The computing evolution: becoming visual

14. Then in the 1970’s, Alan Kay (Xerox) purposed moving from text-based to
image-based interfaces. The idea arose from Pappert studies, the
evaluation done for the interaction with objects and written words,
understanding that people would be more able to act upon objects than on
written words.
The computing evolution: becoming visual

15. In the 2000’s, Mitchel Resnick (MIT) developed a real visual language for
kids, Scratch, which in 2016 was brought into the physical world by Osmo.
The written symbols became visual elements, and finally physical objects.
Programming was born in abstract modelling but is now made of concrete
objects, sort of “programming simulator”.
The computing evolution: becoming physical

16. The computing evolution: in the future
“Holodeck”, machine presented in 1998 in the Star Trek
series, previously dreamed by André Bazin in the paper “The
Myth of Total Cinema” (1948).

17. Multimodal expression evolution
Symbolic, abstraction (text)
Audiovisual, realism (image/sound)
Simulation, surrogate (interactive)

18. Why multimodal literacy evolved for
simulation?
Transforming the abstract topics into realistic
objects, and adding to them a fully interactive layer,
to guarantee a virtual learning by doing.

19. What can you “do” in videogames: 3 levels
1 - Manipulation
Act upon the representation, change and
transform, express yourself trough virtual
objects.
3 - Participation
Take decisions, choose, decide, experiment
with moral values, and check your real beliefs.
2 - Navigation
Follow conventions or define your own
approach to the virtual world. You’re what you
do in the world.

20. Doing to Learn
Brenda Romero, Gaming for understanding, at TEDxPhoenix (2011)
Learnign about Middle Passage at schools

21. Families in Africa Boat to take people to the US
New World , board game improvised
by Brenda Romero
Doing to Learn
Rules
. Takes 10 turns to cross the ocean.
. Can not take complete color sets of
pawns.
. We have 30 units of food.
. Each turn roll the dice.
. Dices determine the food units to be
consumed.

22. Motivation
CompetenceAutonomy Relatedness
Motivation model from the “Self-determination theory” (Deci e Ryan, 1985).
What you do and why?

23. Choice / Power
Autonomy means, to empower the receptor to the limit of experimenting
with moral values, testing beliefs, trying in “practice” your own theories.
To Do: autonomy
When your friend Chloe asks you to put an end to her life,
you need to decide, and choose if you let her die or not.
“Life is Strange” (2015)

24. Skills / Progress
Competence means, to develop the skills requested by the simulation
to keep progressing. Progress is the main fuel of interest.
No text manuals to read, you’re taken by the hand, each action done,
you go up another step in comprehending what’s expected from
you. A proper scaffolding process.
To Do: competence

25. Others / Recognition
Relatedness means, to be part of a community, be as capable as the
others, which can be easily translated by Leaderboards, or school
friends recognition.
To Do: relatedness

26. Videogames are interactive audiovisual
experiences, everything we take into account in
movies must be taken into account here, but the
center of the cognitive effort, the real challenge for
the players, goes beyond all the interpretation you
need to understand the audiovisual medium, you
need to become part of the experience, and in order
to fully understand it you need to perform, to do.
The videogame’s core of the multimodal literacy

27. Videogames and Multimodal Literacy
Nelson Zagalo, Universidade do Minho
http://nelsonzagalo.googlepages.com
http://virtual-illusion.blogspot.com
nzagalo@ics.uminho.pt