This document discusses embedding multimodality and digital literacies in ESP courses. It begins by outlining the emerging needs of ESP learners in a digitally mediated world. It then defines digital multimodal composing as going beyond traditional writing to include modes made available by digital media like videos and infographics. A process-oriented genre approach is proposed where authentic genres and practices are identified, the role of multimodality is examined, and projects are designed. An example course embeds a digital video documentary project alongside a written scientific report to develop students' digital literacies.

1. Multimodality and digital literacies in
ESP courses
Christoph A. Hafner
Department of English, City University of Hong Kong
Plenary Presentation for Taiwan ESP Association
Multimodality Workshop, November 3-4, 2018

4. www.lsppc.org

5. Overview
1. Emerging needs of ESP learners
2. Digital multimodal composing
3. A process-oriented, genre-based approach to
digital literacies in ESP courses

6. What do you mean by English for Specific Purposes?

7. Emerging needs of ESP learners
The role of digital media in contemporary
communication

8. Affordances
of digital
tools
Ways of
doing
Ways of
relating
Ways of
meaning
Ways of
being
Ways of
thinking

9. Scientists urgently need to be able to
speak with clarity to funders, policy-
makers, students, the general public, and
even other scientists.
I’d like to try a playful experiment. Would
you be willing to have a go at writing your
own explanation of what a flame is – one
that an 11-year-old would find intelligible,
maybe even fun? The Centre for
Communicating Science is looking for new
ways to light up people’s minds with
science, and you might point the way.
We’ll try out the entries on real 11-year-
olds and see which work best. (Alda, 2012,
p. 1019)

11. Some other examples
• Crowdfunding proposals (Mehlenbacher, 2017)
– Hybrid genre, aimed at multiple audiences
• Academic blogs (Blanchard, 2011; Luzón, 2012, 2013)
– Writing for a diversified audience
– Interacting with and responding to that audience

12. Additional forms of expression
JoVE began in response to a
universally experienced problem
within the world of biomedical
science: today only 10-30% of
published scientific articles can be
successfully reproduced. As a result,
scientists and researchers around
the world end up spending months
and years of time, effort, and
funding simply trying to replicate
the findings of other labs instead of
advancing toward new discoveries.
(www.jove.com, emphasis in
original) Journal of visualized experiments

13. How a protocol is being executed is being captured visually.
So there is like no mistake on how it is being done,
especially if, say, if we’re doing micro-injection of DNA into
eggs, okay, how do you handle the pipette? How deep do
you penetrate it? So there’s lots of finer details which are
being sort of captured visually (…)
(Interview with biologist at City University of Hong Kong)

14. Another example
• The Article of the Future (Pérez-Llantada, 2013)
– ‘More dynamic’ and ‘user-friendly’ with ‘enhanced
elements’
– Did not affect ‘actual text-composing and reception
practices’

15. ‘The ability to produce and
understand text-visual
interrelations is now an
essential component of an
academic literacy.’ (Hyland
& Hamp-Lyons, 2002, p. 8)
‘In fact, it is now no longer
possible to understand
language and its uses
without understanding the
effect of all modes of
communication that are
copresent in any text.’
(Kress 2000, p. 337)

16. Digital multimodal composing

17. A definition
• Digital multimodal composing in ESP classes
means going beyond traditional writing forms
to include other modes made available by
digital media.
Digital videos Infographics Posters Brochures
Academic
text/image
interaction

18. Benefits of DMC
• Meets the real-world needs of learners in a digitally mediated world;
• Enhances state-of-the-art language teaching strategies like task-
based language teaching and learning, by allowing for the easy
integration of multiple skills;
• Engages students with authentic audiences, providing real-world
motivation;
• Is ‘voice-enhancing’ and can ‘embolden struggling writers to express
themselves’ (p. 82);
• Heightens genre awareness as a result of transformation processes
that occur in the composition process;
• Increases learner autonomy and encourages independent language
practice.
(Belcher, 2017)

19. The controversy
• Does DMC ‘distract’ learners from the ‘real’
work of language learning?

20. Distracting or motivating?
Candy: At the beginning of the video, we want to attract
concentration on the video. So introduction is –
we make a lot of effort on it.
Lindsay: Would you make the same effort if you were just
doing a stand-up presentation with PowerPoint?
Jane: Probably not.
Candy: No. Introduction is not that important maybe in
PowerPoint, maybe the information, maybe the –
in the middle of your presentation, is the main
point, so introduction is not that important. (Focus
group: Phase 2)

21. Implementing DMC
A project-based course in English for science

22. Background and context
• University course in English for science students
– Applied Biology
– Applied Chemistry
– Applied Physics
– Architectural Studies
– Computing Mathematics
– Environmental Science and Management
– Surveying

23. English for science project
A. Digital video project
B. Written scientific
report
English for
science
project
A1. Reading/data
collection
A2. Scripting/
storyboarding
A3. Performing/
recording
A4. Editing
A5. Sharing
B2. Writing
B1. Reading/
outlining
B3. Editing/
proofreading

24. Sample topics
Dim the lights
(2013)
Taste me if you can
(2009)
Vitamin C
(2012)

25. Technology and support

26. Addressing
multimodality by
analysing ‘models’
• BBC documentary
• Al-Jazeera documentary
• Student documentary
Rhetorical
strategies
Linguistic
analysis
Narrative
images
Conceptual
representations
Offer and
demand
Camera angle
Visual
coherence

30. Scaffolding the project
• Week 2, students to form groups for scientific
documentary;
• Week 4, data collection for scientific documentary
complete;
• Week 5, script complete;
• Week 7, final video uploaded and shared through
YouTube.
• Week 11, students to submit a draft scientific
report for individual feedback

31. Negotiating consent
In order to protect privacy in these online
environments, we ask that you follow these
terms of use for online learning on GE2401. In
particular, you agree that you will:
• Only use your first name in our writing;
• Only post pictures of people if you have
asked for their permission;
• Be responsible and sensible in our writing;
• Be positive and constructive in our
comments;
• Respect other users’ opinions and advice;
• Post only original works (for all text types
and files);
• Respect intellectual property; and
• Be responsive and open to opinions and
advice.

32. Dim the lights

33. Discoursal identities

34. Student perspectives on the task

35. The challenge of attention
The first impression of audience is the critical point
to determine the success of a good documentary
since if the audience’s attention cannot be attracted
at the beginning, they will have no interest to
continue to watch the video even the information is
rich and constructive. I agree with t01_john. He said
that visual stimulation would be the easiest way to
make them remember the video. [Student blog post,
October 7th, 2009 at 11:22 pm]

36. The challenge of multimodality
I think the most challenging thing is how to give
an attractive and interesting present[ation]
because we use lots of method involved in our
video. For example, use pictures, use a narrator,
stand in front of the camera for speaking and use
music or many, many elements we involved in
order to give a whole product to make it more
interesting. [Student focus group interview]

37. Thinking about multimodality
• Moving images and animation
• Charts and tables for scientific data
• Subtitles
• Different camera angles and lighting,
• Background music
• Sound effects
• Interesting locations
• Interesting participants
• Facial expression

38. 3 Cases and rhetorical devices
• Investigation of a startling fact
– Did you realize there is a blind spot in your eye?
• Investigation of a social issue
– Why does the cafeteria food taste so bad? Is it only
the taste, or is it the smell as well?
• Investigation of a personal issue
– Why can’t I taste this orange juice? Is there
something wrong with me?

39. Case #1
Investigation of a startling fact

40. Student identity
• Student as ‘scientist’
• Narrator role is backgrounded, increased social
distance between the producers and their
audience

41. Opening: Case 1, shot 3

43. Background and theory: Case 1, shot 10

45. Discussion: Case 1, shots 51-52

47. Case #2
Investigation of a social issue

48. Student identity
• Student as ‘reporter’
• Narrator role is more prominent, with an on-
screen narrator (the reporter), who appears at
various points throughout the documentary
• Watch for intertextual references and use of
sound

49. Opening: Case 2, shots 1-8

51. Case #3
Investigation of a personal issue

52. Student identity
• Student as ‘traveller’ on a ‘journey of
experiment’
• Narrator role is prominent, the narrator is a
part of the action as in a first person
documentary
• Watch for the range of visual information and
effective use of sound

53. Opening: Case 3, shots 55-65

55. Semiotic remediation

56. TRANSFORM
Video documentary
Scientific report
For a public audience
To educate and entertain
Student as...
• Scientist
• Reporter
• Traveller
• Secret Agent
For a specialist audience
To inform and persuade
Student as scientist

57. Key questions
• For each genre:
– What content is appropriate?
– How should it be organized?
– How should it be expressed through lexical,
grammatical and multimodal resources?
– How actively can I engage with the audience?
– How forcefully can I present my stance?

58. Establishing novelty

59. Establishing novelty
Do you know what the
problem of light pollution
is?
Pun and So (2011) present
the night-sky brightness
taken at 199 distinct
locations in Hong Kong
and state that the light
pollution in Hong Kong is
severe. However, they do
not further investigate the
problems of light pollution
and the relationship
between light pollution and
other parameters.

60. Methods

61. Methods
We have organized a group
of special agents to search
for the answer. … We have
selected four locations for
our study, including Sai
Kung, a countryside, Mong
Kok and Causeway Bay,
urban areas, and Sha Tin, a
new town.
Four locations
including Causeway
Bay, Mong Kok, Sha
Tin and Sai Kung,
were selected for this
study because they
show different degrees
of urbanization.

62. Presenting results

63. Presenting results
Narrator: Causeway Bay is also one
of the Hong Kong’s major shopping
districts. As there are many indoor
shopping campuses such as Times
Square and Sogo, many tourists will
live there…
Do tourists living in Causeway Bay
think that light pollution is a great
problem?
The light intensity in Causeway Bay
is 550 LUX.
Fig. 3 shows the average
scores given by residents to
assess the negative impact on
their daily life by light pollution
in the daytime.

64. Interpretations and claims

65. Interpretations and claims
Host1: Attractions should
be lightened up. Tourists
love light. They enjoy to
discover more about the
city.
Fig. 3 and 4 suggest
that in reality, greater
light pollution may not
be perceived as
leading to more
discomfort.

67. Collaboration

68. Digital tools used
• Communication tools
– Facebook groups
– WhatsApp messenger, Facebook messenger
– Email
• Collaboration tools
– Google Drive
– Dropbox

69. Use of language in CMC

70. Why choose English?
• As a convenient input language
• As a conscious choice, for practice
• When focusing on the academic task

71. English for practice
Well, maybe it is because Billy’s English is really
good [laughs] and then I know his English is good
and he will willing to understand what I mean and
reply me in English. So I use English…
(Gina, G11 interview)

72. Why choose Chinese?
• To offer a speedy exchange of views and
creative ideas or quick clarification
• Searching for web-based information
• Using Chinese as an act of identity

73. Chinese in face-threatening
situations

74. Collaborative practices
• supporting/motivating
• managing the group/task
• negotiating the text
• sharing files and links
• requesting and giving feedback
• peer teaching and learning from peers

75. Learning
4:31pm Zhang
I need grammatical checker !
SCript : After having a breif Idea about the 起源 [origin] of the cathedral
Shouldn't We take a look on the 整體佈局 [overall structure] of the Churc?
4:34pm Rafaela
Brief
Origin
erm
A bit weird
Just say Let's look at
[…]
4:37pm Zhang
.....
I need full sentence
[…]
4:44pm Zhang
?!???!???!??AM I CORRECT ?!??
[…]
After having a breif Idea about the origin of the cathedral
Shouldn't We take a look on the elementary design of the Church?

76. A process-oriented genre approach
to digital literacies

77. The process of course
development
English for science
project
Year 1
• Group presentation (ppt) for the class
• Written report
Year 2
• Group digital video documentary for the public
• Written report

78. Embedding ‘digital literacies’
Problem:
There’s no space in
the curriculum for
digital literacies and
multimodality
Solution:
Embed the digital
tools and
multimodal
practices alongside
traditional tasks
Embedding digital literacies is ‘an approach to the use of digital media in
language education, in which digital literacy practices are embedded alongside
more traditional literacy practices, as one element of the course design’ (Hafner,
2014, p. 682)

79. What role do digital tools play in the texts, processes and
practices that we want our students to master?

80. A three-step process
1. Identify and situate authentic genres and
practices
2. Examine the role of multimodality and digital
tools
3. Design projects, tasks and materials

81. STEP ONE
Identify and situate
authentic genres and
practices
a) Communities
b) Audiences
c) Processes and products

82. STEP TWO
Examine the role of
multimodality and
digital tools
a) Ways of meaning
b) Ways of relating
c) Ways of being
d) Ways of thinking
e) Ways of doing

83. STEP THREE
Design projects, tasks
and materials
a) Context analysis
b) Model analysis
c) Guided composition
d) Extended composition
e) Comparison

84. How might you embed multimodality and
digital literacies in your course?

85. Conclusions