Machinima – Multimodal Digital Storytelling Angela Thomas, 2007 angelaathomas.com [email_address] or… what do a Harvard Law Professor and 11 year old Laura have in common?

3D Animations: Frontier Technologies

Machinima: machine cinema “ the convergence of filmmaking, animation and game development… [it] is real-world filmmaking techniques applied within an interactive virtual space where characters and events can be either controlled by humans, scripts or artificial intelligence”. (Paul Marino, Academy of Machinima Arts & Sciences, 2002-2003, online).

“ the convergence of filmmaking, animation and game development… [it] is real-world filmmaking techniques applied within an interactive virtual space where characters and events can be either controlled by humans, scripts or artificial intelligence”. (Paul Marino, Academy of Machinima Arts & Sciences, 2002-2003, online).

Younger kids are using machinima to create their own experimental narratives “ I was a teenage machinima maker” Once I discovered machinima, it became so much more exciting for me to create my own characters and stories than to simply play the game within the confines of the designer's original creative vision. Now I am able to mold the game to reflect my own imagination Harrison Heller, in the US magazine Variety on July 23rd, 2006.

“ I was a teenage machinima maker”

Once I discovered machinima, it became so much more exciting for me to create my own characters and stories than to simply play the game within the confines of the designer's original creative vision. Now I am able to mold the game to reflect my own imagination

From Parodying Fans to Serious Business To push the envelope of film making techniques using cutting edge new media technologies To appeal to the youtube generation To create opportunities for distance education To create conversation and “buzz” (the new form of entertainment in convergence culture) The Bottom Line

To push the envelope of film making techniques using cutting edge new media technologies

To appeal to the youtube generation

To create opportunities for distance education

To create conversation and “buzz” (the new form of entertainment in convergence culture)

The Bottom Line

Machinima Genres Narrative Comedy (often by fans, insiders) Documentary Advertising Interviews News Clips Promotional Educational Blended Genre (news, promotional, interview, advertising)

Narrative

Comedy (often by fans, insiders)

Documentary

Advertising

Interviews

News Clips

Promotional

Educational

Blended Genre (news, promotional, interview, advertising)

Narrative Australian Film, Television and Radio School Kacia is told by a girlfriend Faina that Sasha was seen jumping and must have surely died. She is shown a photograph of the body and cries. kacia (v.o.) Said she’d read a news item about a suicide. There was no doubt in my mind. It was Sasha.

Comedy

Film Noir More sophisticated blends of the real and the virtual

Documentary

Mainstream Media Documentaries about Virtual Worlds

Advertising

An Interview with Governor Warner (US politician reaching “the masses”)

Promotional Harvard Law School

Educational Science Teachers – tour of the solar system

Blended Genre: Promotional / News / Interviews / Music

Child driven improvements: Kahootz as a gaming platform More sophisticated audio for film construction

Kids’ Machinima using Kahootz Engine

Technical difference SL: highly sophisticated 3D gaming environment, with specialised lighting effects, water ripples, high end, high resolution pixelation, texturing potential, high modality (photo-realism) Kahootz: clunky, designed for kids, much lower modality SL: user-controlled camera Kahootz: scripted / keyframe camera work (more control, complex shots can be developed with pinpoint accuracy) (way points vs keypoints) SL: 1-1 avatar/computer Kahootz: group:1 SL: pre-programmed animations Kahootz: more coding required to construct animations

SL: highly sophisticated 3D gaming environment, with specialised lighting effects, water ripples, high end, high resolution pixelation, texturing potential, high modality (photo-realism)

Kahootz: clunky, designed for kids, much lower modality

SL: user-controlled camera

Kahootz: scripted / keyframe camera work (more control, complex shots can be developed with pinpoint accuracy)

(way points vs keypoints)

SL: 1-1 avatar/computer

Kahootz: group:1

SL: pre-programmed animations

Kahootz: more coding required to construct animations

Multimodal Discourse Analysis How the various semiotics (verbal, image, sound) work collaboratively to realise a coherent multimodal text (O’Halloran, 2005) How the various semiotics work synergistically (Royce, 1998; Ravelli, 2000) Specifically comparing the complexities of resources used by experts as opposed to kids in the unfolding of the filmic event, in order to suggest a pedagogical framework for the advancement of children’s innovative use of frontier technologies

How the various semiotics (verbal, image, sound) work collaboratively to realise a coherent multimodal text (O’Halloran, 2005)

How the various semiotics work synergistically (Royce, 1998; Ravelli, 2000)

Specifically comparing the complexities of resources used by experts as opposed to kids in the unfolding of the filmic event, in order to suggest a pedagogical framework for the advancement of children’s innovative use of frontier technologies

Specifically looking at the use of semiotic resources to construct meanings appropriate for context and text, and NOT the affordances of the software Interplay of semiotic salience as the unfolding chain of events of the narrative/text weaves back and forth through modes Modal complexity and modal density (Norris, 2006) – the moments which have maximum impact according to the particular texturing of the resources Question: can children texture the resources to construct a modally dense and complex text?

Interplay of semiotic salience as the unfolding chain of events of the narrative/text weaves back and forth through modes

Modal complexity and modal density (Norris, 2006) – the moments which have maximum impact according to the particular texturing of the resources

Question: can children texture the resources to construct a modally dense and complex text?

Comparison 1 (Narrative): Robbie Dingo’s “Better Life” Nathan Burns’s “Taken”

Better Life Rob Wright aka Robbie Dingo Music Professor in the UK Well known Machinima maker Won several awards at machinima festivals with his work SFX: Wind sound

Accoustic guitar intro Sweeps around darkened room from close up objects to desk to man working at computer

Accoustic guitar intro

Sweeps around darkened room from close up objects to desk to man working at computer

Where do you go When your eyes are closin’ Are you really hopin’ For a better life Acoustic guitar Solo voice Music described as: raw and gutsy roots/rock sound Measured time Monorhythmic Legato (slow, dream-like) and the musical phrase has vocal silences mezzoforte (moderately loud) (think: medium social distance)

Where do you go

When your eyes are closin’

Are you really hopin’

For a better life

Acoustic guitar

Solo voice

Music described as: raw and gutsy roots/rock sound

Measured time

Monorhythmic

Legato (slow, dream-like) and the musical phrase has vocal silences

mezzoforte (moderately loud) (think: medium social distance)

What do you see When you are dreamin Are you really seein Chain of words before Included : Closing eyes Hoping Dreaming For a better life So they suddenly make sense at this moment as we realise the significance of the wheelchair – the meaning is only through connecting the verbal mode with the visual here

What do you see

When you are dreamin

Are you really seein

Chain of words before

Included :

Closing eyes

Hoping

Dreaming

For a better life

So they suddenly make sense at this moment as we realise the significance of the wheelchair – the meaning is only through connecting the verbal mode with the visual here

Such a better world Such a better world Guitar solo Camera angles swinging around and around – for emphasis?

Such a better world

Such a better world

Guitar solo

All of the time That you’ve been wastin’ When you’ve been waitin’

All of the time

That you’ve been wastin’

When you’ve been waitin’

For a better world Better world And now you don’t know Where the next day will leave you Will they ever believe you That there’s a better life Better life

For a better world

Better world

And now you don’t know

Where the next day will leave you

Will they ever believe you

That there’s a better life

Better life

Guitar solo vocal silence - dramatic pause, foreshadowing the next story event Close up shot for maximum contact

Guitar solo vocal silence

- dramatic pause, foreshadowing the next story event

Close up shot for maximum contact

Music adds in synthesised keyboard replacing the vocals of the melodic line – human element gone from music, as narrative shifts from reality to fantasy, but the melodic line of the instrument imitates what the vocal line did, so we transfer the meaning across – this IS a better life Crescendo of music to forte Change of keys Low pitch to higher pitch and back, melodic line rising and falling along with the flight patterns that are animated Fantasy Reality New Given Instrumental Human voice Figure Figure

Music adds in synthesised keyboard replacing the vocals of the melodic line – human element gone from music, as narrative shifts from reality to fantasy, but the melodic line of the instrument imitates what the vocal line did, so we transfer the meaning across – this IS a better life

Crescendo of music to forte

Change of keys

Low pitch to higher pitch and back, melodic line rising and falling along with the flight patterns that are animated

Special animation feature of particle effects on the feet: magical “pixie dust” connotation i.e.: we read: it’s not real, indirect affect Swell to ff corresponding to a swell of emotional impact, realising the interpersonal

Special animation feature of particle effects on the feet: magical “pixie dust” connotation

i.e.: we read: it’s not real, indirect affect

Swell to ff corresponding to a swell of emotional impact, realising the interpersonal

Rit (gradually growing slower) Diminuendo (gradually growing softer) Long sustained final note on the keyboard before it drops out and the melodic line ceases Last quiet strumming chords silence

Rit (gradually growing slower)

Diminuendo (gradually growing softer)

Long sustained final note on the keyboard before it drops out and the melodic line ceases

Last quiet strumming chords

silence

TAKEN Nathan Burns Grade 6, Princes Hill Primary School original poem Normal day just like always

Normal day just like always

SFX: Car tires screeching Then the white man comes He takes me He forces me into his car He hits me He hits me He hits me three times

Then the white man comes

He takes me

He forces me into his car

He hits me

He hits me

He hits me three times

All around me people scream , People cry , People pray Pray for me I can see his eyes Cold No expression No feeling No sympathy My legs give way My body surrenders My eyes cry, And so does my heart

All around me people scream ,

People cry ,

People pray

Pray for me

I can see his eyes

Cold

No expression

No feeling

No sympathy

My legs give way

My body surrenders

My eyes cry,

And so does my heart

He takes me away Takes me away Away from life Away from family Away from home.

He takes me away

Takes me away

Away from life

Away from family

Away from home.

I am stunned and angry and distressed and trapped A new home is here. A home with food. A home with shelter. A home with cleanliness. But not my home. Camera movement for a purpose: power angles

I am stunned

and angry

and distressed

and trapped

A new home is here.

A home with food.

A home with shelter.

A home with cleanliness.

But not my home.

I’m confined in this malicious cage Animated body writhing on bed Motion of animation becomes salient, repeated lopping echoing the repetition of the text before Repetition an intersemiotic cohesive device Flashing floor reminded me of a migraine attack, or burning fires of hell

I’m confined in this malicious cage

Animated body writhing on bed

Motion of animation becomes salient, repeated lopping echoing the repetition of the text before

Repetition an intersemiotic cohesive device

Flashing floor reminded me of a migraine attack, or burning fires of hell

But No More Flashing black and white animation – lights blinking on and off Pan in to extreme close up / frontal angle / demand for maximum contact with viewer Until the eyes are the only salient aspect of the image which is the only semiotic resource at this moment, building to and foreshadowing the next story event

But No More

Flashing black and white animation – lights blinking on and off

Pan in to extreme close up / frontal angle / demand for maximum contact with viewer

Until the eyes are the only salient aspect of the image which is the only semiotic resource at this moment, building to and foreshadowing the next story event

I run to the waterfall And the white man follows SFX: chirping

I run to the waterfall

And the white man follows

He took me away once But not twice . He hit me three times , But not four . I don’t care anymore I just want it to stop . So I jump . SFX: water Repeated SFX of water Splash Repeated SFX of water Waterfall: metaphorical turbulence, foregrounding inner emotions of the boy

He took me away once

But not twice .

He hit me three times ,

But not four .

I don’t care anymore

I just want it to stop .

So I jump .

SFX: water

Repeated SFX of water

Splash

Repeated SFX of water

It all ends . Sometimes the sudden loss of one of the semiotic resources such as the foregrounding of silence or the absence of colour acts as a powerful interpersonal device – because we are shocked into interaction and forced to fill in those gaps

Comparison / Contrast Both are richly layered and textured to realise ideational meanings Both use music/SFX and then the absence of sound as an interpersonal resource Texturing of visual depiction with sound effects / specialised animation effects used effectively in both Both animations shot using filmic devices: sweep around, birds-eye shot, pan back, pan forward, pan up, pan down, and so on Both achieve a high level of interpersonal contact – don’t they? More explicit use of affect in text of poem More modulation of sound used in BL Textual cohesion of written words in poem is very sophisticated (repetition, collocation, synonyms) whereas the lyrical text used much less Textual composition of BL composed of a larger variety of shot types, overlays and transitions Structurally there are less shots per story event in Taken BL uses a more sophisticated “passing the story meanings through modes” technique

Both are richly layered and textured to realise ideational meanings

Both use music/SFX and then the absence of sound as an interpersonal resource

Texturing of visual depiction with sound effects / specialised animation effects used effectively in both

Both animations shot using filmic devices: sweep around, birds-eye shot, pan back, pan forward, pan up, pan down, and so on

Both achieve a high level of interpersonal contact – don’t they?

More explicit use of affect in text of poem

More modulation of sound used in BL

Textual cohesion of written words in poem is very sophisticated (repetition, collocation, synonyms) whereas the lyrical text used much less

Textual composition of BL composed of a larger variety of shot types, overlays and transitions

Structurally there are less shots per story event in Taken

BL uses a more sophisticated “passing the story meanings through modes” technique

Comparison 2 (Promotional) New Media Consortium’s “Seriously Engaging” Corio South Primary School’s “Our School is Cool”

Voice Over: The New Media Consortium is an international not for profit organisation of nearly 200 leading colleges, universities, museums, corporations and other learning focussed bodies dedicated to pushing the edge of new media and new technology. C: Hello my name is Cassandra Eaves D: Hello my name is Daniel McCullogh L: Hello my name is Laura Eaves M: Hello my name is Matthew Robbins J: Hello my name is Jessica Russell Ju:Hello my name is Justin Grovis Chorus: Welcome to our school is cool movie Music: Drums and guitar “rock” style background music Music: Synthesised keyboard and drums “techno” style background music

Voice Over:

The New Media Consortium is an international not for profit organisation of nearly 200 leading colleges, universities, museums, corporations and other learning focussed bodies dedicated to pushing the edge of new media and new technology.

The campus extends the NMC experience into 3D virtual space, providing a space for insightful interaction, collaboration, learning and experimentation Our school has lots of computers . There is a computer for everyone in the Grade 6 room Music: Drums and guitar “rock” style background music Music: Synthesised keyboard and drums “techno” style background music (techno: not human, = science and technology)

The campus extends the NMC experience into 3D virtual space, providing a space for insightful interaction, collaboration, learning and experimentation

Our school has lots of computers . There is a computer for everyone in the Grade 6 room

A wide range of activities are now planned for the NMC campus, from large scale virtual components of real life events like the NMC summer conference This is Mr Baker’s room , and he’s got a data projector. We’ve got computers in the classroom as well. Music: Drums and guitar “rock” style background music Music: Synthesised keyboard and drums “techno” style background music

A wide range of activities are now planned for the NMC campus, from large scale virtual components of real life events like the NMC summer conference

This is Mr Baker’s room , and he’s got a data projector. We’ve got computers in the classroom as well.

“And does what dogs do” Meaning is passed between modes, here the meaning leaves audio and is depicted in the visuals only

To learn more about the NMC’s journey into virtual space and to follow along as the project progresses, see the NMC campus observer, a blog set up to record, not only our insights, but those of other Second Life residents Music: No change Music and Text: Words of the school song sung by Mr Baker playing the guitar – very hip! Oh You can do it / be confident Oh You can do it / be confident Be brave and know that you are special Be confident at Corio South…. Demand, old man with walking stick dancing, medium frontal shot: cool and inviting, realises complex interpersonal meanings here NMC: Seriously engaging? Corio South: Cool School?

To learn more about the NMC’s journey into virtual space and to follow along as the project progresses, see the NMC campus observer, a blog set up to record, not only our insights, but those of other Second Life residents

Comparison / Contrast Both are richly layered and textured to realise meanings Both use music as a background resource Texturing of visual depiction with specialised animation effects used effectively in both Both animations shot using filmic devices: sweep around, birds-eye shot, pan back, pan forward, pan up, pan down, and so on Both achieve their goal in terms of the promotion of experiential meanings More explicit interpersonal connections made in OSIC More demands, more use of personal pronouns More modulation of sound used in OSIC SE is a far more lexically dense and sophisticated text Textual composition of SE composed of a larger variety of shot types, overlays and transitions Structurally there are less shots per text event in OSIC OSIC uses one example of “passing the story meanings through modes” technique

Both are richly layered and textured to realise meanings

Both use music as a background resource

Texturing of visual depiction with specialised animation effects used effectively in both

Both animations shot using filmic devices: sweep around, birds-eye shot, pan back, pan forward, pan up, pan down, and so on

Both achieve their goal in terms of the promotion of experiential meanings

More explicit interpersonal connections made in OSIC

More demands, more use of personal pronouns

More modulation of sound used in OSIC

SE is a far more lexically dense and sophisticated text

Textual composition of SE composed of a larger variety of shot types, overlays and transitions

Structurally there are less shots per text event in OSIC

OSIC uses one example of “passing the story meanings through modes” technique

Conclusions Children are already creating richly textured multimodal texts, texts in which modal complexity and modal density is used effectively at appropriate stages for the context of the text Children have an excellent understanding of film and media literacy in terms of visuals and an excellent understanding of how and when to use particular resources with the unfolding events of a text Texts are of a highly sophisticated construction in terms of narrative, originality and innovation, and are well matched in semiotic resource deployment with those constructed by adults children are learning the high end skills and creativity of approach that will equip them for more sophisticated platforms in the future and for innovation with the frontier technology of tomorrow.

Children are already creating richly textured multimodal texts, texts in which modal complexity and modal density is used effectively at appropriate stages for the context of the text

Children have an excellent understanding of film and media literacy in terms of visuals and an excellent understanding of how and when to use particular resources with the unfolding events of a text

Texts are of a highly sophisticated construction in terms of narrative, originality and innovation, and are well matched in semiotic resource deployment with those constructed by adults

children are learning the high end skills and creativity of approach that will equip them for more sophisticated platforms in the future and for innovation with the frontier technology of tomorrow.