Presentation on Interactive Storytelling delivered at the XMedia Lab Conference in Kuala Lumpur Malaysia, April 2012

1. Interac(ve Storytelling
Derek Woodgate

2. Interac(ve Storytelling is a form of digital
entertainment in which users create or where
users can influence a drama(c storyline through
ac(ons, either by issuing commands to the
story's protagonist, or ac(ng as a general
director of events in the narra(ve. Interac(ve
storytelling is a medium where the narra(ve,
and its evolu(on, can be influenced in real‐(me
by a user.

3. Redefining meaning and context
Plot Engagement
Story
Storytelling User
interac3on

4. Drivers and influencers
 New technologies
 Design skills / approaches
 Social change
 Hybridiza(on and convergence
 Cogni(ve and the behavioral sciences
 User interfaces
The user is the story

5. From gaming to interac(ve narra(ve spaces
 Non‐linear – post‐structuralist
 Abstract – nomadic
 Remix lifestyles
 Transmedia
 Mixed reality
 Collabora(ve interplay and improvisa(on
 Extension of how narratology and ludology intersect

6. From programmable movies to
simulated reali(es
 Interference and guidance
 Alterna(ve plots and storylines
 Spa(al transforma(on
 Programmable environments
 Dialog management
 Cogni(ve feedback and explora(on
The user designer

7. The IS designer
 Narra(ve genera(on engine/system (including inbuilt constraint
selector and poten(al for different POV)
 Authoring tools
 3D Computer graphic soXware/ 3D modelers / anima(on
 camera
 visual rendering system
 interac(on mechanism between mul(ple agents, etc
 interac(ve authoring interface / hap(cs
 representa(on soXware/script language
 iden(fica(on and “registra(on” system,
The never‐ending
 A.I.planner, etc.
list

8. AI and machine learning
 Human computer interac(on: pro‐ac(ve and responsive
ac(ons
 Convergence of skills, performance, analysis and adjustment –
improving the user experience
 Pathfinding and planning (incl. non player characters)
 Natural behavioral language for story‐based believable agents
 Interac(ve actors
 Affec(ve compu(ng
 Social engineering of characters
 Reinforcement learning
 Resource gathering

10. Character building / agents

11. LA Noir: adding emo(on
• Facial emo(on
• Real‐world
actors on the
digital stage
• Morality
systems
• Good
conversa(ons

12. The Watchman _ synthe(c
characters
 CG moves into the future
 Digitally‐replicated actors
who'll go on performing long
aAer the actor is gone
 Rewri3ng the story
 Bringing back deceased actors

13. Avatar – super imposing actors
 Crea(ng an en(re world with
computer graphics and then
directly integra(ng ac(ng
performances
 Japanese Telexistence surrogate
Antropomorphic
 Head gear, gloves , vests with
control gear

14. On‐set mo(on capture

15. AI –human interac(on
 Music Creatures: Synthetic AI animals act as an
interface to direct and act with humans in
movies.

16. The hologram, the performer, the projec(on

17. Touchable holograms
 An array of computer‐controlled
ultrasonic wave emiaers with
fine points that emit pulses of
air that feel like pressure on
your hand.
 System knows where your hand
is – behavioral feedback.
 The sensa(on of weight and
texture.

18. Interac(ng with holograms

19. body>data>space

20. Next‐Genera3on Synthe3c Performers
 Introducing levels of autonomy into the synthe(c
performer (such as eye contact, facial and gestural
movement, and lip sync),
 enabling more flexible, engaging, and directable robot‐
mediated performance on the set.
 Second Skin is a project to build a wearable fabric that
supports millimeter‐accurate loca(on and bio‐parameter
tracking at thousands of points on the body.
 Such a fabric can compute and predict 3‐D
representa(ons of human ac(vity and use the
informa(on to augment human performance.

21. Remote interac(on with a robot
 Robo(c avatars will allow humans to fully
experience the environment of other
worlds.
 Through the eyes of robo(c avatars we
will watch the sunrise over the rusty, red
crater rims without having to "experience
suffoca(on, the icy death of ‐200 degrees
C on their skin or the s(ng of microscopic
dust in their eyes.”

22. Learning the character’s soul
Revisi(ng “Creatures” and
Black and White” – the
interac(ve storytelling
learning versions.
Game characters that can
predict the behavior of
players

23. The fantas3c flying books of Mr. Morris Lessmore

24. Interac(ve for the ipad version

25. Interac(ve Storytelling in music videos
 Moonbot Studios and The
Polyphonic Spree together have
created an interac(ve, character‐
based narra(ve music video.
Features:
‐ Seamless world of interac(on
‐ Touch clouds, plants and animals to life
‐ Evolving visuals that change with the music
‐ Follow and interact with a creature named
You‐Me
‐ Tilt your device to sway You‐Me as he falls
‐ Choose between the Studio or Acous(c
song tracks

26. Context based interac(ve storytelling

27. Heavy Rain
 Interac(ve narra(ve if the choices we can make are
merely altering specific events rather than the actual
context. Surely the whole point of having the story play
out differently depending on player choices is to add
replay value.

28. Programmable movies
 Movies that change with context (observer, emo(ons,
place, or (me).
 Long‐ Distance Barcodes work to make both cameras and
the world more intelligent, by allowing users to piece
together and merge separate images using metadata
encoded into the image.
 This work will allow for the merging of mul(ple
viewpoints to create richer stories from varied
storytellers.

29. Sensing Spaces
 Interac(ve reconfigurable spaces
 Integrated visual, virtual, real spaces
 Use of robo(c and AI s(mula(on and
simula(on
 Sensor clusters and networks to drive
performance cues and audio‐visual
interfaces

30. Sensing Spaces
 Gravity to create performance effects
 Wearable compu(ng to create performance and effects
synchronicity
 Narra(ve‐changing sensors using audience feedback
 Use of avatars as performer interfaces

31. Dynamic spatial interfaces
Interac(ve installa(on uses
dynamic spa3al interfaces
and a human‐scale display
system to allow visitors to
explore the photographic
work of ar(st Ta(ana Parcero
on mul(ple levels of their
own choosing.

32. Interac(ve storytelling in
annotated environments
 iRiS (Immediate Remote
Interac(on System
 User can “paint”, create a
game like a jigsaw puzzle, an
interac(ve story with one or
more par(cipants on a façade
from a mobile device.
 ANW (ar(culated Naturality
Web and computer vision

34. Interac(ve human body museum
• Tac(le interac(ve museum experience
that takes visitors through the spaces
inside their own bodies.
• At the museum you can experience hay
fever from inside the nose, replete with
a rollicking sneeze.
• You can don 3D glasses and watch as
cartoon sperm fer(lize an egg. You can
bounce your way across a rubber
tongue as you navigate taste buds and
hear burps welling up in the deep.

35. Adding affect and atmosphere
 Programmable surfaces and addi(onal displays
 New hap(cs, with gesture, smell, vibra(on and temperature
amplifica(on
 Immersive and responsive / Reac(ve video and portraits
 Socially aware systems, ambient and interac(on enhancers
such as: causal mappings between past and future events or
between an ac(on and its reac(on
 Programmed atmospheres and aesthe(c visual effects

36. Situa(on awareness and Behavioral iden((es
 Ambient detec(on ‐ tools to share habits
 Annotated environments and Distributed Interac3ve Video
Arrays for Event Capture and Enhanced Situa3onal and
Interac3ve Awareness
 Human dynamics and emo(on modeling ‐ affec(ve and
cogni(ve percep(on, feedback and awareness ‐ modeling
social interfacing and feedback ‐crea(ng behavioral iden((es

37. You are the performance
‐ SpherAleas device – mobile device
with sensors that create sonorous
shapes
‐ The Theater of the Senses ‐ "Echo of
the Shadow,” Audience par(cipa(on
in a labyrinth of performances
‐ Hidden Worlds of Noise and Voice
(voice to vision)

38. Projec(ng yourself into the game
 Being There
 This projector‐based approach provides a way to visualize
re‐crea(ons of real and imaginary sites that are both
visually and spa(ally realis(c.
 Users have a strong sense of immersion and natural
interac(on as they walk around a virtual site.

39. Cameras
 Computa(onal Cinematography and Display:
 ‐ future cameras for able to share visual content, and
crea(ng 4‐D and 6‐D displays for richer collec(on and
presenta(on of informa(on.
 Video of the future (video manipula(on, video coding,
3D, and perceptual studies), computa(onal
cinematography, human and facial anima(on, and
capture technologies.
 Addi(onal strengths include wireless networking and
computa(onal materials

40. Sensor robots
 Capturing 3D models of environments, outdoor and
indoor, and capturing ac(vity that is happening in
environments, using cameras mounted on mobile robots.
A new mode of deploying computer vision is now
appearing as autonomous robots

41. Displays
 Interac(ve / HolographicTV
 No border screens
 Roll out portable OLED screens

42. Morphable movie studios
 This technology allows one physical space to
represent a variety of things.
 Projects involve techniques to augment and
programmatically change the appearance of
physical objects.
 The physical object is illuminated with a data (or
video or slide) projector.
 The images to be projected are computed with a
3‐D graphics‐rendering program. This allows you
to change the appearance of real objects, adding
special effects to the world around you.

43. Mobile projec(on: Electric Poetry
A web interface for construc(ng poems to be broadcast in
real‐(me on the walls of London's underground sta(ons, for
London Transport's Plalorm for Art
Similar to the popular fridge game, people assemble words
into poems which then appear in public or commercial spaces
around the city

44. Storyteller
 Storyteller: Tools and technologies developed into
toolkits for children to empower them craX their own
compelling stories and characters to foster crea(vity
and learning goals.
 Researchers will also leverage synthe(c performer
technologies to create compelling robo(c or virtual
characters that serve as learning companions for
children

46. Educa(on

47. Interac(ve storytelling infancy
 Interac(ve Storytelling technologies in terms of performance
and scalability
 To make the next genera(on of Interac(ve Storytelling
technologies more accessible
 To develop a more integrated approach to interac(ve
storytelling technologies
 Methodologies to evaluate interac(ve storytelling systems as
well as the media experience of interac(ve narra(ve

48. Going forward
 Performance and scalability
 Adap(ng to new IS and user desires
 Adap(ng new hybrids of converged mix media and
technologies
 Pumng the human in the landscape.
 There are roles of the integra(on of IS in so many
different areas – think beyond gaming and film
Past – Present ‐ Future

49. Assassin Creed 3