SlideShare a Scribd company logo
Improvised theatre

with artificial
intelligence
Piotr Mirowski

Albert & A.L.Ex

HumanMachine.live
London Creative AI Meetup

18 January 2017
Language as sequences

“How did you come up with A.L.Ex?”
• Personal experience: learning English as a foreign language.

Learn from patterns of words rather than from grammatical rules.
• Statistical language models:

Learn to compute likelihood of a sentence, based on data.
• Improvised musical (Showstoppers, The Maydays):

rhymes will come naturally… with some practice.
A.L.Ex*LAN server

(plugging extra components)
User interface

(visualisation)
Dialogue system

(turn taking)
Recurrent Neural Network

(text generation)
Remote control

(visualisation, camera)
Physical avatar

(stage partner)
*Artificial Language Experiment
Speech

recognition
Text-to-

speech
Dataset

“Was A.L.Ex trained on movie lines?”
• OpenSubtitles

http://www.opensubtitles.org

http://opus.lingfil.uu.se/OpenSubtitles.php
• 100k movies (1902-2016)
• 880M word tokens
• Dataset used to train dialogue systems

[Vinyals & Le (2015) “A Neural Conversational Model”, ICML Deep Learning Workshop]
• Improv actors work from a huge selection of scripts

[Martin, Harrison & Riedl (2016) “Improvisational Computational Storytelling in Open Worlds”, ICIDS]
Statistical Language Models
• Claude Shannon’s N-grams: P(wn | wn-1, wn-2, …, w1)
• Example of n-gram generation from an improv textbook

[Keith Johnstone (1979) “Impro: Improvisation and the theatre”, Faber and Faber]

dismissed as “not the sort of thing spewed out by the unconscious”

“The head and frontal attack on an English writer
that the character of this point is therefore
another method for the letters that the time of
whoever told the problem for an unexpected […]”
[Jeffrey L Elman (1991) “Distributed representations, simple recurrent networks and grammatical structure”, Machine Learning;

Tomas Mikolov et al. (2010) “Recurrent neural network based language model”, INTERSPEECH]
Recurrent Neural Networks

“How long is A.L.Ex’s memory?”
“persistent memory”:

state variable

for arbitrarily

long contexts
Long Short-Term Memory (LSTM)
+⨉
+
forget gate
⨉
input

gate
⨉
output gate
state

ht-1
state

ht
output yt
input xt
cell

ct-1
cell

ct
[Sepp Hochreiter and Jürgen Schmidhuber (1997) “Long Short-Term Memory”, Neural Computation;
Alex Graves (2013) “Generating sequences with recurrent neural networks”, arXiv 1308.0850]
4 LSTMs

stacked

on top of

each other

(hierarchical

representation

of text)
Topic models

“Can A.L.Ex stay on topic?”
• Latent Dirichlet Allocation
(LDA)

with 64 topics
• Computed per movie

at training time
• Computed in real-time

during improv
• Extra input

to “stay on topic”
[Mirowski et al. (2010) “Feature-Rich Continuous Language Models for Speech Recognition”, SLT]
Topic 62:
vaccine
tox
hodgins

e.r.

rayna

bp

ct

serum

mri

cdc

biopsy

karev

surgeries

abdominal

scalpel
Topic 46:
solar

galaxy
nasa
s.h.i.e.l.d.
orbit
nadia
galaxies
sonic
reactor
asteroid
kraang
activate
satellites
tardis
spaceship
Topic 21:
samurai
sensei
yakuza
naruto
angelina
yoko
honda
shinichi
yamato
kato
kimura
kyoto
yamamoto
shogun
jutsu
Topic 6:
homicide
defendant
prosecutor
nypd
forensics
unsure
callen
weeks
dci
ween
css
chi
annalise
priors
provenza
“Could A.L.Ex be rewarded

for funny or successful scenes ?”
• Currently word by word text generation, supervised training
• Still thinking about proper way for reinforcement learning…
• Reward structure: amount of laughs?
• Improv and theatre in open-world setting are more than a game
• Reward dialogue system if it stays on track of an emotional trajectory?

[Hernandez, Bulitko et al (2015) “Keeping the Player on an Emotional Trajectory in Interactive Storytelling”, AAAI]
• Reward dialogue system for informative and coherent dialogue, train on self-play?

[Li et al (2016) “Deep Reinforcement Learning for Dialogue Generation”, arXiv]
ELIZA [Joseph Weizenbaum (1966)]
Physical avatar

“Could you have a robot on stage?”
• Educational robot [www.ez-robot.com]

with C# Software Development Kit
• 16 servos, control angles
• Two 3x3 LED grids, control colour
• Camera: add face tracking (OpenCV)
[Image credit: www.ez-robot.com]
A fun, personal, DIY project
• Not a research project - but relying on (relatively) state-of-the-art research
• Coded in Lua / Torch, Python, C# and Javascript
• Recurrent Neural Language Model: Torch RNN

github.com/jcjohnson/torch-rnn
• Topic Model: Vowpal Wabbit

https://github.com/JohnLangford/vowpal_wabbit
• EZ-Robot JD Humanoid, EZ-SDK Mono

https://www.ez-robot.com/
• Trained on a GPU in the cloud
April 2016, Pyggy
http://korymathewson.com/building-an-artificial-improvisor/
Kory Mathewson
RNNs in theatre and cinema
“Sunspring” (2016)

Ross Goodwin (rossgoodwin.com), Oscar Sharp

[http://arstechnica.com/the-multiverse/2016/06/an-ai-wrote-this-movie-and-its-strangely-moving/]
“Beyond the Fence” (2016)

Benjamin Till, Nathan Taylor
Photograph: Tristram Kenton for the Guardian
A computer stage partner
[Image credits: Kory Mathewson,
http://korymathewson.com/building-an-artificial-improvisor/]
• Improv lessons from

playing with an AI
• Humility lesson

from improv and theatre
Improv is about doing the obvious thing
• Surprisingly hard…
• Trying to be “funny” or “interesting” makes improv boring.
• Need to overcome social fears.
• “I began to think of children not as immature adults,

but of adults as atrophied children.”

[Keith Johnstone (1979) “Impro: Improvisation and the theatre”, Faber and Faber]
• “Know that you knew how to do this when you were six years old,

other stuff just got in the way.”

[Jill Bernard (2002) “Small Cute Book of Improv”, YESand.com]
Search query auto-completion 

and listening skills in improv
• Simply suggest the most obvious (relevant) query given the context:

query prefix, location, time of day, day of year, previous searches…
• Improvisers: listen to cues to extend the context!

(explicit) characters, story, reincorporation of past facts…

(implicit) body language, theory of mind…
improv games
improv classes london

improv everywhere
improv comedy
impossible quizz
impetigo
imperial war museum
imperial college london
instagram
itv player
indeed
iplayer
imp improvi
A.L.Ex: an exercise in justification
• When the machine gives difficult suggestions…
• Justification game: “real-time dynamic problem solving”.

Improvisers need to (observe, repair, accept) divergences.

[Magerko et al (2009) “An Empirical Study of Cognition and Theatrical Improvisation”, C&C]
• Being able to improvise with anybody:

e.g., Kory would improvise with a non-improviser audience member.
• Make the stage partner look good!
Fortunate failures
• With tech on stage, preparation is key…
• Make contingency plans!
• But one should embrace failure.
• 21 September 2016, at the Miller Pub:

had to restart A.L.Ex on stage…
• Decided to show code in next shows.
• Get inspired by failures.
Konstantin
Stanislavski

(1863-1938)

and the actor’s System
Stanislavski (far left) in The Lower Depths

at The Moscow Art Theatre,1902
[Credit: Stanislavski Centre/ArenaPal, BBC]
Diagram of Stanislavski's 'system', based on his "Plan of Experiencing" (1935)
[Credit: Wikimedia Commons]
[Credit: Jean Benedetti (2008) “Stanislavski:
An Introduction”, Bloomsbury]
Determine Given Circumstances:

Where, when, who, what, why,
obstacle, how (to overcome obstacle)
Research the context of the play
and of its characters: original text,
facts, social conventions…
Rely on intellect

to understand the character
Practice physical characterisations
of the character to acquire reflexes

(e.g., animal work)
Each line of the script is actioned,

actor “does” something to others

(how to get what one wants)
Draw on memory and experience

to give emotional depth to play
Determine the units of the script

with their own objectives

(what the character wants)
Aim towards
super-objective 

of the character and of the play
Create a “third being”

character creation 

as self-transformation
A lesson in humility from theatre
• Creating an AS (Artificial
Stanislavski) actor is AI-hard…
• … and somewhat pointless:
• Audience can suspend disbelief
and anthropomorphise robots.
• What is interesting in theatre or
improv is how the human actor
overcomes adversity.
[Image credit: www.pixar.com]
Thank you!

https://humanmachine.live
And many thanks to:

Kory Mathewson,

Alessia Pannese,

Stuart Moses,

Roisin Rae,
John Agapiou,

Katy Schutte,

Benoist Brucker,

Stephen Davidson,
Luba Elliot,
Shama Rahman,

Steve Roe,

Michael Littman…
Love, sex and
marriage…with a robot?
Fri 3 Feb, 18.30–21.30
British Academy Late
31 March, 1 April 2017

More Related Content

Similar to Improvised Theatre with Artificial Intelligence

Moving Image Education (BEd 3)
Moving Image Education (BEd 3)Moving Image Education (BEd 3)
Moving Image Education (BEd 3)
Athole McLauchlan
 
INCORPORATING DIGITAL ARTS IN IGBO-THEMED MOVIE PRODUCTIONS: The Way Forward
INCORPORATING DIGITAL ARTS IN IGBO-THEMED MOVIE PRODUCTIONS: The Way ForwardINCORPORATING DIGITAL ARTS IN IGBO-THEMED MOVIE PRODUCTIONS: The Way Forward
INCORPORATING DIGITAL ARTS IN IGBO-THEMED MOVIE PRODUCTIONS: The Way Forward
ikennaaghanya
 
Creative cosmos, film, creativity and derivative work in OER
Creative cosmos, film, creativity and derivative work in OER Creative cosmos, film, creativity and derivative work in OER
Creative cosmos, film, creativity and derivative work in OER
Tita Beaven
 
chatcptkk.ppt
chatcptkk.pptchatcptkk.ppt
Artificial intelligence
Artificial intelligenceArtificial intelligence
Artificial intelligence
Syed Zaid Irshad
 
AI
AIAI
Ai long forseminar
Ai long forseminarAi long forseminar
Ai long forseminar
chandrakantk1
 
How to Talk to a Cognitive Computer, NLDB2015
How to Talk to a Cognitive Computer,  NLDB2015How to Talk to a Cognitive Computer,  NLDB2015
How to Talk to a Cognitive Computer, NLDB2015
Csaba Veres
 
CityLIS talk, Feb 1st 2016
CityLIS talk, Feb 1st 2016CityLIS talk, Feb 1st 2016
CityLIS talk, Feb 1st 2016
benosteen
 
Mint meeting aug 06 2013
Mint meeting aug 06 2013Mint meeting aug 06 2013
Mint meeting aug 06 2013
Christopher Yap
 
ARIN2600 2009 L6 Space
ARIN2600 2009  L6  SpaceARIN2600 2009  L6  Space
ARIN2600 2009 L6 Space
Chris Chesher
 
Gnomedex 09 Cyborg Anthropology Caseorganic
Gnomedex 09 Cyborg Anthropology CaseorganicGnomedex 09 Cyborg Anthropology Caseorganic
Gnomedex 09 Cyborg Anthropology Caseorganic
Amber Case
 
Creativity in the Remix (2011)
Creativity in the Remix (2011)Creativity in the Remix (2011)
Creativity in the Remix (2011)
Nelson Zagalo
 
On the nature of AI, and the relation between symbolic and statistical approa...
On the nature of AI, and the relation between symbolic and statistical approa...On the nature of AI, and the relation between symbolic and statistical approa...
On the nature of AI, and the relation between symbolic and statistical approa...
Frank van Harmelen
 
Essay Butterfly Effect
Essay Butterfly EffectEssay Butterfly Effect
Essay Butterfly Effect
Jacqueline Ramirez
 
The UnLibrary: library within a library
The UnLibrary: library within a libraryThe UnLibrary: library within a library
The UnLibrary: library within a library
Damien_Wang
 
Gabe smedresman interactivity and the arts
Gabe smedresman interactivity and the artsGabe smedresman interactivity and the arts
Gabe smedresman interactivity and the arts
gabesmed
 
AILongForSemi very useful material for any AI course
AILongForSemi very useful material for any AI courseAILongForSemi very useful material for any AI course
AILongForSemi very useful material for any AI course
KhanKhaja1
 
AILongForSeminar.ppt
AILongForSeminar.pptAILongForSeminar.ppt
AILongForSeminar.ppt
KhanKhaja1
 
Narratives
NarrativesNarratives
Narratives
NINANC
 

Similar to Improvised Theatre with Artificial Intelligence (20)

Moving Image Education (BEd 3)
Moving Image Education (BEd 3)Moving Image Education (BEd 3)
Moving Image Education (BEd 3)
 
INCORPORATING DIGITAL ARTS IN IGBO-THEMED MOVIE PRODUCTIONS: The Way Forward
INCORPORATING DIGITAL ARTS IN IGBO-THEMED MOVIE PRODUCTIONS: The Way ForwardINCORPORATING DIGITAL ARTS IN IGBO-THEMED MOVIE PRODUCTIONS: The Way Forward
INCORPORATING DIGITAL ARTS IN IGBO-THEMED MOVIE PRODUCTIONS: The Way Forward
 
Creative cosmos, film, creativity and derivative work in OER
Creative cosmos, film, creativity and derivative work in OER Creative cosmos, film, creativity and derivative work in OER
Creative cosmos, film, creativity and derivative work in OER
 
chatcptkk.ppt
chatcptkk.pptchatcptkk.ppt
chatcptkk.ppt
 
Artificial intelligence
Artificial intelligenceArtificial intelligence
Artificial intelligence
 
AI
AIAI
AI
 
Ai long forseminar
Ai long forseminarAi long forseminar
Ai long forseminar
 
How to Talk to a Cognitive Computer, NLDB2015
How to Talk to a Cognitive Computer,  NLDB2015How to Talk to a Cognitive Computer,  NLDB2015
How to Talk to a Cognitive Computer, NLDB2015
 
CityLIS talk, Feb 1st 2016
CityLIS talk, Feb 1st 2016CityLIS talk, Feb 1st 2016
CityLIS talk, Feb 1st 2016
 
Mint meeting aug 06 2013
Mint meeting aug 06 2013Mint meeting aug 06 2013
Mint meeting aug 06 2013
 
ARIN2600 2009 L6 Space
ARIN2600 2009  L6  SpaceARIN2600 2009  L6  Space
ARIN2600 2009 L6 Space
 
Gnomedex 09 Cyborg Anthropology Caseorganic
Gnomedex 09 Cyborg Anthropology CaseorganicGnomedex 09 Cyborg Anthropology Caseorganic
Gnomedex 09 Cyborg Anthropology Caseorganic
 
Creativity in the Remix (2011)
Creativity in the Remix (2011)Creativity in the Remix (2011)
Creativity in the Remix (2011)
 
On the nature of AI, and the relation between symbolic and statistical approa...
On the nature of AI, and the relation between symbolic and statistical approa...On the nature of AI, and the relation between symbolic and statistical approa...
On the nature of AI, and the relation between symbolic and statistical approa...
 
Essay Butterfly Effect
Essay Butterfly EffectEssay Butterfly Effect
Essay Butterfly Effect
 
The UnLibrary: library within a library
The UnLibrary: library within a libraryThe UnLibrary: library within a library
The UnLibrary: library within a library
 
Gabe smedresman interactivity and the arts
Gabe smedresman interactivity and the artsGabe smedresman interactivity and the arts
Gabe smedresman interactivity and the arts
 
AILongForSemi very useful material for any AI course
AILongForSemi very useful material for any AI courseAILongForSemi very useful material for any AI course
AILongForSemi very useful material for any AI course
 
AILongForSeminar.ppt
AILongForSeminar.pptAILongForSeminar.ppt
AILongForSeminar.ppt
 
Narratives
NarrativesNarratives
Narratives
 

More from Luba Elliott

Luba Elliott - AI art - ICCV Conference
Luba Elliott - AI art - ICCV ConferenceLuba Elliott - AI art - ICCV Conference
Luba Elliott - AI art - ICCV Conference
Luba Elliott
 
Luba Elliott - AI in contemporary art practice - Oxford
Luba Elliott - AI in contemporary art practice - OxfordLuba Elliott - AI in contemporary art practice - Oxford
Luba Elliott - AI in contemporary art practice - Oxford
Luba Elliott
 
Luba Elliott - AI in recent art practice - ML Prague
Luba Elliott - AI in recent art practice - ML PragueLuba Elliott - AI in recent art practice - ML Prague
Luba Elliott - AI in recent art practice - ML Prague
Luba Elliott
 
Three Images of the New - Richard Hames - Creative AI meetup
Three Images of the New - Richard Hames - Creative AI meetupThree Images of the New - Richard Hames - Creative AI meetup
Three Images of the New - Richard Hames - Creative AI meetup
Luba Elliott
 
AI Art Gallery Overview - Luba Elliott - NeurIPS Creativity Workshop
AI Art Gallery Overview - Luba Elliott - NeurIPS Creativity WorkshopAI Art Gallery Overview - Luba Elliott - NeurIPS Creativity Workshop
AI Art Gallery Overview - Luba Elliott - NeurIPS Creativity Workshop
Luba Elliott
 
Creativity is Intelligence - Kenneth Stanley - NeurIPS Creativity Workshop
Creativity is Intelligence - Kenneth Stanley - NeurIPS Creativity WorkshopCreativity is Intelligence - Kenneth Stanley - NeurIPS Creativity Workshop
Creativity is Intelligence - Kenneth Stanley - NeurIPS Creativity Workshop
Luba Elliott
 
Seen by machine: Computational Spectatorship in the BBC Archive
Seen by machine: Computational Spectatorship in the BBC ArchiveSeen by machine: Computational Spectatorship in the BBC Archive
Seen by machine: Computational Spectatorship in the BBC Archive
Luba Elliott
 
Luba Elliott AI art overview
Luba Elliott AI art overview Luba Elliott AI art overview
Luba Elliott AI art overview
Luba Elliott
 
Natasha Jaques - Learning via Social Awareness - Creative AI meetup
Natasha Jaques - Learning via Social Awareness - Creative AI meetupNatasha Jaques - Learning via Social Awareness - Creative AI meetup
Natasha Jaques - Learning via Social Awareness - Creative AI meetup
Luba Elliott
 
Sander Dieleman - Generating music in the raw audio domain - Creative AI meetup
Sander Dieleman - Generating music in the raw audio domain - Creative AI meetupSander Dieleman - Generating music in the raw audio domain - Creative AI meetup
Sander Dieleman - Generating music in the raw audio domain - Creative AI meetup
Luba Elliott
 
Design in AI
Design in AIDesign in AI
Design in AI
Luba Elliott
 
Marco Marchesi - Practical uses of style transfer in the creative industry
Marco Marchesi - Practical uses of style transfer in the creative industryMarco Marchesi - Practical uses of style transfer in the creative industry
Marco Marchesi - Practical uses of style transfer in the creative industry
Luba Elliott
 
Hooman Shayani - CAD/CAM in the Age of AI: Designers’ Journey from Earth to Sky
Hooman Shayani - CAD/CAM in the Age of AI: Designers’ Journey from Earth to SkyHooman Shayani - CAD/CAM in the Age of AI: Designers’ Journey from Earth to Sky
Hooman Shayani - CAD/CAM in the Age of AI: Designers’ Journey from Earth to Sky
Luba Elliott
 
Lucas Theis - Compressing Images with Neural Networks - Creative AI meetup
Lucas Theis - Compressing Images with Neural Networks - Creative AI meetupLucas Theis - Compressing Images with Neural Networks - Creative AI meetup
Lucas Theis - Compressing Images with Neural Networks - Creative AI meetup
Luba Elliott
 
Emily Denton - Unsupervised Learning of Disentangled Representations from Vid...
Emily Denton - Unsupervised Learning of Disentangled Representations from Vid...Emily Denton - Unsupervised Learning of Disentangled Representations from Vid...
Emily Denton - Unsupervised Learning of Disentangled Representations from Vid...
Luba Elliott
 
Luba Elliott - Seeing AI through Art
Luba Elliott - Seeing AI through ArtLuba Elliott - Seeing AI through Art
Luba Elliott - Seeing AI through Art
Luba Elliott
 
Georgia Ward Dyer - O Time thy pyramids - Creative AI meetup
Georgia Ward Dyer - O Time thy pyramids - Creative AI meetupGeorgia Ward Dyer - O Time thy pyramids - Creative AI meetup
Georgia Ward Dyer - O Time thy pyramids - Creative AI meetup
Luba Elliott
 
Daniel Berio - Graffiti synthesis, a motion centric approach - Creative AI me...
Daniel Berio - Graffiti synthesis, a motion centric approach - Creative AI me...Daniel Berio - Graffiti synthesis, a motion centric approach - Creative AI me...
Daniel Berio - Graffiti synthesis, a motion centric approach - Creative AI me...
Luba Elliott
 
Ali Eslami - Artificial Intelligence and Computer Aided Design - Creative AI ...
Ali Eslami - Artificial Intelligence and Computer Aided Design - Creative AI ...Ali Eslami - Artificial Intelligence and Computer Aided Design - Creative AI ...
Ali Eslami - Artificial Intelligence and Computer Aided Design - Creative AI ...
Luba Elliott
 
Daghan Cam - Adaptive Autonomous Manufacturing with AI - Creative AI meetup
Daghan Cam - Adaptive Autonomous Manufacturing with AI - Creative AI meetupDaghan Cam - Adaptive Autonomous Manufacturing with AI - Creative AI meetup
Daghan Cam - Adaptive Autonomous Manufacturing with AI - Creative AI meetup
Luba Elliott
 

More from Luba Elliott (20)

Luba Elliott - AI art - ICCV Conference
Luba Elliott - AI art - ICCV ConferenceLuba Elliott - AI art - ICCV Conference
Luba Elliott - AI art - ICCV Conference
 
Luba Elliott - AI in contemporary art practice - Oxford
Luba Elliott - AI in contemporary art practice - OxfordLuba Elliott - AI in contemporary art practice - Oxford
Luba Elliott - AI in contemporary art practice - Oxford
 
Luba Elliott - AI in recent art practice - ML Prague
Luba Elliott - AI in recent art practice - ML PragueLuba Elliott - AI in recent art practice - ML Prague
Luba Elliott - AI in recent art practice - ML Prague
 
Three Images of the New - Richard Hames - Creative AI meetup
Three Images of the New - Richard Hames - Creative AI meetupThree Images of the New - Richard Hames - Creative AI meetup
Three Images of the New - Richard Hames - Creative AI meetup
 
AI Art Gallery Overview - Luba Elliott - NeurIPS Creativity Workshop
AI Art Gallery Overview - Luba Elliott - NeurIPS Creativity WorkshopAI Art Gallery Overview - Luba Elliott - NeurIPS Creativity Workshop
AI Art Gallery Overview - Luba Elliott - NeurIPS Creativity Workshop
 
Creativity is Intelligence - Kenneth Stanley - NeurIPS Creativity Workshop
Creativity is Intelligence - Kenneth Stanley - NeurIPS Creativity WorkshopCreativity is Intelligence - Kenneth Stanley - NeurIPS Creativity Workshop
Creativity is Intelligence - Kenneth Stanley - NeurIPS Creativity Workshop
 
Seen by machine: Computational Spectatorship in the BBC Archive
Seen by machine: Computational Spectatorship in the BBC ArchiveSeen by machine: Computational Spectatorship in the BBC Archive
Seen by machine: Computational Spectatorship in the BBC Archive
 
Luba Elliott AI art overview
Luba Elliott AI art overview Luba Elliott AI art overview
Luba Elliott AI art overview
 
Natasha Jaques - Learning via Social Awareness - Creative AI meetup
Natasha Jaques - Learning via Social Awareness - Creative AI meetupNatasha Jaques - Learning via Social Awareness - Creative AI meetup
Natasha Jaques - Learning via Social Awareness - Creative AI meetup
 
Sander Dieleman - Generating music in the raw audio domain - Creative AI meetup
Sander Dieleman - Generating music in the raw audio domain - Creative AI meetupSander Dieleman - Generating music in the raw audio domain - Creative AI meetup
Sander Dieleman - Generating music in the raw audio domain - Creative AI meetup
 
Design in AI
Design in AIDesign in AI
Design in AI
 
Marco Marchesi - Practical uses of style transfer in the creative industry
Marco Marchesi - Practical uses of style transfer in the creative industryMarco Marchesi - Practical uses of style transfer in the creative industry
Marco Marchesi - Practical uses of style transfer in the creative industry
 
Hooman Shayani - CAD/CAM in the Age of AI: Designers’ Journey from Earth to Sky
Hooman Shayani - CAD/CAM in the Age of AI: Designers’ Journey from Earth to SkyHooman Shayani - CAD/CAM in the Age of AI: Designers’ Journey from Earth to Sky
Hooman Shayani - CAD/CAM in the Age of AI: Designers’ Journey from Earth to Sky
 
Lucas Theis - Compressing Images with Neural Networks - Creative AI meetup
Lucas Theis - Compressing Images with Neural Networks - Creative AI meetupLucas Theis - Compressing Images with Neural Networks - Creative AI meetup
Lucas Theis - Compressing Images with Neural Networks - Creative AI meetup
 
Emily Denton - Unsupervised Learning of Disentangled Representations from Vid...
Emily Denton - Unsupervised Learning of Disentangled Representations from Vid...Emily Denton - Unsupervised Learning of Disentangled Representations from Vid...
Emily Denton - Unsupervised Learning of Disentangled Representations from Vid...
 
Luba Elliott - Seeing AI through Art
Luba Elliott - Seeing AI through ArtLuba Elliott - Seeing AI through Art
Luba Elliott - Seeing AI through Art
 
Georgia Ward Dyer - O Time thy pyramids - Creative AI meetup
Georgia Ward Dyer - O Time thy pyramids - Creative AI meetupGeorgia Ward Dyer - O Time thy pyramids - Creative AI meetup
Georgia Ward Dyer - O Time thy pyramids - Creative AI meetup
 
Daniel Berio - Graffiti synthesis, a motion centric approach - Creative AI me...
Daniel Berio - Graffiti synthesis, a motion centric approach - Creative AI me...Daniel Berio - Graffiti synthesis, a motion centric approach - Creative AI me...
Daniel Berio - Graffiti synthesis, a motion centric approach - Creative AI me...
 
Ali Eslami - Artificial Intelligence and Computer Aided Design - Creative AI ...
Ali Eslami - Artificial Intelligence and Computer Aided Design - Creative AI ...Ali Eslami - Artificial Intelligence and Computer Aided Design - Creative AI ...
Ali Eslami - Artificial Intelligence and Computer Aided Design - Creative AI ...
 
Daghan Cam - Adaptive Autonomous Manufacturing with AI - Creative AI meetup
Daghan Cam - Adaptive Autonomous Manufacturing with AI - Creative AI meetupDaghan Cam - Adaptive Autonomous Manufacturing with AI - Creative AI meetup
Daghan Cam - Adaptive Autonomous Manufacturing with AI - Creative AI meetup
 

Recently uploaded

BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
DianaGray10
 
The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
Zilliz
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
ankush9927
 
Finetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and DefendingFinetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and Defending
Priyanka Aash
 
COVID-19 and the Level of Cloud Computing Adoption: A Study of Sri Lankan Inf...
COVID-19 and the Level of Cloud Computing Adoption: A Study of Sri Lankan Inf...COVID-19 and the Level of Cloud Computing Adoption: A Study of Sri Lankan Inf...
COVID-19 and the Level of Cloud Computing Adoption: A Study of Sri Lankan Inf...
AimanAthambawa1
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
Enterprise Knowledge
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
Retrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with RagasRetrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with Ragas
Zilliz
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
FIDO Alliance
 
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
OnBoard
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
It's your unstructured data: How to get your GenAI app to production (and spe...
It's your unstructured data: How to get your GenAI app to production (and spe...It's your unstructured data: How to get your GenAI app to production (and spe...
It's your unstructured data: How to get your GenAI app to production (and spe...
Zilliz
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024
Michael Price
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
The Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - CoatueThe Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - Coatue
Razin Mustafiz
 
UX Webinar Series: Aligning Authentication Experiences with Business Goals
UX Webinar Series: Aligning Authentication Experiences with Business GoalsUX Webinar Series: Aligning Authentication Experiences with Business Goals
UX Webinar Series: Aligning Authentication Experiences with Business Goals
FIDO Alliance
 

Recently uploaded (20)

BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
 
The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
 
Finetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and DefendingFinetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and Defending
 
COVID-19 and the Level of Cloud Computing Adoption: A Study of Sri Lankan Inf...
COVID-19 and the Level of Cloud Computing Adoption: A Study of Sri Lankan Inf...COVID-19 and the Level of Cloud Computing Adoption: A Study of Sri Lankan Inf...
COVID-19 and the Level of Cloud Computing Adoption: A Study of Sri Lankan Inf...
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
Retrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with RagasRetrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with Ragas
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
 
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
It's your unstructured data: How to get your GenAI app to production (and spe...
It's your unstructured data: How to get your GenAI app to production (and spe...It's your unstructured data: How to get your GenAI app to production (and spe...
It's your unstructured data: How to get your GenAI app to production (and spe...
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
The Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - CoatueThe Path to General-Purpose Robots - Coatue
The Path to General-Purpose Robots - Coatue
 
UX Webinar Series: Aligning Authentication Experiences with Business Goals
UX Webinar Series: Aligning Authentication Experiences with Business GoalsUX Webinar Series: Aligning Authentication Experiences with Business Goals
UX Webinar Series: Aligning Authentication Experiences with Business Goals
 

Improvised Theatre with Artificial Intelligence

  • 1. Improvised theatre
 with artificial intelligence Piotr Mirowski
 Albert & A.L.Ex
 HumanMachine.live London Creative AI Meetup
 18 January 2017
  • 2. Language as sequences
 “How did you come up with A.L.Ex?” • Personal experience: learning English as a foreign language.
 Learn from patterns of words rather than from grammatical rules. • Statistical language models:
 Learn to compute likelihood of a sentence, based on data. • Improvised musical (Showstoppers, The Maydays):
 rhymes will come naturally… with some practice.
  • 3. A.L.Ex*LAN server
 (plugging extra components) User interface
 (visualisation) Dialogue system
 (turn taking) Recurrent Neural Network
 (text generation) Remote control
 (visualisation, camera) Physical avatar
 (stage partner) *Artificial Language Experiment Speech
 recognition Text-to-
 speech
  • 4. Dataset
 “Was A.L.Ex trained on movie lines?” • OpenSubtitles
 http://www.opensubtitles.org
 http://opus.lingfil.uu.se/OpenSubtitles.php • 100k movies (1902-2016) • 880M word tokens • Dataset used to train dialogue systems
 [Vinyals & Le (2015) “A Neural Conversational Model”, ICML Deep Learning Workshop] • Improv actors work from a huge selection of scripts
 [Martin, Harrison & Riedl (2016) “Improvisational Computational Storytelling in Open Worlds”, ICIDS]
  • 5. Statistical Language Models • Claude Shannon’s N-grams: P(wn | wn-1, wn-2, …, w1) • Example of n-gram generation from an improv textbook
 [Keith Johnstone (1979) “Impro: Improvisation and the theatre”, Faber and Faber]
 dismissed as “not the sort of thing spewed out by the unconscious”
 “The head and frontal attack on an English writer that the character of this point is therefore another method for the letters that the time of whoever told the problem for an unexpected […]”
  • 6. [Jeffrey L Elman (1991) “Distributed representations, simple recurrent networks and grammatical structure”, Machine Learning;
 Tomas Mikolov et al. (2010) “Recurrent neural network based language model”, INTERSPEECH] Recurrent Neural Networks
 “How long is A.L.Ex’s memory?” “persistent memory”:
 state variable
 for arbitrarily
 long contexts
  • 7. Long Short-Term Memory (LSTM) +⨉ + forget gate ⨉ input
 gate ⨉ output gate state
 ht-1 state
 ht output yt input xt cell
 ct-1 cell
 ct [Sepp Hochreiter and Jürgen Schmidhuber (1997) “Long Short-Term Memory”, Neural Computation; Alex Graves (2013) “Generating sequences with recurrent neural networks”, arXiv 1308.0850] 4 LSTMs
 stacked
 on top of
 each other
 (hierarchical
 representation
 of text)
  • 8. Topic models
 “Can A.L.Ex stay on topic?” • Latent Dirichlet Allocation (LDA)
 with 64 topics • Computed per movie
 at training time • Computed in real-time
 during improv • Extra input
 to “stay on topic” [Mirowski et al. (2010) “Feature-Rich Continuous Language Models for Speech Recognition”, SLT] Topic 62: vaccine tox hodgins
 e.r.
 rayna
 bp
 ct
 serum
 mri
 cdc
 biopsy
 karev
 surgeries
 abdominal
 scalpel Topic 46: solar
 galaxy nasa s.h.i.e.l.d. orbit nadia galaxies sonic reactor asteroid kraang activate satellites tardis spaceship Topic 21: samurai sensei yakuza naruto angelina yoko honda shinichi yamato kato kimura kyoto yamamoto shogun jutsu Topic 6: homicide defendant prosecutor nypd forensics unsure callen weeks dci ween css chi annalise priors provenza
  • 9. “Could A.L.Ex be rewarded
 for funny or successful scenes ?” • Currently word by word text generation, supervised training • Still thinking about proper way for reinforcement learning… • Reward structure: amount of laughs? • Improv and theatre in open-world setting are more than a game • Reward dialogue system if it stays on track of an emotional trajectory?
 [Hernandez, Bulitko et al (2015) “Keeping the Player on an Emotional Trajectory in Interactive Storytelling”, AAAI] • Reward dialogue system for informative and coherent dialogue, train on self-play?
 [Li et al (2016) “Deep Reinforcement Learning for Dialogue Generation”, arXiv]
  • 11. Physical avatar
 “Could you have a robot on stage?” • Educational robot [www.ez-robot.com]
 with C# Software Development Kit • 16 servos, control angles • Two 3x3 LED grids, control colour • Camera: add face tracking (OpenCV) [Image credit: www.ez-robot.com]
  • 12. A fun, personal, DIY project • Not a research project - but relying on (relatively) state-of-the-art research • Coded in Lua / Torch, Python, C# and Javascript • Recurrent Neural Language Model: Torch RNN
 github.com/jcjohnson/torch-rnn • Topic Model: Vowpal Wabbit
 https://github.com/JohnLangford/vowpal_wabbit • EZ-Robot JD Humanoid, EZ-SDK Mono
 https://www.ez-robot.com/ • Trained on a GPU in the cloud
  • 14. RNNs in theatre and cinema “Sunspring” (2016)
 Ross Goodwin (rossgoodwin.com), Oscar Sharp
 [http://arstechnica.com/the-multiverse/2016/06/an-ai-wrote-this-movie-and-its-strangely-moving/] “Beyond the Fence” (2016)
 Benjamin Till, Nathan Taylor Photograph: Tristram Kenton for the Guardian
  • 15. A computer stage partner [Image credits: Kory Mathewson, http://korymathewson.com/building-an-artificial-improvisor/] • Improv lessons from
 playing with an AI • Humility lesson
 from improv and theatre
  • 16. Improv is about doing the obvious thing • Surprisingly hard… • Trying to be “funny” or “interesting” makes improv boring. • Need to overcome social fears. • “I began to think of children not as immature adults,
 but of adults as atrophied children.”
 [Keith Johnstone (1979) “Impro: Improvisation and the theatre”, Faber and Faber] • “Know that you knew how to do this when you were six years old,
 other stuff just got in the way.”
 [Jill Bernard (2002) “Small Cute Book of Improv”, YESand.com]
  • 17. Search query auto-completion 
 and listening skills in improv • Simply suggest the most obvious (relevant) query given the context:
 query prefix, location, time of day, day of year, previous searches… • Improvisers: listen to cues to extend the context!
 (explicit) characters, story, reincorporation of past facts…
 (implicit) body language, theory of mind… improv games improv classes london
 improv everywhere improv comedy impossible quizz impetigo imperial war museum imperial college london instagram itv player indeed iplayer imp improvi
  • 18. A.L.Ex: an exercise in justification • When the machine gives difficult suggestions… • Justification game: “real-time dynamic problem solving”.
 Improvisers need to (observe, repair, accept) divergences.
 [Magerko et al (2009) “An Empirical Study of Cognition and Theatrical Improvisation”, C&C] • Being able to improvise with anybody:
 e.g., Kory would improvise with a non-improviser audience member. • Make the stage partner look good!
  • 19. Fortunate failures • With tech on stage, preparation is key… • Make contingency plans! • But one should embrace failure. • 21 September 2016, at the Miller Pub:
 had to restart A.L.Ex on stage… • Decided to show code in next shows. • Get inspired by failures.
  • 20. Konstantin Stanislavski
 (1863-1938)
 and the actor’s System Stanislavski (far left) in The Lower Depths
 at The Moscow Art Theatre,1902 [Credit: Stanislavski Centre/ArenaPal, BBC] Diagram of Stanislavski's 'system', based on his "Plan of Experiencing" (1935) [Credit: Wikimedia Commons]
  • 21. [Credit: Jean Benedetti (2008) “Stanislavski: An Introduction”, Bloomsbury] Determine Given Circumstances:
 Where, when, who, what, why, obstacle, how (to overcome obstacle) Research the context of the play and of its characters: original text, facts, social conventions… Rely on intellect
 to understand the character Practice physical characterisations of the character to acquire reflexes
 (e.g., animal work) Each line of the script is actioned,
 actor “does” something to others
 (how to get what one wants) Draw on memory and experience
 to give emotional depth to play Determine the units of the script
 with their own objectives
 (what the character wants) Aim towards super-objective 
 of the character and of the play Create a “third being”
 character creation 
 as self-transformation
  • 22. A lesson in humility from theatre • Creating an AS (Artificial Stanislavski) actor is AI-hard… • … and somewhat pointless: • Audience can suspend disbelief and anthropomorphise robots. • What is interesting in theatre or improv is how the human actor overcomes adversity. [Image credit: www.pixar.com]
  • 23. Thank you!
 https://humanmachine.live And many thanks to:
 Kory Mathewson,
 Alessia Pannese,
 Stuart Moses,
 Roisin Rae, John Agapiou,
 Katy Schutte,
 Benoist Brucker,
 Stephen Davidson, Luba Elliot, Shama Rahman,
 Steve Roe,
 Michael Littman… Love, sex and marriage…with a robot? Fri 3 Feb, 18.30–21.30 British Academy Late 31 March, 1 April 2017