Human Neural Machine

Voice of the Machine
-
Human Neural Network
Georgios Spithourakis
PhD Candidate, UCL
Part 1
Voice of the Machine
Humans Describe What They See
Encoding Decoding
???
Information Processing
Information Processing
Machines Describe What they See
Encoding Decoding
???
Encoding an Image
• Convolutional Neural Networks (CNNs)
Encoding an Image
• Convolutional Neural Networks (CNNs)
Layer 1 Layer 2
Encoding an Image
• Convolutional Neural Networks (CNNs)
Layer 1 Layer 2
Generating Text (decoding) (1)
Nude Descending a Staircase,
Duchamp, 1912
Generating Text (decoding) (1)
Toe upon ___, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the ____
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging ___
That parts to let her parts go __.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
Generating Text (decoding) (1)
Toe upon toe, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the ____
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging ___
That parts to let her parts go __.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
Generating Text (decoding) (1)
Toe upon toe, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the stairs
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging ___
That parts to let her parts go __.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
Generating Text (decoding) (1)
Toe upon toe, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the stairs
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging air
That parts to let her parts go __.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
Generating Text (decoding) (1)
Toe upon toe, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the stairs
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging air
That parts to let her parts go by.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
• Recurrent Neural Networks (RNNs)
Input
Output
Generating Text (decoding) (2)
<START> Toe upon
Toe upon toe
• Recurrent Neural Networks (RNNs)
Input
Output
Generating Text (decoding) (2)
<START> Toe upon
Toe upon toe
• Recurrent Neural Networks (RNNs)
Input
Output
Generating Text (decoding) (2)
<START> Toe upon
Toe upon toe
Aardvark 0.1%
.
.
.
Toe 20%
.
.
.
Zebra 0.2%
Part 2
Human Neural Network
Exercise Goals
• Humans become a machine
• Each group is a neural machine
• Each individual is a neuron
• Adaptation
• Communicate in words (not numbers)
• More flexibility
Encoding an image
• Describe what you see
• Pieces of an image
• Objects in a scene
• A story for the scene
Encoding an Image – Pieces
• For each piece, choose 3 words that describe its content
Blue
Uniform
Empty
Sea
Blue
Calm
Sky
Cloud
Sunny
Cloud
Sky
Stick
Sky
Cloud
Cotton
Tightrope
Man
Balancing
Clouds
Sky
Lines
Grey
Cloud
Tripod
City
Landscape
River
Skyscrapers
Landscape
Horizon
Corner
Buildings
Window
Puddle
Mirror
Pavement
Riverside
Town
Park
Buildings
Roads
Below
City
Brownish
Metal
River
Brown
City
Human Neural Machine
Encoding an Image – Objects (individual)
• Group together pieces to identify up to 5 objects
• Describe each with 1 word
• A story should start forming
Blue
Uniform
Empty
Sea
Blue
Calm
Sky
Cloud
Sunny
Cloud
Sky
Stick
Sky
Cloud
Cotton
Tightrope
Man
Balancing
Clouds
Sky
Lines
Grey
Cloud
Tripod
City
Landscape
River
Skyscrapers
Landscape
Horizon
Corner
Buildings
Window
Puddle
Mirror
Pavement
Riverside
Town
Park
Buildings
Roads
Below
City
Brownish
Metal
River
Brown
City
Sky
Man
Roof
City
Encoding an Image – Objects (group)
• Reach an agreement as a group (up to 5 objects, 1 word each)
Sky
Man Tightrope
Roof
Town
Sky
Someone
Reflection
Town
Sky
Man Tightrope
Roof
City
Human Neural Machine
Encoding an Image – Scene/Story
• Decide on story of up to 5 words
Man
walks
tightrope
above
town
Sky
Man Tightrope
Roof
Town
Man walks tightrope above town
Blue
Uniform
Empty
Sea
Blue
Calm
Sky
Cloud
Sunny
Cloud
Sky
Stick
Sky
Cloud
Cotton
Tightrope
Man
Balancing
Clouds
Sky
Lines
Grey
Cloud
Tripod
City
Landscape
River
Skyscrapers
Landscape
Horizon
Corner
Buildings
Window
Puddle
Mirror
Pavement
Riverside
Town
Park
Buildings
Roads
Below
City
Brownish
Metal
River
Brown
City
Sky
Man Tightrope
Roof
Town
The Encoded Image
Generating the Poem (Decoding)
• React to what we saw
• Write a poem word-by-word
• Individually propose alternative continuations
• Collaboratively select one
Decoding – Choose First Word
I
He
The
High
I 4
He 1
The 0
High 1
• Each person
proposes 1 word
• Each person votes
(up to 2 votes) for
‘best’ word
• Cannot vote
yourself!
• Count votes
• Write highest
scoring word to
poem
• If tied, repeat
voting only
between tied
words (or flip coin)
POEM
I
Decoding – Choose Next Word
really
only
walk
stand
really 3
only 0
walk 0
stand 1
• Each person
proposes 1 word
• Each person votes
(up to 2 votes) for
‘best’ word
• Cannot vote
yourself!
• Count votes
• Write highest
scoring word to
poem
• If tied, repeat
voting only
between tied
words (or flip coin)
POEM
I really
Decoding – Choose Next Word
knowing
walking
trotting
standing
knowing 0
walking 3
trotting 0
standing 1
• Each person
proposes 1 word
• Each person votes
(up to 2 votes) for
‘best’ word
• Cannot vote
yourself!
• Count votes
• Write highest
scoring word to
poem
• If tied, repeat
voting only
between tied
words (or flip coin)
POEM
I really had an easy way of walking
Decoding – Speed it up
• One word at a time is too slow for humans…
• Propose whole phrases (as many words as you like)
Decoding – Choose Next Phrase
However tall
I never thought
Looking at a small
Despite
However tall 0
I never thought 1
Looking at a small 3
Despite 1
• Each person
proposes 1 phrase
• Each person votes
(up to 2 votes) for
‘best’ phrase
• Cannot vote
yourself!
• Count votes
• Write highest
scoring phrase to
poem
• If tied, repeat
voting only
between tied
words (or flip coin)
POEM
I really had an easy way of walking
Looking at a small
Human Neural Machine
Part 3
Conclusion
Image: Man leads caravan through desert
The camel holds the hand of the poor man
He’ll watch us tread, he’ll watch us fall
The long shadow of a donkey across the cracked sand
Shabbily clad but standing tall
We each of us must go, all
Surrounded by another expedition,
Across an ocean on the sea of sand,
Mountains gaze upon a vast Egyptian,
A field of sand beneath the silver strand.
Image: Man free falls to ground
We gotta hide behind the Beaver lake!
I wanna know a better place or where,
Another day a little kiss and take,
An angel on the other side of there.
The forest and cliffs standing against the sky
A human being in free fall
Sailing through the air like a fly
Come to me, the grass, a call.
Image: Crowd watches fishes at aquarium
In a box of Nothing
far from the deep
where waves are thrusting
where light goes to sleep
Expecting something from an empty zoo,
Surrounded by an ocean full of fish,
On the other side of me and you,
Beneath the carpet like a jellyfish.
Image: Two men round a campfire
A living fire becomes a doubles title.
To stay protected by the sons of men,
We stuck together like a semi final,
The one and two and three or four of ten.
Marshmallows at dawn
Freshly cut wood burns in the fire
Time slips out a wide yawn
They all sing out in choir
Acknowledgements
• Zena Edwards, CV:iD
• Daniela Paolucci, Apples and Snakes
• Sebastian Riedel, UCL
• Piotr Mirowski, HumanMachine/Deepmind
• Mandana Seyfeddinipur, SOAS
• Marjan Ghazvininejad, USC
• Generating Topical Poetry, EMNLP 2016
Thank you!
1 of 42

Recommended

Genre and sub genre by
Genre and sub  genreGenre and sub  genre
Genre and sub genreEvieTheodore
332 views4 slides
Print screen by
Print screenPrint screen
Print screencopelandadam
67 views1 slide
Heartless of Stone by
Heartless of StoneHeartless of Stone
Heartless of StoneMichael Fullilove
221 views120 slides
Heartless of Stone by
Heartless of StoneHeartless of Stone
Heartless of StoneMichael Fullilove
49 views120 slides
Weakly Supervised Machine Reading by
Weakly Supervised Machine ReadingWeakly Supervised Machine Reading
Weakly Supervised Machine ReadingIsabelle Augenstein
1.5K views31 slides
USFD at SemEval-2016 - Stance Detection on Twitter with Autoencoders by
USFD at SemEval-2016 - Stance Detection on Twitter with AutoencodersUSFD at SemEval-2016 - Stance Detection on Twitter with Autoencoders
USFD at SemEval-2016 - Stance Detection on Twitter with AutoencodersIsabelle Augenstein
1K views1 slide

More Related Content

Viewers also liked

Distant Supervision with Imitation Learning by
Distant Supervision with Imitation LearningDistant Supervision with Imitation Learning
Distant Supervision with Imitation LearningIsabelle Augenstein
2.5K views49 slides
Relation Extraction from the Web using Distant Supervision by
Relation Extraction from the Web using Distant SupervisionRelation Extraction from the Web using Distant Supervision
Relation Extraction from the Web using Distant SupervisionIsabelle Augenstein
1.9K views18 slides
Semantic Search tutorial at SemTech 2012 by
Semantic Search tutorial at SemTech 2012Semantic Search tutorial at SemTech 2012
Semantic Search tutorial at SemTech 2012Peter Mika
5.4K views109 slides
Mapping Keywords to by
Mapping Keywords to Mapping Keywords to
Mapping Keywords to Isabelle Augenstein
1.7K views21 slides
Efectos en la luz by
Efectos en la luzEfectos en la luz
Efectos en la luzdanellyusuga
90 views3 slides
Efetos en la luz el sonido, el magnetismo y la electricidad by
Efetos en la luz el sonido, el magnetismo  y la electricidad Efetos en la luz el sonido, el magnetismo  y la electricidad
Efetos en la luz el sonido, el magnetismo y la electricidad danellyusuga
415 views3 slides

Viewers also liked(15)

Relation Extraction from the Web using Distant Supervision by Isabelle Augenstein
Relation Extraction from the Web using Distant SupervisionRelation Extraction from the Web using Distant Supervision
Relation Extraction from the Web using Distant Supervision
Isabelle Augenstein1.9K views
Semantic Search tutorial at SemTech 2012 by Peter Mika
Semantic Search tutorial at SemTech 2012Semantic Search tutorial at SemTech 2012
Semantic Search tutorial at SemTech 2012
Peter Mika5.4K views
Efetos en la luz el sonido, el magnetismo y la electricidad by danellyusuga
Efetos en la luz el sonido, el magnetismo  y la electricidad Efetos en la luz el sonido, el magnetismo  y la electricidad
Efetos en la luz el sonido, el magnetismo y la electricidad
danellyusuga415 views
Experiencias con la Dependencia (2) by AMFARSalamanca
Experiencias con la Dependencia (2)Experiencias con la Dependencia (2)
Experiencias con la Dependencia (2)
AMFARSalamanca189 views
Session2 g by bfnd
Session2 gSession2 g
Session2 g
bfnd135 views
Vortrag marketing club übersicht ulrich holzbaur by Ulrich Holzbaur
Vortrag marketing club übersicht ulrich holzbaurVortrag marketing club übersicht ulrich holzbaur
Vortrag marketing club übersicht ulrich holzbaur
Ulrich Holzbaur1.1K views
Webinar EU by bfnd
Webinar EUWebinar EU
Webinar EU
bfnd105 views
Question Answering over Linked Data (Reasoning Web Summer School) by Andre Freitas
Question Answering over Linked Data (Reasoning Web Summer School)Question Answering over Linked Data (Reasoning Web Summer School)
Question Answering over Linked Data (Reasoning Web Summer School)
Andre Freitas1.5K views
Natural Language Processing for the Semantic Web by Isabelle Augenstein
Natural Language Processing for the Semantic WebNatural Language Processing for the Semantic Web
Natural Language Processing for the Semantic Web
Isabelle Augenstein4.4K views
Lecture: Question Answering by Marina Santini
Lecture: Question AnsweringLecture: Question Answering
Lecture: Question Answering
Marina Santini3.1K views

Similar to Human Neural Machine

Online03 chapter2 by
Online03 chapter2Online03 chapter2
Online03 chapter2jengoff13
930 views44 slides
Advanced Writing - Session One by
Advanced Writing - Session OneAdvanced Writing - Session One
Advanced Writing - Session OneNima Yousefi
547 views18 slides
Density chatt by
Density chattDensity chatt
Density chattcjjonesin
203 views10 slides
Chapter 1 restless earth by
Chapter 1   restless earthChapter 1   restless earth
Chapter 1 restless earthRhy Trinidad
1.1K views152 slides
Methods in developing a paragraph by
Methods in developing a paragraphMethods in developing a paragraph
Methods in developing a paragraphKat Manuel
4.8K views15 slides
Hertz Case by
Hertz CaseHertz Case
Hertz CaseTiffany Castro
4 views44 slides

Similar to Human Neural Machine(20)

Online03 chapter2 by jengoff13
Online03 chapter2Online03 chapter2
Online03 chapter2
jengoff13930 views
Advanced Writing - Session One by Nima Yousefi
Advanced Writing - Session OneAdvanced Writing - Session One
Advanced Writing - Session One
Nima Yousefi547 views
Density chatt by cjjonesin
Density chattDensity chatt
Density chatt
cjjonesin203 views
Chapter 1 restless earth by Rhy Trinidad
Chapter 1   restless earthChapter 1   restless earth
Chapter 1 restless earth
Rhy Trinidad1.1K views
Methods in developing a paragraph by Kat Manuel
Methods in developing a paragraphMethods in developing a paragraph
Methods in developing a paragraph
Kat Manuel4.8K views
ParadigmFest: sapientia (science quiz) by swarnabha7
ParadigmFest: sapientia (science quiz)ParadigmFest: sapientia (science quiz)
ParadigmFest: sapientia (science quiz)
swarnabha71.9K views
Secondary vocational class 1 by Linda Yi
Secondary vocational class 1Secondary vocational class 1
Secondary vocational class 1
Linda Yi255 views
Merrill Garbus Water Fountain Meaning by Rachel Walters
Merrill Garbus Water Fountain MeaningMerrill Garbus Water Fountain Meaning
Merrill Garbus Water Fountain Meaning
Rachel Walters2 views
Literary Criticism: Deconstruction by kadoskracker
Literary Criticism: DeconstructionLiterary Criticism: Deconstruction
Literary Criticism: Deconstruction
kadoskracker88.3K views
Essay On Human Rights And John Rawls The Law Of Peoples by Jenna Welch
Essay On  Human Rights And John Rawls The Law Of PeoplesEssay On  Human Rights And John Rawls The Law Of Peoples
Essay On Human Rights And John Rawls The Law Of Peoples
Jenna Welch2 views
Lithotripsy Lab Report by Sara Reed
Lithotripsy Lab ReportLithotripsy Lab Report
Lithotripsy Lab Report
Sara Reed2 views
Roll of Thunder, Hear My Cry vocabulary by LAWJH
Roll of Thunder, Hear My Cry vocabularyRoll of Thunder, Hear My Cry vocabulary
Roll of Thunder, Hear My Cry vocabulary
LAWJH16.2K views
AKI Enschede - lecture day 4: THINK - Powers of Scale - Our Universal Domain,... by Ronald van Tienhoven Studio
AKI Enschede - lecture day 4: THINK - Powers of Scale - Our Universal Domain,...AKI Enschede - lecture day 4: THINK - Powers of Scale - Our Universal Domain,...
AKI Enschede - lecture day 4: THINK - Powers of Scale - Our Universal Domain,...
Mysterious Entities Of The Pacific Northwest Part 1 Summary by Kelly Flores
Mysterious Entities Of The Pacific Northwest Part 1 SummaryMysterious Entities Of The Pacific Northwest Part 1 Summary
Mysterious Entities Of The Pacific Northwest Part 1 Summary
Kelly Flores2 views

Recently uploaded

UNIDAD 3 6º C.MEDIO.pptx by
UNIDAD 3 6º C.MEDIO.pptxUNIDAD 3 6º C.MEDIO.pptx
UNIDAD 3 6º C.MEDIO.pptxMarcosRodriguezUcedo
124 views32 slides
Education and Diversity.pptx by
Education and Diversity.pptxEducation and Diversity.pptx
Education and Diversity.pptxDrHafizKosar
177 views16 slides
Women from Hackney’s History: Stoke Newington by Sue Doe by
Women from Hackney’s History: Stoke Newington by Sue DoeWomen from Hackney’s History: Stoke Newington by Sue Doe
Women from Hackney’s History: Stoke Newington by Sue DoeHistory of Stoke Newington
157 views21 slides
AUDIENCE - BANDURA.pptx by
AUDIENCE - BANDURA.pptxAUDIENCE - BANDURA.pptx
AUDIENCE - BANDURA.pptxiammrhaywood
89 views44 slides
CUNY IT Picciano.pptx by
CUNY IT Picciano.pptxCUNY IT Picciano.pptx
CUNY IT Picciano.pptxapicciano
54 views17 slides
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx by
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptxPharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptxMs. Pooja Bhandare
93 views51 slides

Recently uploaded(20)

Education and Diversity.pptx by DrHafizKosar
Education and Diversity.pptxEducation and Diversity.pptx
Education and Diversity.pptx
DrHafizKosar177 views
AUDIENCE - BANDURA.pptx by iammrhaywood
AUDIENCE - BANDURA.pptxAUDIENCE - BANDURA.pptx
AUDIENCE - BANDURA.pptx
iammrhaywood89 views
CUNY IT Picciano.pptx by apicciano
CUNY IT Picciano.pptxCUNY IT Picciano.pptx
CUNY IT Picciano.pptx
apicciano54 views
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx by Ms. Pooja Bhandare
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptxPharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx
Pharmaceutical Inorganic Chemistry Unit IVMiscellaneous compounds Expectorant... by Ms. Pooja Bhandare
Pharmaceutical Inorganic Chemistry Unit IVMiscellaneous compounds Expectorant...Pharmaceutical Inorganic Chemistry Unit IVMiscellaneous compounds Expectorant...
Pharmaceutical Inorganic Chemistry Unit IVMiscellaneous compounds Expectorant...
Ms. Pooja Bhandare109 views
Ch. 8 Political Party and Party System.pptx by Rommel Regala
Ch. 8 Political Party and Party System.pptxCh. 8 Political Party and Party System.pptx
Ch. 8 Political Party and Party System.pptx
Rommel Regala53 views
S1_SD_Resources Walkthrough.pptx by LAZAROAREVALO1
S1_SD_Resources Walkthrough.pptxS1_SD_Resources Walkthrough.pptx
S1_SD_Resources Walkthrough.pptx
LAZAROAREVALO153 views
Classification of crude drugs.pptx by GayatriPatra14
Classification of crude drugs.pptxClassification of crude drugs.pptx
Classification of crude drugs.pptx
GayatriPatra1492 views
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB... by Nguyen Thanh Tu Collection
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
Solar System and Galaxies.pptx by DrHafizKosar
Solar System and Galaxies.pptxSolar System and Galaxies.pptx
Solar System and Galaxies.pptx
DrHafizKosar94 views
ISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks Effectively by PECB
ISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks EffectivelyISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks Effectively
ISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks Effectively
PECB 598 views
Create a Structure in VBNet.pptx by Breach_P
Create a Structure in VBNet.pptxCreate a Structure in VBNet.pptx
Create a Structure in VBNet.pptx
Breach_P75 views
Psychology KS4 by WestHatch
Psychology KS4Psychology KS4
Psychology KS4
WestHatch90 views
Dance KS5 Breakdown by WestHatch
Dance KS5 BreakdownDance KS5 Breakdown
Dance KS5 Breakdown
WestHatch86 views
Sociology KS5 by WestHatch
Sociology KS5Sociology KS5
Sociology KS5
WestHatch76 views

Human Neural Machine

  • 1. Voice of the Machine - Human Neural Network Georgios Spithourakis PhD Candidate, UCL
  • 2. Part 1 Voice of the Machine
  • 3. Humans Describe What They See Encoding Decoding ???
  • 6. Machines Describe What they See Encoding Decoding ???
  • 7. Encoding an Image • Convolutional Neural Networks (CNNs)
  • 8. Encoding an Image • Convolutional Neural Networks (CNNs) Layer 1 Layer 2
  • 9. Encoding an Image • Convolutional Neural Networks (CNNs) Layer 1 Layer 2
  • 10. Generating Text (decoding) (1) Nude Descending a Staircase, Duchamp, 1912
  • 11. Generating Text (decoding) (1) Toe upon ___, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  • 12. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  • 13. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  • 14. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  • 15. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go by. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  • 16. • Recurrent Neural Networks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe
  • 17. • Recurrent Neural Networks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe
  • 18. • Recurrent Neural Networks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe Aardvark 0.1% . . . Toe 20% . . . Zebra 0.2%
  • 20. Exercise Goals • Humans become a machine • Each group is a neural machine • Each individual is a neuron • Adaptation • Communicate in words (not numbers) • More flexibility
  • 21. Encoding an image • Describe what you see • Pieces of an image • Objects in a scene • A story for the scene
  • 22. Encoding an Image – Pieces • For each piece, choose 3 words that describe its content Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City
  • 24. Encoding an Image – Objects (individual) • Group together pieces to identify up to 5 objects • Describe each with 1 word • A story should start forming Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City Sky Man Roof City
  • 25. Encoding an Image – Objects (group) • Reach an agreement as a group (up to 5 objects, 1 word each) Sky Man Tightrope Roof Town Sky Someone Reflection Town Sky Man Tightrope Roof City
  • 27. Encoding an Image – Scene/Story • Decide on story of up to 5 words Man walks tightrope above town Sky Man Tightrope Roof Town
  • 28. Man walks tightrope above town Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City Sky Man Tightrope Roof Town The Encoded Image
  • 29. Generating the Poem (Decoding) • React to what we saw • Write a poem word-by-word • Individually propose alternative continuations • Collaboratively select one
  • 30. Decoding – Choose First Word I He The High I 4 He 1 The 0 High 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I
  • 31. Decoding – Choose Next Word really only walk stand really 3 only 0 walk 0 stand 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really
  • 32. Decoding – Choose Next Word knowing walking trotting standing knowing 0 walking 3 trotting 0 standing 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really had an easy way of walking
  • 33. Decoding – Speed it up • One word at a time is too slow for humans… • Propose whole phrases (as many words as you like)
  • 34. Decoding – Choose Next Phrase However tall I never thought Looking at a small Despite However tall 0 I never thought 1 Looking at a small 3 Despite 1 • Each person proposes 1 phrase • Each person votes (up to 2 votes) for ‘best’ phrase • Cannot vote yourself! • Count votes • Write highest scoring phrase to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really had an easy way of walking Looking at a small
  • 37. Image: Man leads caravan through desert The camel holds the hand of the poor man He’ll watch us tread, he’ll watch us fall The long shadow of a donkey across the cracked sand Shabbily clad but standing tall We each of us must go, all Surrounded by another expedition, Across an ocean on the sea of sand, Mountains gaze upon a vast Egyptian, A field of sand beneath the silver strand.
  • 38. Image: Man free falls to ground We gotta hide behind the Beaver lake! I wanna know a better place or where, Another day a little kiss and take, An angel on the other side of there. The forest and cliffs standing against the sky A human being in free fall Sailing through the air like a fly Come to me, the grass, a call.
  • 39. Image: Crowd watches fishes at aquarium In a box of Nothing far from the deep where waves are thrusting where light goes to sleep Expecting something from an empty zoo, Surrounded by an ocean full of fish, On the other side of me and you, Beneath the carpet like a jellyfish.
  • 40. Image: Two men round a campfire A living fire becomes a doubles title. To stay protected by the sons of men, We stuck together like a semi final, The one and two and three or four of ten. Marshmallows at dawn Freshly cut wood burns in the fire Time slips out a wide yawn They all sing out in choir
  • 41. Acknowledgements • Zena Edwards, CV:iD • Daniela Paolucci, Apples and Snakes • Sebastian Riedel, UCL • Piotr Mirowski, HumanMachine/Deepmind • Mandana Seyfeddinipur, SOAS • Marjan Ghazvininejad, USC • Generating Topical Poetry, EMNLP 2016

Editor's Notes

  1. Toe upon toe, a snowing flesh,   A gold of lemon, root and rind,   She sifts in sunlight down the stairs   With nothing on. Nor on her mind.   We spy beneath the banister   A constant thresh of thigh on thigh--   Her lips imprint the swinging air   That parts to let her parts go by.   One-woman waterfall, she wears   Her slow descent like a long cape   And pausing, on the final stair   Collects her motions into shape.
  2. TOE/STAIRS/AIR/BY
  3. TOE/STAIRS/AIR/BY
  4. TOE/STAIRS/AIR/BY
  5. TOE/STAIRS/AIR/BY
  6. TOE/STAIRS/AIR/BY