Download to read offline
Human Neural Machine
- 1. Voice of theMachine - Human Neural Network Georgios Spithourakis PhD Candidate, UCL
- 2. Part 1 Voice ofthe Machine
- 3. Humans Describe WhatThey See Encoding Decoding ???
- 4.
- 5.
- 6. Machines Describe Whatthey See Encoding Decoding ???
- 7. Encoding an Image •Convolutional Neural Networks (CNNs)
- 8. Encoding an Image •Convolutional Neural Networks (CNNs) Layer 1 Layer 2
- 9. Encoding an Image •Convolutional Neural Networks (CNNs) Layer 1 Layer 2
- 10. Generating Text (decoding)(1) Nude Descending a Staircase, Duchamp, 1912
- 11. Generating Text (decoding)(1) Toe upon ___, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 12. Generating Text (decoding)(1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 13. Generating Text (decoding)(1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 14. Generating Text (decoding)(1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 15. Generating Text (decoding)(1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go by. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 16. • Recurrent NeuralNetworks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe
- 17. • Recurrent NeuralNetworks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe
- 18. • Recurrent NeuralNetworks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe Aardvark 0.1% . . . Toe 20% . . . Zebra 0.2%
- 19. Part 2 Human NeuralNetwork
- 20. Exercise Goals • Humansbecome a machine • Each group is a neural machine • Each individual is a neuron • Adaptation • Communicate in words (not numbers) • More flexibility
- 21. Encoding an image •Describe what you see • Pieces of an image • Objects in a scene • A story for the scene
- 22. Encoding an Image– Pieces • For each piece, choose 3 words that describe its content Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City
- 24. Encoding an Image– Objects (individual) • Group together pieces to identify up to 5 objects • Describe each with 1 word • A story should start forming Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City Sky Man Roof City
- 25. Encoding an Image– Objects (group) • Reach an agreement as a group (up to 5 objects, 1 word each) Sky Man Tightrope Roof Town Sky Someone Reflection Town Sky Man Tightrope Roof City
- 27. Encoding an Image– Scene/Story • Decide on story of up to 5 words Man walks tightrope above town Sky Man Tightrope Roof Town
- 28. Man walks tightropeabove town Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City Sky Man Tightrope Roof Town The Encoded Image
- 29. Generating the Poem(Decoding) • React to what we saw • Write a poem word-by-word • Individually propose alternative continuations • Collaboratively select one
- 30. Decoding – ChooseFirst Word I He The High I 4 He 1 The 0 High 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I
- 31. Decoding – ChooseNext Word really only walk stand really 3 only 0 walk 0 stand 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really
- 32. Decoding – ChooseNext Word knowing walking trotting standing knowing 0 walking 3 trotting 0 standing 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really had an easy way of walking
- 33. Decoding – Speedit up • One word at a time is too slow for humans… • Propose whole phrases (as many words as you like)
- 34. Decoding – ChooseNext Phrase However tall I never thought Looking at a small Despite However tall 0 I never thought 1 Looking at a small 3 Despite 1 • Each person proposes 1 phrase • Each person votes (up to 2 votes) for ‘best’ phrase • Cannot vote yourself! • Count votes • Write highest scoring phrase to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really had an easy way of walking Looking at a small
- 36.
- 37. Image: Man leadscaravan through desert The camel holds the hand of the poor man He’ll watch us tread, he’ll watch us fall The long shadow of a donkey across the cracked sand Shabbily clad but standing tall We each of us must go, all Surrounded by another expedition, Across an ocean on the sea of sand, Mountains gaze upon a vast Egyptian, A field of sand beneath the silver strand.
- 38. Image: Man freefalls to ground We gotta hide behind the Beaver lake! I wanna know a better place or where, Another day a little kiss and take, An angel on the other side of there. The forest and cliffs standing against the sky A human being in free fall Sailing through the air like a fly Come to me, the grass, a call.
- 39. Image: Crowd watchesfishes at aquarium In a box of Nothing far from the deep where waves are thrusting where light goes to sleep Expecting something from an empty zoo, Surrounded by an ocean full of fish, On the other side of me and you, Beneath the carpet like a jellyfish.
- 40. Image: Two menround a campfire A living fire becomes a doubles title. To stay protected by the sons of men, We stuck together like a semi final, The one and two and three or four of ten. Marshmallows at dawn Freshly cut wood burns in the fire Time slips out a wide yawn They all sing out in choir
- 41. Acknowledgements • Zena Edwards,CV:iD • Daniela Paolucci, Apples and Snakes • Sebastian Riedel, UCL • Piotr Mirowski, HumanMachine/Deepmind • Mandana Seyfeddinipur, SOAS • Marjan Ghazvininejad, USC • Generating Topical Poetry, EMNLP 2016
- 42.
Editor's Notes
- #11 Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go by. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.
- #12 TOE/STAIRS/AIR/BY
- #13 TOE/STAIRS/AIR/BY
- #14 TOE/STAIRS/AIR/BY
- #15 TOE/STAIRS/AIR/BY
- #16 TOE/STAIRS/AIR/BY