Recommended
More Related Content
Viewers also liked
Viewers also liked (15)
Similar to Human Neural Machine
Similar to Human Neural Machine (20)
Recently uploaded
Recently uploaded (20)
Human Neural Machine
- 1. Voice of the Machine - Human Neural Network Georgios Spithourakis PhD Candidate, UCL
- 2. Part 1 Voice of the Machine
- 3. Humans Describe What They See Encoding Decoding ???
- 6. Machines Describe What they See Encoding Decoding ???
- 7. Encoding an Image • Convolutional Neural Networks (CNNs)
- 8. Encoding an Image • Convolutional Neural Networks (CNNs) Layer 1 Layer 2
- 9. Encoding an Image • Convolutional Neural Networks (CNNs) Layer 1 Layer 2
- 10. Generating Text (decoding) (1) Nude Descending a Staircase, Duchamp, 1912
- 11. Generating Text (decoding) (1) Toe upon ___, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 12. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 13. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 14. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 15. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go by. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
- 16. • Recurrent Neural Networks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe
- 17. • Recurrent Neural Networks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe
- 18. • Recurrent Neural Networks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe Aardvark 0.1% . . . Toe 20% . . . Zebra 0.2%
- 19. Part 2 Human Neural Network
- 20. Exercise Goals • Humans become a machine • Each group is a neural machine • Each individual is a neuron • Adaptation • Communicate in words (not numbers) • More flexibility
- 21. Encoding an image • Describe what you see • Pieces of an image • Objects in a scene • A story for the scene
- 22. Encoding an Image – Pieces • For each piece, choose 3 words that describe its content Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City
- 24. Encoding an Image – Objects (individual) • Group together pieces to identify up to 5 objects • Describe each with 1 word • A story should start forming Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City Sky Man Roof City
- 25. Encoding an Image – Objects (group) • Reach an agreement as a group (up to 5 objects, 1 word each) Sky Man Tightrope Roof Town Sky Someone Reflection Town Sky Man Tightrope Roof City
- 27. Encoding an Image – Scene/Story • Decide on story of up to 5 words Man walks tightrope above town Sky Man Tightrope Roof Town
- 28. Man walks tightrope above town Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City Sky Man Tightrope Roof Town The Encoded Image
- 29. Generating the Poem (Decoding) • React to what we saw • Write a poem word-by-word • Individually propose alternative continuations • Collaboratively select one
- 30. Decoding – Choose First Word I He The High I 4 He 1 The 0 High 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I
- 31. Decoding – Choose Next Word really only walk stand really 3 only 0 walk 0 stand 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really
- 32. Decoding – Choose Next Word knowing walking trotting standing knowing 0 walking 3 trotting 0 standing 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really had an easy way of walking
- 33. Decoding – Speed it up • One word at a time is too slow for humans… • Propose whole phrases (as many words as you like)
- 34. Decoding – Choose Next Phrase However tall I never thought Looking at a small Despite However tall 0 I never thought 1 Looking at a small 3 Despite 1 • Each person proposes 1 phrase • Each person votes (up to 2 votes) for ‘best’ phrase • Cannot vote yourself! • Count votes • Write highest scoring phrase to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really had an easy way of walking Looking at a small
- 37. Image: Man leads caravan through desert The camel holds the hand of the poor man He’ll watch us tread, he’ll watch us fall The long shadow of a donkey across the cracked sand Shabbily clad but standing tall We each of us must go, all Surrounded by another expedition, Across an ocean on the sea of sand, Mountains gaze upon a vast Egyptian, A field of sand beneath the silver strand.
- 38. Image: Man free falls to ground We gotta hide behind the Beaver lake! I wanna know a better place or where, Another day a little kiss and take, An angel on the other side of there. The forest and cliffs standing against the sky A human being in free fall Sailing through the air like a fly Come to me, the grass, a call.
- 39. Image: Crowd watches fishes at aquarium In a box of Nothing far from the deep where waves are thrusting where light goes to sleep Expecting something from an empty zoo, Surrounded by an ocean full of fish, On the other side of me and you, Beneath the carpet like a jellyfish.
- 40. Image: Two men round a campfire A living fire becomes a doubles title. To stay protected by the sons of men, We stuck together like a semi final, The one and two and three or four of ten. Marshmallows at dawn Freshly cut wood burns in the fire Time slips out a wide yawn They all sing out in choir
- 41. Acknowledgements • Zena Edwards, CV:iD • Daniela Paolucci, Apples and Snakes • Sebastian Riedel, UCL • Piotr Mirowski, HumanMachine/Deepmind • Mandana Seyfeddinipur, SOAS • Marjan Ghazvininejad, USC • Generating Topical Poetry, EMNLP 2016
- 42. Thank you!
Editor's Notes
- Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go by. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.
- TOE/STAIRS/AIR/BY
- TOE/STAIRS/AIR/BY
- TOE/STAIRS/AIR/BY
- TOE/STAIRS/AIR/BY
- TOE/STAIRS/AIR/BY