Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Human Neural Machine

767 views

Published on

Introduction to natural language generation with artificial neural networks (ANNs) and a group poetry writing exercise where humans pretend to be neurons in an ANN.

Published in: Education
  • Be the first to comment

Human Neural Machine

  1. 1. Voice of the Machine - Human Neural Network Georgios Spithourakis PhD Candidate, UCL
  2. 2. Part 1 Voice of the Machine
  3. 3. Humans Describe What They See Encoding Decoding ???
  4. 4. Information Processing
  5. 5. Information Processing
  6. 6. Machines Describe What they See Encoding Decoding ???
  7. 7. Encoding an Image • Convolutional Neural Networks (CNNs)
  8. 8. Encoding an Image • Convolutional Neural Networks (CNNs) Layer 1 Layer 2
  9. 9. Encoding an Image • Convolutional Neural Networks (CNNs) Layer 1 Layer 2
  10. 10. Generating Text (decoding) (1) Nude Descending a Staircase, Duchamp, 1912
  11. 11. Generating Text (decoding) (1) Toe upon ___, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  12. 12. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  13. 13. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  14. 14. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go __. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  15. 15. Generating Text (decoding) (1) Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind. We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go by. One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape. Nude Descending a Staircase, Duchamp, 1912 X. J. Kennedy (1961)
  16. 16. • Recurrent Neural Networks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe
  17. 17. • Recurrent Neural Networks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe
  18. 18. • Recurrent Neural Networks (RNNs) Input Output Generating Text (decoding) (2) <START> Toe upon Toe upon toe Aardvark 0.1% . . . Toe 20% . . . Zebra 0.2%
  19. 19. Part 2 Human Neural Network
  20. 20. Exercise Goals • Humans become a machine • Each group is a neural machine • Each individual is a neuron • Adaptation • Communicate in words (not numbers) • More flexibility
  21. 21. Encoding an image • Describe what you see • Pieces of an image • Objects in a scene • A story for the scene
  22. 22. Encoding an Image – Pieces • For each piece, choose 3 words that describe its content Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City
  23. 23. Encoding an Image – Objects (individual) • Group together pieces to identify up to 5 objects • Describe each with 1 word • A story should start forming Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City Sky Man Roof City
  24. 24. Encoding an Image – Objects (group) • Reach an agreement as a group (up to 5 objects, 1 word each) Sky Man Tightrope Roof Town Sky Someone Reflection Town Sky Man Tightrope Roof City
  25. 25. Encoding an Image – Scene/Story • Decide on story of up to 5 words Man walks tightrope above town Sky Man Tightrope Roof Town
  26. 26. Man walks tightrope above town Blue Uniform Empty Sea Blue Calm Sky Cloud Sunny Cloud Sky Stick Sky Cloud Cotton Tightrope Man Balancing Clouds Sky Lines Grey Cloud Tripod City Landscape River Skyscrapers Landscape Horizon Corner Buildings Window Puddle Mirror Pavement Riverside Town Park Buildings Roads Below City Brownish Metal River Brown City Sky Man Tightrope Roof Town The Encoded Image
  27. 27. Generating the Poem (Decoding) • React to what we saw • Write a poem word-by-word • Individually propose alternative continuations • Collaboratively select one
  28. 28. Decoding – Choose First Word I He The High I 4 He 1 The 0 High 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I
  29. 29. Decoding – Choose Next Word really only walk stand really 3 only 0 walk 0 stand 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really
  30. 30. Decoding – Choose Next Word knowing walking trotting standing knowing 0 walking 3 trotting 0 standing 1 • Each person proposes 1 word • Each person votes (up to 2 votes) for ‘best’ word • Cannot vote yourself! • Count votes • Write highest scoring word to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really had an easy way of walking
  31. 31. Decoding – Speed it up • One word at a time is too slow for humans… • Propose whole phrases (as many words as you like)
  32. 32. Decoding – Choose Next Phrase However tall I never thought Looking at a small Despite However tall 0 I never thought 1 Looking at a small 3 Despite 1 • Each person proposes 1 phrase • Each person votes (up to 2 votes) for ‘best’ phrase • Cannot vote yourself! • Count votes • Write highest scoring phrase to poem • If tied, repeat voting only between tied words (or flip coin) POEM I really had an easy way of walking Looking at a small
  33. 33. Part 3 Conclusion
  34. 34. Image: Man leads caravan through desert The camel holds the hand of the poor man He’ll watch us tread, he’ll watch us fall The long shadow of a donkey across the cracked sand Shabbily clad but standing tall We each of us must go, all Surrounded by another expedition, Across an ocean on the sea of sand, Mountains gaze upon a vast Egyptian, A field of sand beneath the silver strand.
  35. 35. Image: Man free falls to ground We gotta hide behind the Beaver lake! I wanna know a better place or where, Another day a little kiss and take, An angel on the other side of there. The forest and cliffs standing against the sky A human being in free fall Sailing through the air like a fly Come to me, the grass, a call.
  36. 36. Image: Crowd watches fishes at aquarium In a box of Nothing far from the deep where waves are thrusting where light goes to sleep Expecting something from an empty zoo, Surrounded by an ocean full of fish, On the other side of me and you, Beneath the carpet like a jellyfish.
  37. 37. Image: Two men round a campfire A living fire becomes a doubles title. To stay protected by the sons of men, We stuck together like a semi final, The one and two and three or four of ten. Marshmallows at dawn Freshly cut wood burns in the fire Time slips out a wide yawn They all sing out in choir
  38. 38. Acknowledgements • Zena Edwards, CV:iD • Daniela Paolucci, Apples and Snakes • Sebastian Riedel, UCL • Piotr Mirowski, HumanMachine/Deepmind • Mandana Seyfeddinipur, SOAS • Marjan Ghazvininejad, USC • Generating Topical Poetry, EMNLP 2016
  39. 39. Thank you!

×