Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Creative AI & multimodality: looking ahead

2,391 views

Published on

Lecture on Creative AI (Generative Deep Learning) at Imperial College London, 1 December 2015

  • Be the first to comment

Creative AI & multimodality: looking ahead

  1. 1. Creative AI & multimodality: looking ahead Roelof Pieters @graphific Imperial College London, 
 1 Dec 2015 roelof@graph-technologies.comhttp://artificialexperience.com/http://www.csc.kth.se/~roelof/
  2. 2. AICreative
  3. 3. AI I kinda expect the audience to know AI & Machine Learning
 Let’s move on shall we ?
  4. 4. AI All references to: - Arxiv or - GitXiv if the “code” or “dataset” is available Collaborative Open Computer Science more info (Medium)
  5. 5. AI > today’s focus
  6. 6. AI > today’s focus
  7. 7. “Deep learning is a set of algorithms in machine learning that attempt to learn in multiple levels, corresponding to different levels of abstraction.”
  8. 8. AI > today’s focus use of several modes (media) to create a single artifact. Multimodality “Mode” Socially and culturally shaped resource for making meaning. — Gunther Kress
  9. 9. Creativity
  10. 10. Creativity • Many definitions: philosophical, sociological, historical, practical
  11. 11. Creativity 1. Making unfamiliar combinations of familiar ideas. 2. Explore a structured conceptual space 3. (Radically) transforming ones structured conceptual space “Exploration” “Remix” “The Creative Mind”
 — Margaret Boden “Transformation”
  12. 12. • Skill • Appreciation • Imagination • Learning • Innovation • Accountability, • Subjectivity • Intentionality. Creativity > “Traits” software has to exhibit in order to avoid easy criticism of being “non-creative”. (Simon Colton)
  13. 13. • Skill • Appreciation • Imagination • Learning • Innovation • Accountability, • Subjectivity • Intentionality Creativity > software traits
  14. 14. • Skill • Appreciation • Imagination • Learning • Innovation • Accountability, • Subjectivity • Intentionality Creativity > software traits
  15. 15. • Skill • Appreciation • Imagination • Learning • Innovation • Accountability, • Subjectivity • Intentionality Creativity > software traits
  16. 16. • Skill • Appreciation • Imagination • Learning • Innovation • Accountability, • Subjectivity • Intentionality Creativity > software traits
  17. 17. • Skill • Appreciation • Imagination • Learning • Innovation • Accountability, • Subjectivity • Intentionality Creativity > software traits
  18. 18. • Skill • Appreciation • Imagination • Learning • Innovation • Accountability, • Subjectivity • Intentionality Creativity > software traits
  19. 19. • Skill • Appreciation • Imagination • Learning • Innovation • Accountability, • Subjectivity • Intentionality Creativity > software traits
  20. 20. • Skill • Appreciation • Imagination • Learning • Innovation • Accountability, • Subjectivity • Intentionality Creativity > software traits
  21. 21. AICreative
  22. 22. Creative AI > Current possibilities • Appropriating “standard” nets for creative use • Reinforcement Learning: Creativity as a Game • RNNs/LSTMs/GRUs • Sequence-to-Sequence: Creativity as a Translation Task • Auto-Encoders • Attention-based Models • Generative Adversarial Nets
  23. 23. Creative AI > Current possibilities • Appropriating “standard” nets for creative use • Reinforcement Learning: Creativity as a Game • RNNs/LSTMs/GRUs • Sequence-to-Sequence: Creativity as a Translation Task • Auto-Encoders • Attention-based Models • Generative Adversarial Nets
  24. 24. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream see also: www.csc.kth.se/~roelof/deepdream/
  25. 25. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream see also: www.csc.kth.se/~roelof/deepdream/ codeyoutubeRoelof Pieters 2015
  26. 26. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Deep Dream see also: www.csc.kth.se/~roelof/deepdream/ C.M.Kosemen & 
 Roelof Pieters (2015) Gizmodo
  27. 27. Creative AI > Current possibilities > Appropriating “standard” nets for creative use Leon A. Gatys, Alexander S. Ecker, Matthias Bethge , 2015. 
 A Neural Algorithm of Artistic Style (GitXiv) Style Net
  28. 28. Gene Kogan, 2015. Why is a Raven Like a Writing Desk? (vimeo)
  29. 29. Creative AI > Current possibilities • Appropriating “standard” nets for creative use • Reinforcement Learning: Creativity as a Game • RNNs/LSTMs/GRUs • Sequence-to-Sequence: Creativity as a Translation Task • Auto-Encoders • Attention-based Models • Generative Adversarial Nets
  30. 30. Creative AI > Current possibilities > Reinforcement Learning • AMN: Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov 2015, Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning (arxiv) • DQN: Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforcement learning. Nature, 518(7540): 529–533, 2015.
  31. 31. Creative AI > Current possibilities > Reinforcement Learning Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente, 2015 
 Multiagent Cooperation and Competition with Deep Reinforcement Learning (GitXiv) (YouTube)
  32. 32. Reinforcement Learning Ning Xie, Hirotaka Hachiya, Masashi Sugiyama, 2013 , 
 Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting (Paper, Lecture, YouTube)
  33. 33. (YouTube)
  34. 34. Ning Xie, Hirotaka Hachiya, Masashi Sugiyama, 2013
 Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting (Paper, Lecture, YouTube)
  35. 35. Creative AI > Current possibilities • Appropriating “standard” nets for creative use • Reinforcement Learning: Creativity as a Game • RNNs/LSTMs/GRUs • Sequence-to-Sequence: Creativity as a Translation Task • Auto-Encoders • Attention-based Models • Generative Adversarial Nets
  36. 36. Creative AI > Current possibilities • Appropriating “standard” nets for creative use • Reinforcement Learning: Creativity as a Game • RNNs/LSTMs/GRUs • Sequence-to-Sequence: Creativity as a Translation Task • Auto-Encoders • Attention-based Models • Generative Adversarial Nets
  37. 37. Creative AI > Current possibilities • Appropriating “standard” nets for creative use • Reinforcement Learning: Creativity as a Game • RNNs/LSTMs/GRUs • Sequence-to-Sequence: Creativity as a Translation Task • Auto-encoders • Attention-based Models • Generative Adversarial Nets
  38. 38. Creative AI > Current possibilities • Standard (“denoising”) Autoencoders • Variational Autoencoder (VAE) / Stochastic Gradient VB • Deep Convolutional Inverse Graphics Network • Variational RNN (VRNN) Vincent et al, 2010. Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion (paper) (code)
  39. 39. Creative AI > Current possibilities • Standard “denoising” Autoencoders • Variational Autoencoder (VAE) / Stochastic Gradient VB • Deep Convolutional Inverse Graphics Network • Variational RNN (VRNN) • Diederik P Kingma, Max Welling, 2013. 
 Auto-Encoding Variational Bayes (GitXiv)
  40. 40. Creative AI > Current possibilities • Standard “denoising” Autoencoders • Variational Autoencoder (VAE) • Deep Convolutional Inverse Graphics Network (modified VAE) • Variational RNN (VRNN) Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, Joshua B. Tenenbaum, 2015 Deep Convolutional Inverse Graphics Network (GitXiv)
  41. 41. Creative AI > Current possibilities • Standard “denoising” Autoencoders • Variational Autoencoder (VAE) • Deep Convolutional Inverse Graphics Network • Variational RNN (VRNN) (VAE at every time step) Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio, 2015
 A Recurrent Latent Variable Model for Sequential Data (GitXiv) VAEVAEVAE
  42. 42. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio , 2015. 
 A Recurrent Latent Variable Model for Sequential Data (GitXiv) (Audio Samples)
  43. 43. Creative AI > Current possibilities • Appropriating “standard” nets for creative use • Reinforcement Learning: Creativity as a Game • RNNs/LSTMs/GRUs • Sequence-to-Sequence: Creativity as a Translation Task • Auto-Encoders • Attention-based Models • Generative Adversarial Nets
  44. 44. Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra, 2015
 DRAW: A Recurrent Neural Network For Image Generation (GitXiv) Variational Auto-Encoder Deep Recurrent Attentive Writer (DRAW) Network
  45. 45. (YouTube)
  46. 46. Creative AI > Current possibilities • Appropriating “standard” nets for creative use • Reinforcement Learning: Creativity as a Game • RNNs/LSTMs/GRUs • Sequence-to-Sequence: Creativity as a Translation Task • Auto-Encoders • Attention-based Models • Generative Adverserial Nets
  47. 47. Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus, 2015. 
 Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (GitXiv)
  48. 48. Alec Radford, Luke Metz, Soumith Chintala , 2015. 
 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
  49. 49. Alec Radford, Luke Metz, Soumith Chintala , 2015. 
 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
  50. 50. ”turn” vector created from four averaged samples of faces looking left vs looking right. Alec Radford, Luke Metz, Soumith Chintala , 2015. 
 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (GitXiv)
  51. 51. walking through the manifold
  52. 52. top: unmodified samples bottom: same samples dropping out ”window” filters
  53. 53. Autonomy Supervision Creativity? - unsupervised training - generator/discrimator - latent/z space - auto encoders - multimodality - query - target/class
  54. 54. Creativity? Process Result
  55. 55. Creative AI > Needs as I see it Creative AI as a “tool”
 or “brush” to paint with
  56. 56. A system which marries the need for a creative process with the need for a creative output • with as less human input as possible (data) • with its own style • with the possibility for human level supervision for rapid experimentation Creative AI > a “brush”
  57. 57. A system which marries the need for a creative process with the need for a creative output • with as less human input as possible ( ) • with its own style • with the possibility for human level supervision for rapid experimentation Creative AI > a “brush” data
  58. 58. Creative AI > a “brush” > data • reuse nets as much as possible • combining unsupervised & supervised • multiple modalities • plug in external knowledge bases
  59. 59. Creative AI > a “brush” > data input • unlabeled & labeled data • external knowledge bases (dbpedia, wikipedia) • one-shot learning • zero-shot learning Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
 Zero-Shot Learning Through Cross-Modal Transfer a zero-shot model that can predict both seen and unseen classes
  60. 60. Creative AI > a “brush” > data input Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
 Zero-Shot Learning Through Cross-Modal Transfer (slides)
  61. 61. Creative AI > a “brush” > data input Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
 Zero-Shot Learning Through Cross-Modal Transfer (slides)
  62. 62. Creative AI > a “brush” > data input Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng, 2013
 Zero-Shot Learning Through Cross-Modal Transfer (slides)
  63. 63. A system which marries the need for a creative process with the need for a creative output • with as less human input as possible (data) • with its own style • with the possibility for human level 
 for rapid experimentation Creative AI > a “brush” supervision
  64. 64. Creative AI > a “brush” > data • “rich” latent (“z”) space • easy user supervision over output: • priors • constrain network (units, layers, etc) • guided input • mixed input • latent space
  65. 65. Creative AI > a “brush” > data • “rich” latent (“z”) space • easy user supervision over output: • priors • constrain network (units, layers, etc) • guided input • mixed input • latent space
  66. 66. Creative AI > a “brush” > data Deep Dream Alexander Mordvintsev, Christopher Olah, Mike Tyka, 2015. 
 Inceptionism: Going Deeper into Neural Networks Google Research Blog
  67. 67. Creative AI > a “brush” > data Deep Dream Roelof Pieters, 2015 DeepDream - Class visualization Experiment (link)
  68. 68. Roelof Pieters, 2015 DeepDream - Class visualization Experiment (link)
  69. 69. Creative AI > a “brush” > data • “rich” latent (“z”) space • easy user supervision over output: • priors • constrain network (units, layers, etc) • guided input • mixed input • latent space
  70. 70. Creative AI > a “brush” > data Deep Dream Roelof Pieters, 2015 DeepDream - Overview of standard bvlc googlenet (inception) layers (link) Constrain Layers
  71. 71. Creative AI > a “brush” > data Deep Dream Roelof Pieters, 2015 Single Unit Activations (early layer) (Flickr Album) Constrain Units
  72. 72. Creative AI > a “brush” > data • “rich” latent (“z”) space • easy user supervision over output: • priors • constrain network (units, layers, etc) • guided input • mixed input • latent space
  73. 73. Creative AI > a “brush” > data Deep Dream Roelof Pieters, 2015 DeepDream Video (GitHub)
  74. 74. Creative AI > a “brush” > data • “rich” latent (“z”) space • easy user supervision over output: • priors • constrain network (units, layers, etc) • guided input • mixed input • latent space
  75. 75. Creative AI > a “brush” > data Style Net Roelof Pieters (graphific) (tweet) Roelof Pieters (graphific) (tweet)
  76. 76. Creative AI > a “brush” > data • “rich” latent (“z”) space • easy user supervision over output: • priors • constrain network (units, layers, etc) • guided input • mixed input • latent space
  77. 77. Image -> Text “A person riding a motorcycle on a dirt road.”???
  78. 78. Image -> Text “Two hockey players are fighting over the puck.”???
  79. 79. Image -> Text Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (arxiv) (info) (code) Andrej Karpathy Li Fei-Fei , 2015. 
 Deep Visual-Semantic Alignments for Generating Image Descriptions (pdf) (info) (code) Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan , 2015. Show and Tell: A Neural Image Caption Generator (arxiv)
  80. 80. Text -> Image “A stop sign is flying in blue skies.” “A herd of elephants flying in the blue skies.” Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015. Generating Images from Captions with Attention (arxiv) (examples)
  81. 81. Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov, 2015. Generating Images from Captions with Attention (arxiv) (examples) Text -> Image
  82. 82. Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko , 2015. Sequence to Sequence -- Video to Text (GitXiv) Video -> Text
  83. 83. A system which marries the need for a creative process with the need for a creative output • with as less human input as possible (data) • with its own style • with the possibility for human level supervision for 
 Creative AI > a “brush” rapid experimentation
  84. 84. Creative AI > a “brush” > rapid experimentation
  85. 85. Widening Deepening Tianqi Chen, Ian Goodfellow, Jonathon Shlens, 2015. Net2Net: Accelerating Learning via Knowledge Transfer (arxiv) / code (torch) Reusing Nets: Bigger Net
  86. 86. Teacher and Student net Hint training Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio, 2014. FitNets: Hints for Thin Deep Nets (arxiv) Knowledge distillation SVHN Error MNIST Error Reusing Nets: Smaller Net
  87. 87. Hashed Net Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, Yixin Chen, 2015. Compressing Neural Networks with the Hashing Trick (arxiv) Shrinking Nets: Hashing
  88. 88. Song Han, Huizi Mao, William J. Dally, 2015. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding (arxiv) Shrinking Nets: Pruning, Quantization & Huffman coding
  89. 89. Creative AI > a “brush” > rapid experimentation • experiments need “tooling”, specialised design software to • try things • explore latent spaces (z-space) • push the AI in the right direction • be surprised by AI
  90. 90. Creative AI > a “brush” > rapid experimentation human-machine collaboration
  91. 91. Creative AI > a “brush” > rapid experimentation (YouTube, Paper)
  92. 92. Creative AI > a “brush” > rapid experimentation (YouTube, Paper)
  93. 93. Creative AI > a “brush” > rapid experimentation (Vimeo, Paper)
  94. 94. Creative AI > a “brush” > rapid experimentation • Advertising and marketing • Architecture • Crafts • Design: product, graphic and fashion design • Film, TV, video, radio and photography • IT, software and computer services • Publishing • Museums, galleries and libraries • Music, performing and visual arts
  95. 95. Questions? love letters? existential dilemma’s? academic questions? gifts? find me at:
 www.csc.kth.se/~roelof/ roelof@kth.se

×