This document summarizes a presentation about representation learning and deep learning. It discusses how machine learning algorithms require data to be represented in a way that captures meaningful features, and how representation learning aims to automatically discover these features from data. It also covers topics like semi-supervised learning, transfer learning, deep neural networks, convolutional neural networks, word embeddings with Word2Vec, and neural style transfer. Examples of representation learning applications include document classification using Doc2Vec embeddings and neural art style transfer.

1. Representation learning
PyData Warsaw 2015
Michael Jamroz
Matthew Opala
24’th september 2015

2. ● Goals of AI
● Learning representations
● Deep learning
● Examples
Presentation Plan

3. AI
● Goal: build the intelligent machine
● It needs knowledge to make decisions
● Impossible to put the knowledge into
computer program
● Knowledge gained by learning from data

4. Data representation
● Representation - features passed to ML
algorithms, crucial for good performance on
various tasks
● Features can be handcrafted or learned
automatically
● Representation learning: discovering
meaningful features by the computer

5. ML in industry nowadays
● Most of the time spent on manual feature
extraction
● We would like to have

6. Why representation learning ?
● Previous slide (time-consuming, incomplete)
● Unsupervised feature learning
○ Collected data are mostly unlabeled
(bigger datasets)
○ Labels do not provide enough information
○ Process of learning is independent of the
ML task performed on data

7. Semi-supervised, transfer learning
● Transfer learning - transferring knowledge
from previous learning to the new machine
learning task
● Semi-supervised learning
few labeled
examples
many unlabeled
examples

8. Need for Deep Architectures
● deep architecture can represent certain
functions more compactly than shallow one
● any boolean function (e. g. AND, OR, XOR)
can be represented by a single hidden layer
- however it may require exponential number
of hidden units

9. Formally
● shown by Yao in 1985 that d-bit parity
circuits of depth 2 have exponential size
● generalised to perceptrons with linear
threshold units in 1991 by Hastad

10. How deep representation
do we need?

11. Informal arguments

12. Shallow program

13. Deep program

14. Biology inspirations

15. Learning multiple levels of
representation

16. “I'm sorry, Dave. I'm afraid I can't do that.”

17. 对不起，戴夫。恐怕我不能这样做。

18. Let’s build deep
representation

19. Multilayer Perceptron
input layer
hidden layers
output layer

20. Reminder - Gradient Descent

21. But MLPs have their problems
● vanishing, exploding gradients
● stucking in poor local optima
● lack of good initializations
● lack of labeled data
● hard time to encourage for research
● slow hardware

22. Breakthrough 2006

23. Greedy layer-wise pretraining

24. Restricted Boltzmann Machine

25. Stacking RBMs

26. ● but for natural images we would like to be
invariant to translations, rotations and other
non-changing class transformations
● fully connected networks do not introduce
such invariance
Limitations of fully connected
networks

27. Convolutional Neural
Nets

28. Convolution = sparse connectivity +
parameters sharing

29. Sparse connectivity

30. Parameter sharing

31. Convolution

32. Pooling

33. Architecture

34. Examples

35. Word2Vec / Doc2Vec
● Tomas Mikolov et al 2013
● Embedding words / documents in vector
space
● Neural network with one hidden layer
● Trained in unsupervised way
● Representation for word obtained by
computing hidden layer activation
● Good explanation: http://arxiv.org/pdf/1411.
2738v1.pdf

37. Problem
● ~180k documents - reports made by
american companies of activity
● companies belonging to different industry
segments (260)
● ~9k labeled documents (given industry the
company operates in)
● example of semi-supervised learning
● task: classify the remaining part of
documents

38. Doc2Vec - document embedding

39. Doc2Vec - classification
● Division of labeled set to training/test data
with ratio 70/30
● Test set: ~2700 examples, 260 classes
● Classification performed on representation
obtained from Doc2Vec
● Accuracy on test set:
○ KNN with voting: ~85 %
○ SVM one-versus-one: ~83 %
○ Random forest: ~80 %

40. Neural Art Style Transfer

44. Pretrain CNN

45. Content representation

46. Art style representation

47. Objective function

48. Summing up
● define loss function for content
● define loss function for art
● define total loss
● perform gradient-based optimization
● compute derivatives with respect to data

50. ● Theano & Lasagne
● NViDIA GTX
● https://github.com/Craftinity/art_style
● http://deeplearning.net

51. Contact
● http://www.craftinity.com
● https://www.facebook.com/craftinitycom
● https://twitter.com/craftinitycom
● mateuszopala@craftinity.com
● michaljamroz@craftinity.com
● contact@craftinity.com

52. Q&A

53. The End