Theano is a Python library that allows defining, optimizing, and evaluating mathematical expressions involving multi-dimensional arrays efficiently. It can compile expressions into optimized C code for fast CPU and GPU execution. Theano uses symbolic differentiation to automatically compute gradients for neural network training via backpropagation. It represents computations as a graph with variable nodes and operation nodes. This graph can be optimized before generating efficient C code. Theano is useful for machine learning algorithms that require large-scale numeric optimization like neural networks. The document discusses implementing word embedding models in Theano including autoencoder, GloVe, and skip-gram negative sampling models. Code examples are provided in GitHub links.

1. Introduction to theano
Case study of word embedding models
Shashank
Gupta
MS, SIEL IIIT-H

2. Numpy is fast, why theano?
Bound by CPU
Bound by Python interpreter (less scope of runtime optimization like lazy
evaluation etc.)
Lack of symbolic differentiation
Finite difference method exists, but prone to numerical errors
SymPy supports symbolic differentiation but expression graphs not optimized as
theano

3. What’s theano?
Math expression compiler
Compiles a math expression to optimized C code
Targeted for GPU (CPU if GPU not supported)
Strongly typed as compared to python’s dynamically typed system
Works as GPU metaprogrammer (abstraction on top of GPU programming)

4. Alternative to theano
CUDA (PyCUDA it’s python wrapper)
Asynchronous Stochastic Gradient Descent - smart way to parallelize SGD
Main idea - Don’t create lock for one update, let threads run in async.
Could speed up ML algos more than matrix-vector parallelization

5. Automatic Differentiation
Crude way - Finite difference method
df / dx = (f(x + h) - f(x - h)) / 2*h
Prone to coding bugs
Prone to numerical error

6. Symbolic differentiation
Calculate gradient analytically
For a given math expression construct symbolic computation graph
Ref:
http://cs231n.github.io/optimization-2/

7. Backprop using symbolic differentiation
Once graph is computed forward pass is just flow of input to compute output
Backprop is going backwards from last node to compute gradients locally at each
node and accumulate those local gradients to compute global gradient (using
chain rule)
Ex :

8. Cont.
Each entity in the computation graph is a symbolic variable (a node in
computational graph)
Each operation is an ‘op’ node (theano specific)
‘Op’ node takes some inputs and produces some output
‘Apply’ node which applies op to inputs
‘Type’ nodes with associated type information of symbolic variable

9. Cont.
This is kind of abstract representation of math expression
Can think as intermediate representation in standard compiler phase
This is how theano represents the computational graph internally

10. Cont.
Ref: http://deeplearning.net/software/theano/extending/graphstructures.html

11. Optimization
Once this ‘intermediate representation’ is generated theano optimizes this graph
for efficient computation
Can think of it as compiler optimization phase
It reorders, remove redundant expression etc. to generate an ‘equivalent’ graph
which gives same output as input

12. Shared variables
Symbolic variables with predefined values
In Machine learning used to define parameters of the models with pre-defined
values (W, b matrices in NN)
Sends these values to host GPU with optimised storage

13. Theano functions
Compiles the ‘abstract’ computation graph to optimised C code
Compiles it targeting GPU
C code is highly optimized for numeric computation
Sort of interface between theano code and it’s calling python code

14. Case study : Word embedding models
1st Model : Autoencoder based word embedding
Ref : http://arxiv.org/pdf/1412.4930v2.pdf

15. Cont.
U is the embedding matrix which is learned by optimizing this objective

16. Code
Refer to github gist :
https://gist.github.com/shashankg7/aec2303803e7b39b150a9f78cb59db09
Only theano part included, I/O and preprocessing omitted

17. Model 2 - GloVe word embedding
Optimizes squared loss function
Challenges in practical implementation in theano
Main thing to remember - VECTORIZED implementation
How to handle large matrices
Refer to github gist :
https://gist.github.com/shashankg7/aec2303803e7b39b150a9f78cb59db09

18. Model 3 : Skip-gram negative sampling
Alpha stage: Not able to figure out vectorization of loss function
Refer to github gist :
https://gist.github.com/shashankg7/aec2303803e7b39b150a9f78cb59db09

19. THANKS !!!