Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

alphablues - ML applied to text and image in chat bots

566 views

Published on

Machine Learning Estonia Meetup 28.02.2017

Published in: Data & Analytics
  • Be the first to comment

alphablues - ML applied to text and image in chat bots

  1. 1. 1 Indrek  Vainu &  Hendrik  Luuk ML applied  to  text and  image in  chat  bots
  2. 2. 2 Transform  customer  service into  a  truly   competitive  advantage Utilize  AI to  automate, personalize and scale  business  processes
  3. 3. 3 Chat  bots  are  intelligent  robots. They  are  able  to infer what  you  want. As  a  result,  they  respond with  appropriate   actions.  They  operate  autonomously i.e.  without human  supervision.
  4. 4. 4 Almost  everybody  uses  Facebook  Messenger  – growing  fast  to  1  billion  people.
  5. 5. 5 Apps  and  call  centers replaced   by companies’  chat  bots in  Messenger/Twitter  
  6. 6. 6
  7. 7. 7 Short Bio • >15y of programming, >10y research lab • neuroscience PhD (2009, UT) • Published papers in bioinformatics, neuroanatomy, general linguistics • full-stack engineer + data scientist • ML expertise • RoR, javascript, R, C++, python
  8. 8. 8 Applications of  ML • Image similarity • Video encoding and reconstruction • Intent prediction from text
  9. 9. 9 Image similarity Reference   image
  10. 10. 10 Image similarity AlphaBlues ResNet
  11. 11. 11 Improved cost  function  for  reconstruction • Autoencoder (AE) combined with Generative Adversarial Net (GAN) is good at forming latent representations for accurate reconstruction of image content Autoencoding beyond  pixels  using  a  learned  similarity  metric https://arxiv.org/pdf/1512.09300.pdf
  12. 12. 12 Improved cost  function  for  reconstruction Autoencoding beyond  pixels  using  a  learned  similarity  metric https://arxiv.org/pdf/1512.09300.pdf
  13. 13. 13 Improved cost  function  for  reconstruction Autoencoding beyond  pixels  using  a  learned  similarity  metric https://arxiv.org/pdf/1512.09300.pdf
  14. 14. 14 Improved cost  function  for  reconstruction Autoencoding beyond  pixels  using  a  learned  similarity  metric https://arxiv.org/pdf/1512.09300.pdf
  15. 15. 15 Improved cost  function  for  reconstruction Autoencoding beyond  pixels  using  a  learned  similarity  metric https://arxiv.org/pdf/1512.09300.pdf
  16. 16. 16 Improved cost  function  for  reconstruction Autoencoding beyond  pixels  using  a  learned  similarity  metric https://arxiv.org/pdf/1512.09300.pdf
  17. 17. 17 Improved cost  function  for  reconstruction Autoencoding beyond  pixels  using  a  learned  similarity  metric https://arxiv.org/pdf/1512.09300.pdf
  18. 18. 18 Preserving  covariance structure  in  video • Why preserve covariance? • Is L2 the best objective? INPUT RECONSTRUCTION
  19. 19. 19 Preserving  covariance structure • L2-norm (Euclidean distance) • Covariance-based alternative L2  =  sum((x-­‐y)2) dL2x =  2  *  (x-­‐y) CovCost =  sum((x*xT – y*yT)2) dCovCostx =  sum(4  *  x  *  (x*xT – y*yT),  1)
  20. 20. 20 Preserving  covariance  structure L2 CovCost -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 42.043.044.045.0 Cost function xn cost -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 -3-112 Empirical derivative wrt x[1] xn dcost -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 -3-112 Analytical derivative wrt x[1] xn fdcost -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 236240244248 Cost function xn cost -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 -100515 Empirical derivative wrt x[1] xn dcost -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 -100515 Analytical derivative wrt x[1] xn fdcost target  =  0.76 target  =  0.76
  21. 21. 21 Preserving  covariance  structure Original CovCost L2
  22. 22. 22 Preserving  covariance  structure Original CovCost L2
  23. 23. 23 Preserving  covariance  structure Original CovCost L2
  24. 24. 24 Preserving  covariance  structure Original CovCost L2
  25. 25. 25 Create your  own  objective function • intuition à equation • Minimize the difference between prediction and target • L(f(X), Y) à real number • X – input • f(X) – output from the model • Y – target • L – objective function
  26. 26. 26 Create your  own  objective function • Figure out the derivative using sympy
  27. 27. 27 Create your  own  objective function • Implement on your favourite platform
  28. 28. 28 Create your  own  objective function • Plug in and enjoy! CovCost CovCos t L2 L2
  29. 29. 29 Create your  own  objective function • Benchmark CovSq OriginalL2
  30. 30. 30 Create your  own  objective function • Finally, optimize. O(n2 ) O(1002) 28x28  pixels 28x28  pixels
  31. 31. 31 Automated chat solutions
  32. 32. 32 Basic chat  bot
  33. 33. 33 Extending  memory of  RNN • Sequence length == # of layers in unfolded RNN • Vanishing gradient degrades memory A A A A
  34. 34. 34 Extending  memory of  RNN • What is a good activation function? • little saturation (mostly non-zero gradient) • mean activation 0 (faster convergence) Batch  Normalization:  Accelerating  Deep  Network  Training  by  Reducing  Internal  Covariate  Shift https://arxiv.org/pdf/1502.03167.pdf FAST  AND  ACCURATE  DEEP  NETWORK  LEARNING  BY  EXPONENTIAL  LINEAR  UNITS  (ELUS)   https://arxiv.org/pdf/1511.07289.pdf
  35. 35. 35 Extending  memory of  RNN -4 -2 0 2 4 -1.00.00.51.0 fun(x) x y -4 -2 0 2 4 0.00.40.8 Empirical derivative wrt x x edy -4 -2 0 2 4 0.00.40.8 Analytical derivative wrt x x dy TANH ELU -4 -2 0 2 4 -101234 fun(x) x y -4 -2 0 2 4 0.00.40.8 Empirical derivative wrt x x edy -4 -2 0 2 4 0.00.40.8 Analytical derivative wrt x x dy -4 -2 0 2 4 01234 fun(x) x y -4 -2 0 2 4 0.00.40.8 Empirical derivative wrt x x edy -4 -2 0 2 4 0.00.40.8 Analytical derivative wrt x x dy RELU
  36. 36. 36 Extending  memory of  RNN • Does it matter? • sRNN convergence probability wrt length of sequence Learning  long  term  dependencies  with  gradient  descent  is  difficult.  Bengio et  al.  1994 http://www.dsi.unifi.it/~paolo/ps/tnn-­‐94-­‐gradient.pdf
  37. 37. 37 Finally  finished hl@alphablues.com      |      @alpha_blues      |      www.alphablues.com

×