2017-08-05 @ Tokyo Web Mining

1. 2017-08-05 @ Tokyo Web Mining

2. Yuta Kashino ( )
BakFoo, Inc. CEO
Astro Physics /Observational Cosmology
Zope / Python
Realtime Data Platform for Enterprise / Prototyping

3. Yuta Kashino ( )
arXiv
stat.ML, stat.TH, cs.CV, cs.CL, cs.LG
math-ph, astro-ph
PyCon2016
@yutakashino
https://www.slideshare.net/yutakashino/pyconjp2016

4. -
-
-

6. http://bayesiandeeplearning.org/

7. Shakir Mohamed
http://blog.shakirm.com/wp-content/uploads/2015/11/CSML_BayesDeep.pdf

8. -
Denker, Schwartz, Wittner, Solla, Howard, Jackel, Hopfield (1987)
Denker and LeCun (1991)
MacKay (1992)
Hinton and van Camp (1993)
Neal (1995)
Barber and Bishop (1998)
Graves (2011)
Blundell, Cornebise, Kavukcuoglu, and Wierstra (2015)
Hernandez-Lobato and Adam (2015)

9. -
Yarin Gal
Zoubin Ghahramani
Shakir Mohamed
Dastin Tran
Rajesh Ranganath
David Blei
Ian Goodfellow
Columbia U
U of Cambridge

10. -
- :
- :
- :
- :
- : SGD + BackProp
…
…x1 x2 xd
✓(2)
✓(1)
x
y
y(n)
=
X
j
✓
(2)
j (
X
i
✓
(1)
ji x
(n)
i ) + ✏(n)
p(y(n)
| x(n)
, ✓) = (
X
i
✓
(n)
i x
(n)
i )
✓
D = {x(n)
, y(n)
}N
n=1 = (X, y)

11. :
- +
- 2012 ILSVRC
→ 2015
-
-
-
-

12. :
-
- ReLU, DropOut, Mini Batch, SGD(Adam), LSTM…
-
- ImageNet, MSCoCo…
- : GPU,
- :
- Theano, Torch, Caffe, TensorFlow, Chainer, MxNet, PyTorch…

13. :
-
-
-
-
-
https://lossfunctions.tumblr.com/

14. :
-
-
-
- Adversarial examples
-

15. -
=
=
-

16. -
- :
- :
- :
- :
- : SGD + BackProp
…
…x1 x2 xd
✓(2)
✓(1)
x
y
y(n)
=
X
j
✓
(2)
j (
X
i
✓
(1)
ji x
(n)
i ) + ✏(n)
p(y(n)
| x(n)
, ✓) = (
X
i
✓
(n)
i x
(n)
i )
✓
D = {x(n)
, y(n)
}N
n=1 = (X, y)

17. - data hypothesis
-
-
P(H | D) =
P(H)P(D | H)
P
H P(H)P(D|H)
P(x) =
X
y
P(x, y)
P(x, y) = P(x)P(y | x)

18. - :
- :
-
-
- :
P(H | D) =
P(H)P(D | H)
P
H P(H)P(D|H)
likelihood priorposterior
P(✓ | D, m) =
P(D | ✓, m)P(✓ | m)
P(D | m)
m:
P(x | D, m) =
Z
P(x | ✓, D, m)P(✓ | D, m)d✓
P(m | D) =
P(D | m)P(m)
P(D)

19. -
- :
- :
- :
- :
…
…x1 x2 xd
✓(2)
✓(1)
x
y
✓
D = {x(n)
, y(n)
}N
n=1 = (X, y)
P(✓ | D, m) =
P(D | ✓, m)P(✓ | m)
P(D | m)
m:
P(x | D, m) =
Z
P(x | ✓, D, m)P(✓ | D, m)d✓
prior

20. - (Variational Bayes)
- (MCMC)
P(✓ | D, m) =
P(D | ✓, m)P(✓ | m)
P(D | m)
m:

21. p(θ|D) KL q(θ)
ELBO
⇤
= argmin KL(q(✓; ) || p(✓ | D))
= argmin Eq(✓; )[logq(✓; ) p(✓ | D)]
ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )]
⇤
= argmax ELBO( )

22. - VI
-
- David MacKay “Lecture 14 of the Cambridge Course”
- PRML 10
http://www.inference.org.uk/itprnn_lectures/

23. - KL =ELBO
q(✓; 1)
q(✓; 5)
p(✓, D) p(✓, D)
✓✓
⇤
= argmax ELBO( )
ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )]

24. - p q
- :
- ADVI: Automatic Differentiation Variational Inference
- BBVI: Blackbox Variational Inference
q(✓; 1)
q(✓; 5)
p(✓, D) p(✓, D)
✓✓
arxiv:1603.00788
arxiv:1401.0118

25. Reference
- Zoubin Ghahramani “History of Bayesian neural
networks” NIPS 2016 Workshop Bayesian Deep
Learning
- Yarin Gal “Bayesian Deep Learning"O'Reilly
Artificial Intelligence in New York, 2017

27. - Probabilistic Programing Library/Langage
- Stan, PyMC3, Anglican, Church, Venture,Figaro, WebPPL,
Edward
- : Edward / PyMC3
- (VI)
Metropolis Hastings
Hamilton Monte Carlo
Stochastic Gradient Langevin Dynamics
No-U-Turn Sampler
Blackbox Variational Inference
Automatic Differentiation Variational Inference

28. Edward

29. Edward
- Dustin Tran (Open AI)
- Blei Lab
- (PPL)
- Stan, PyMC3, Anglican, Church, Venture,Figaro, WebPPL
- 2016 2 PPL
- TensorFlow
- George Edward Pelham Box
Box-Cox Trans., Box-Jenkins, Ljung-Box test box plot Tukey,
3 2 RA Fisher

30. PPL
Edward
TensorFlow(TF) + (PPL)
TF:
PPL: + +
Python/Numpy

31. 1. TF:
-
- :

32. 1. TF:

33. 1. TF:
-
-
- GPU / TPU
Inception v3 Inception v4

34. 1. TF:
- Keras, Slim
- TensorBoard

35. 2.
x:
edward
x⇤
s P(x | ↵)
✓⇤
⇠ Beta(✓ | 1, 1)

36. 2.
- ( )
Edward
p(x, ✓) = Beta(✓ | 1, 1)
50Y
n=1
Bernoulli(xn | ✓),

37. 2.
-
log_prob()
-
mean()
-
sample()

38. 3.
Edward TF

39. 3.
256 28*28

40. 4.
X, Z Z
- (Variational Bayes)
- (MCMC)
p(z | x) =
p(x, z)
R
p(x, z)dz
.

41. 4.

42. 4.
p(z|x) KL q(z)
ELBO

43. 4.
Edward KLqp

44. 5. Box’s loop
George Edward Pelham Box
Blei 2014

45. 5. Box’s loop

46. Edward
- Edward = TensorFlow + +
- TensorFlow
-
- TF GPU, TPU, TensorBoard, Keras
-
- Box’s Loop
- Python

48. Refrence
•D. Tran, A. Kucukelbir, A. Dieng, M. Rudolph, D. Liang, and
D.M. Blei. Edward: A library for probabilistic modeling,
inference, and criticism.(arXiv preprint arXiv:1610.09787)
•D. Tran, M.D. Hoffman, R.A. Saurous, E. Brevdo, K. Murphy,
and D.M. Blei. Deep probabilistic programming.(arXiv
preprint arXiv:1701.03757)
•Box, G. E. (1976). Science and statistics. (Journal of the
American Statistical Association, 71(356), 791–799.)
•D.M. Blei. Build, Compute, Critique, Repeat: Data Analysis
with Latent Variable Models. (Annual Review of Statistics
and Its Application Volume 1, 2014)

50. Dropout
- Yarin Gal ”Uncertainty in Deep Learning”
- Dropout
- Dropout : conv
- LeNet with Dropout http://mlg.eng.cam.ac.uk/yarin/blog_2248.html

51. Dropout
- LeNet DNN
- conv Dropout MNIST

52. Dropout
- CO2

53. -
-
-

54. Questions
kashino@bakfoo.com
@yutakashino

55. BakFoo, Inc.
NHK NMAPS: +

56. BakFoo, Inc.
PyConJP 2015
Python

57. BakFoo, Inc.

58. BakFoo, Inc.
: SNS +