TensorFlow User Group Meetup - ベイズ分科会

1. tfp.sts
( )

2. Yuta Kashino ( )
❖ BakFoo, Inc. CEO
❖
❖ Kubernetes /
❖ POC / →
❖
❖ k8s, Web ,
❖ Zope 3 / Python 2.1
❖ Zope Japan
❖ Yahoo! Japan
❖ (~20 )
❖
❖ @yutakashino

3. ❖ tfp
❖
❖
❖ VI/MCMC
❖ CO2

4. tfp.sts
❖ Structural Time Series
❖

5. ❖
❖
❖
❖
|
t-2
|
t-1
|
t
|
t+1
|
t+2

6. ❖
❖
X0
P0
Xk-1
Pk-1
Xk
p = A Xk-1 + B uk + wk
Pk
p = A Pk-1 AT + Qk
Yk = C Xk + zk
K = Pk
p H / (H Pk
p HT + R)
Xk = Xk
p + K (Yk - Hkp )
Xk
Pk = ( I - KH ) Pk
p
Xk
Pk
|
k-1
|
k
k -> k-1
( )
uk :
wk :
Qk :
Yk :
zk :
K :
R :
X :
P :
G(X, P)

7. ❖ Michel van Biezen (Raytheon)
❖ https://www.youtube.com/playlist?list=PLX2gX-
ftPVXU3oUFNATxGXY90AULiqnWT
❖ 27

8. (STS)
❖ Structual Time Series Model
❖
❖
❖ : ( )
|
t-2
|
t-1
|
t
|
t+1
|
t+2

9. TFP VI: tfp.vi
❖ sts model
❖ sts.LocalLinearTrend
❖ sts.Sum([trend, seasonal]…)
❖ sts.build_factored_surrogate_posterier
❖ arg: model
❖ vi.fit_surrogate_porterier
❖ arg: model
❖ arg: surrogate_posterier
❖ arg: num_step
❖ sts.forecast

10. sts model
❖ sts.LocalLinearTrend
❖ sts.Sum([trend, seasonal]…)

11. sts.LocalLinearTrend
❖ level, scope
LocalLinearTrend
StructuralTimeSeries
LocalLenearTrendSpaceSteteModel
LinearGaussianStateSpaceModel

12. tfd.LinearGaussianStateSpaceModel
❖
❖ https://www.tensorflow.org/probability/api_docs/python/tfp/
distributions/LinearGaussianStateSpaceModel?hl=ja

13. sts.Sum([trend, seasonal]…)
❖ sts model
|
t-1
|
t
|
t+1

14. sts.build_factored_surrogate_posterier
❖ sts.build_factored_surrogate_posterier
❖ return: variational_posterior
(tfd.JointDistributionNamed)
❖ ↑
❖ q

15. •- data hypothesis( )
- :
-
-
P(H | D) =
P(H)P(D | H)
P
H P(H)P(D|H)
P(x) =
X
y
P(x, y)
P(x, y) = P(x)P(y | x)
posterior likelihoodprior
evidence

16. - :
- :
-
-
- :
P(H | D) =
P(H)P(D | H)
P
H P(H)P(D|H)
likelihood priorposterior
P(✓ | D, m) =
P(D | ✓, m)P(✓ | m)
P(D | m)
m:
P(x | D, m) =
Z
P(x | ✓, D, m)P(✓ | D, m)d✓
P(m | D) =
P(D | m)P(m)
P(D)
evidence
✓ ⇠ Beta(✓ | 2, 2)

17. - (MCMC)
- (Variational Inference)
P(✓ | D, m) =
P(D | ✓, m)P(✓ | m)
P(D | m) Z
P(D | ✓, m)P(✓)d✓
evidence

18. P(θ|D,m) KL q(θ)
ELBO
⇤
= argmin KL(q(✓; ) || p(✓ | D))
= argmin Eq(✓; )[logq(✓; ) p(✓ | D)]
ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )]
⇤
= argmax ELBO( )
P(✓ | D, m) =
P(D | ✓, m)P(✓ | m)
P(D | m)

19. - KL =ELBO
- P q
q(✓; 1)
q(✓; 5)
p(✓, D) p(✓, D)
✓✓
⇤
= argmax ELBO( )
ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )]

20. - P q
- :
- ADVI: Automatic Differentiation Variational Inference
- BBVI: Blackbox Variational Inference
arxiv:1603.00788
arxiv:1401.0118
https://github.com/HIPS/autograd/blob/master/examples/bayesian_neural_net.py

21. - VI
-
- David MacKay “Lecture 14 of the Cambridge Course”
- PRML 10
http://www.inference.org.uk/itprnn_lectures/

22. sts.build_factored_surrogate_posterier
❖ sts.build_factored_surrogate_posterier
❖ return: variational_posterior
(tfd.JointDistributionNamed)
❖ q
q(✓; 1)
q(✓; 5)
p(✓, D) p(✓, D)
✓✓

23. ELBO
❖ vi.fit_surrogate_porterier
⇤
= argmax ELBO( )
ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )]

24. vi.fit_surrogate_posterier
❖ vi.fit_surrogate_porterier
❖ model
❖ surrogate_posterier
❖ num_step

25. sts.forecast
❖ sts.forecast
❖

26. TFP MCMC: sts.fit_with_hmc
❖ sts model
❖ sts.Sum([trend, seasonal]…)
❖ sts.fit_with_hmc
❖ model
❖ data

27. TFP MCMC: sts.fit_with_hmc
❖ CO2
❖ 70
❖ 10 90
❖ …

28. -
-
P(✓ | D, m) = P(D | ✓, m)P(✓ | m)
liklihood priorposterior
✓
https://github.com/dfm/corner.py

29. ✓
http://twiecki.github.io/blog/2014/01/02/visualizing-mcmc/
NUTS (HMC)
Metropolis
-Hastings

31. tfp: normal
❖ tfp.python.distributions.normal
❖ https://github.com/tensorflow/probability/blob/v0.11.0/tensorflow_probability/
python/distributions/normal.py
❖ tf.random.normal
❖ https://www.tensorflow.org/api_docs/python/tf/random/normal
❖ tf.python.ops.random_ops
❖ https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/python/ops/
random_ops.py#L43-L98
❖ tf.python.ops.distributions.distribution.py
❖ https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/python/ops/
distributions/distribution.py
❖ tensorflow::ops::RandomNormal
❖ https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/random-normal
❖ https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/core/ops/
random_ops.cc
❖ https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/core/kernels/
parameterized_truncated_normal_op.cc
❖ https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/core/kernels/
parameterized_truncated_normal_op.h
❖ Eigen::MatrixExponential

32. PyMC3: normal
❖ pymc3.distributions.continuous.normal
❖ https://github.com/pymc-devs/pymc3/blob/master/pymc3/
distributions/continuous.py
❖ scipy.stas.norm
❖ https://docs.scipy.org/doc/scipy/reference/generated/
scipy.stats.norm.html#scipy.stats.norm
❖ scipy.stats._continuous_distns.norm_gen
❖ https://github.com/scipy/scipy/blob/master/scipy/stats/
_continuous_distns.py
❖ np.exp
❖ https://numpy.org/doc/stable/reference/generated/numpy.exp.html
❖ np.core.src.npymath.npy_math_complex.c.src
❖ https://github.com/numpy/numpy/blob/master/numpy/core/src/
npymath/npy_math_complex.c.src#L196

33. tfp.sts: pros
❖ tfp.distributions (tfd)
❖ colaboratory
❖ @tf.function(experimental_compile=true)
❖ Bijector API
❖ API
❖ BigQuery

34. tfp.sts: cons
❖ MCMC …
❖
❖ Traceback
❖ API Tensor shape
❖ API
❖
❖ PyMC4, edword2 …
❖

35. FBProphet
❖ Python cmdstan
❖ stan
❖ https://github.com/facebook/prophet/blob/master/python/stan/unix/
prophet.stan
❖ Prophet
❖ https://github.com/facebook/prophet/blob/master/python/fbprophet/
forecaster.py
❖
❖ https://peerj.com/preprints/3190/
❖
❖ /
❖ Stan GAM
❖
❖ GAM

36. TFP TF

37. PPL
TFP
TensorFlow(TF) + (PPL)
TF:
PPL: + +

38. 1. TF:
-
- :

39. 1. TF:

40. 1. TF:
-
-
- GPU / TPU
Inception v3 Inception v4
# of parameters: 42,679,816
# of layers: 48

41. 1. TF:
- Keras, Slim
- TensorBoard

42. 1. TF:
-
- tf .distributions

43. 2.
x:
x⇤
s P(x | ↵)
✓⇤
⇠ Beta(✓ | 1, 1)

44. 2.
- ( )
p(x, ✓) = Beta(✓ | 1, 1)
50Y
n=1
Bernoulli(xn | ✓),

45. 2.
-
log_prob()
-
mean()
-
sample()
- tfp.distributions

46. 3.
Edward TF

47. TFP
- TFP = TensorFlow +
+ +
- TensorFlow
-
- TF GPU, TPU, TensorBoard, Keras
-
- TensorFlow

48. Questions
kashino@bakfoo.com
@yutakashino

49. BakFoo, Inc.
NHK NMAPS:
+

50. BakFoo, Inc.
PyConJP 2015
Python

51. BakFoo, Inc.

52. BakFoo, Inc.
QUICK: Workstation, Qr1 | AI

53. BakFoo, Inc.
: SNS +