Uncertainty Awareness in Integrating Machine Learning and Game Theory

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Uncertainty Awareness in Integrating
Machine Learning and Game Theory
不確実性を通して見る
機械学習とゲーム理論とのつながり
Rikiya Takahashi
SmartNe...

About Myself
●
Rikiya TAKAHASHI, Ph.D. (高橋力矢)
– Engineer in SmartNews, Inc., from 2015 to current
– Research Staff Member...

Example of Previous Work
● Budget-Constrained Markov Decision Process for
Marketing-Mix Optimization (Takahashi+, 2013 & 2...

Example of Previous Work
● Travel-Time Distribution Prediction on a Large
Road Network (Takahashi+, 2012)
A
B
rN/L
rN/L
rN...

Example of Previous Work
● Bayesian Discrete Choice Modeling for Irrational
Compromise Effect (Takahashi & Morimura, 2015)...

Agenda
1.Uncertainty Awareness as an Essence in
Data-Oriented Real-World Decision Making
2.From Machine Learning to Game T...

Machine Learning (ML)
● Set of inductive disciplines to design probabilistic
model and estimate its parameters that maximi...

Estimation via Bayes' theorem
● Basis behind today's most ML algorithm
posterior distribution: p(θ∣D)=
p(D∣θ ) p(θ)
∫θ
p(D...

E.g., Gaussian Process Regression (GPR)
● Bayesian Ridge Regression
– Unlike MAP Ridge regression (dark gray), input-
depe...

Gap between Deduction & Induction
Today's AI is integrating both.
Do not divide the work between
inductive & deductive res...

Induction ↔ Deduction
Dataset
Typical Problem Solving
in the Real World
Estimate of
Environment
Inductive Process
Machine ...

Induction ↔ Deduction
Dataset
Typical Problem Solving
in the Real World
Estimate of
Environment
Inductive Process
Machine ...

Induction ↔ Deduction
Dataset
Typical Problem Solving
in the Real World
Estimate of
Environment
Inductive Process
Machine ...

See the Difference
Typical Problem Solving
in the Real World:
Unnecessarily too much effort
in solving each subproblem
Vul...

See the Difference
Typical Problem Solving
in the Real World:
State-of-the-art AI
Solve a Hard
Inductive Problem
Solve ano...

Optimization under Uncertainty
● Interval Estimation
(e.g., Bayesian)
– Quantify uncertainty
– Optimize over all
possible ...

Vapnik's Principle (Vapnik, 1995)
When solving a problem of interest, do not solve a
more general problem as an intermedia...

Batch Reinforcement Learning
● A good example of involving both inductive and
deductive processes.
● Also a good example o...

Markov Decision Process
● Framework for long-term-optimal decision making
– S: set of states, A: set of actions
P(s'|s,a):...

Markov Decision Process
● Easy to solve If the environment is known
– Via dynamic programming or linear programming
when P...

Types of Reinforcement Learning
● Model-based ↔ Model-free
● On policy ↔ Off policy
● Value iteration ↔ policy search
● Mo...

Model-free approach
● Remember: our aim is to get the optimal policy.
No need of estimating environment, in principle.
– A...

Bellman Optimality Equation
● Policy is derived if we have an estimate of Q(s,a).
– Simpler than estimating P(s'|s,a) & r(...

Fitted Q-Iteration (Ernst+, 2005)
● For k=1,2,... iterate 1) value computation and
2) regression as
∀i∈{1,…, n} vi
(k)
:=r...

Policy Gradient
●
Accurately fit policy　　 while roughly fit Q(s,a)
– More directness to the final aim
– Applicable for con...

Functional Approximation in Practice
● Concrete functional form of Q(s,a) and/or
– Q should be a universal functional appr...

Functional Approximation in Practice
● Is any univ. approximator OK? – No, unfortunately.
– Universal approximator is mere...

Message
● Uncertainty awareness is essential on data-
oriented decision making.
– No division between induction and deduct...

Agenda
1.Uncertainty Awareness as an Essence in
Data-Oriented Real-World Decision Making
2.From Machine Learning to Game T...

Shrinkage Matters in the Real World.
● Q. Why prior helps avoid over-fitting?
– A. shrinkage towards prior mean (e.g., 0 i...

Random Utility Theory as a Rational Model
Each human is a rational maximizer of random utility.
Theoretical basis behind m...

Complexity of Real Human’s Choice
An example of choosing PC (Kivetz et al., 2004)
Each subject chooses 1 option from a cho...

Similarity E↵ect (Tversky, 1972)
Top-share choice can change due to correlated utilities.
E.g., one color from {Blue, Red}...

Attraction E↵ect (Huber et al., 1982)
Introduction of an absolutely-inferior option A (=decoy)
causes irregular increase o...

Compromise E↵ect (Simonson, 1989)
Moderate options within each chosen set are preferred.
Di↵erent from non-linear utility ...

Positioning of the Proposed Work
Sim.: similarity, Attr.: attraction, Com.: compromise
Sim. Attr. Com. Mechanism Predict. ...

Key Idea #1: a Dual Personality Model
Regard human as an estimator of her/his own utility function.
Assumption 1: DMS does...

Utility Calculator as Rational Personality
For every context i and option j, UC computes noiseless
sample of utility vij b...

Key Idea #2: DMS is a Bayesian estimator
DMS does not know fUC but has utility samples {vij }
m[i]
j=1 .
Assumption 2: DMS...

Convex Optimization for Model Parameters
Likelihood of the entire model is tractable, assuming the choice
is given by a lo...

Irrationality as Bayesian Shrinkage
Implication from the posterior-mean utility in (1)
Each option’s utility is shrunk int...

Recovered Context-Dependent Choice Criteria
For a speaker dataset: successfully captured mixture of
objective preference a...

A Result of p-beauty Contest by Real Humans
Guess 2/3 of all votes (0-100). Mean is apart from the Nash
equilibrium 0 (Cam...

Modeling Bounded Rationality
Early stopping at step k: Level-k thinking or Cognitive
Hierarchy Theory (Camerer et al., 200...

Linking ML with Game Theory (GT)
via Shrinkage Principle
Optimization
without shrinkage
Optimization
with shrinkage
ML GT
...

Early Stopping and Regularization
ML as a Dynamical System
to find the optimal parameters
GT as a Dynamical System
to find...

Message
● Bayesian shrinkage ↔ Bounded rationality
– Dual-personality model for contextual effects
– Towards data-oriented...

Agenda
1.Uncertainty Awareness as an Essence in
Data-Oriented Real-World Decision Making
2.From Machine Learning to Game T...

Additional Implications from ML
● Multiple equilibria or saddle points?
● Equilibria or “typical” transitional states?
– S...

Recent history in ML
● Waste of ~20 years for local optimality issue
– Neural Networks (NNs) have been criticized for thei...

Why most are saddle points?
● See spectrum of Hessian matrices of a random-
drawn non-linear function from a Gaussian proc...

Open Questions for Multiple Equilibria
● If a game is very complex involving lots of
parameters in pay-off or utility func...

See Dynamics:
“Typical” Transitional State?
● MLers are sensitive to convergence rate in fitting.
– We are in the finite-s...

See Dynamics:
“Typical” Transitional State?
● Mixing time of Markov processes of some games
is exponential to the number o...

See Dynamics: Trapped in Plateau?
● Fitting of a Deep NN is often trapped in plateaus.
– Natural gradient descent (Amari, ...

Conclusion
● Discussed how uncertainty should be incorporated
in inductive & deductive decision making.
– Quantifying unce...

THANK YOU FOR ATTENDING!
Download this material from
https://www.slideshare.net/rikija/uncertainty-awareness-in-integratin...

References
References I
Amari, S. (1997). Neural learning in structured parameter spaces -
natural Riemannian gradient. In...

References
References II
Chapelle, O. and Harchaoui, Z. (2005). A machine learning approach to
conjoint analysis. In Advan...

References
References III
Dotson, J. P., Lenk, P., Brazell, J., Otter, T., Maceachern, S. N., and
Allenby, G. M. (2009). A...

References
References IV
Kakade, S. M. (2002). A natural policy gradient. In Dietterich, T. G.,
Becker, S., and Ghahramani...

References
References V
McFadden, D. L. (1980). Econometric models of probabilistic choice
among products. Journal of Busi...

References
References VI
Roe, R. M., Busemeyer, J. R., and Townsend, J. T. (2001).
Multialternative decision ﬁeld theory: ...

References
References VII
Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour, Y. (2000).
Policy gradient methods ...

References
References VIII
Wen, C.-H. and Koppelman, F. (2001). The generalized nested logit
model. Transportation Researc...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

15分でわかる(範囲の)ベイズ統計学 by Ken'ichi Matsui 56238 views
Twitter炎上分析事例 2014年 by Takeshi Sakaki 2547 views
Generative adversarial networks by Shuyo Nakatani 5336 views
第35回強化学習勉強会・論文紹介　[Lantao Yu : 2016] by Takayuki Sekine 3452 views
「人工知能」の表紙に関するTweetの分析・続報 by Fujio Toriumi 3401 views
オープンソースを利用した新時代を生き抜くためのデータ解析 by nakapara 2600 views

View on Slideshare

1 of 66 Ad

View on Slideshare

1 of 66 Ad