1. Machine Learning Proposed Term Paper Topics
Robert Stengel
Robotics and Intelligent Systems MAE 345,
Princeton University, 2009
MAE 345, Fall 2009
! Multistep NN with Memory
• Markov Decision Processes ! Maze-Navigating Robot
– Optimal and near-optimal control ! Robotic Prosthetic Device
• Finding Decision Rules in Data ! Optimal Control of an Ambiguous Robot
– ID3 algorithm ! Game-Playing NN
• Search ! NN for Object Recognition
! Robotic Cloth Folder
! SAGA Simulated Creature
! NN to Optimize Problem Set Solution
! Blob-Tracking NN
! Dust-Collecting Robot that Learns
! NN for Stock Return Prediction
Copyright 2009 by Robert Stengel. All rights reserved. For educational use only.
http://www.princeton.edu/~stengel/MAE345.html
Finding Decision Example of On-Line
Rules in Data Code Modification
• Identification of key attributes and • Execute a decision tree
outcomes – Get wrong answer
• Add logic to distinguish between right and wrong
• Taxonomies developed by experts cases
• First principles of science and – If Comfort Zone = Water,
• then Animal = Hippo,
mathematics • else Animal = Rhino
• Trial and error – True, but Animal is Dinosaur, not Hippo
– Ask user for right answer
• Probability theory and fuzzy logic – Ask user for a rule that distinguishes between right and
wrong answer: If Animal is extinct, …
• Simulation and empirical results

2. Maximizing the Utility Function
Markov Decision Process of a Markov Process
• Model for decision making under uncertainty "
Utility function: J = # ! (t)Ra(t ) [ x(t), x(t + 1)]
! S, A, Pam ( x k , x ') , Ram ( x k , x ') #
t =0
" $ ! (t) : discount rate, 0<! (t)<1
where
S : finite set of states, x1 , x 2 ,…, x K "
A : finite set of actions, a1 , a2 ,…, aM Utility function to go = Value function: V = # ! (t)Ra(t ) [ x(t), x(t + 1)]
Pam ( x k , x ') = Pr ! x k ( ti +1 ) = x ' | x k ( ti ) = x k , a ( ti ) = am #
t =t current
" $
Ram ( x k , x ') = Expected immediate reward for transition from x k to x ' • Optimal control at t
$
& " (
&
• Optimal decision maximizes expected total reward (or u opt ( t ) = arg max % Ra(t ) [ x(t), x(t + 1)] + ! (t) # Pa(t ) [ x(t), x(t + 1)]V [ x(t + 1)])
minimizes expected total cost) by choosing best set of a &
' t =t current &
*
actions (or control policy) • Optimized value function
– Linear-quadratic-Gaussian (LQG) control "
– Dynamic programming -> HJB equation ~> A* search V * ( t ) = Ruopt (t ) [ x * (t)] + ! (t) # Puopt (t ) [ x * (t), x est * (t + 1)]V [ x est * (t + 1)]
t =t current
– Reinforcement learning ~> Heuristic search
Reinforcement (“Q”) Learning Q Learning Control of a Markov
Control of a Markov Process Process is Analogous to LQG
• Q: quality of a state-action function Control in the LTI Case
• Heuristic value function
• One-step philosophy for heuristic optimization $ {
Q [ x(t + 1), u(t + 1)] = Q [ x(t), u(t)] + ! (t) # Ru(t ) [ x(t)] + " (t)max Q [ x(t + 1), u ]% ' Q [ x(t), u(t)]
u & }
! (t) : learning rate, 0<! (t)<1
$ { u & }
Q [ x(t + 1), u(t + 1)] = Q [ x(t), u(t)] + ! (t) # Ru(t ) [ x(t)] + " (t)max Q [ x(t + 1), u ]% ' Q [ x(t), u(t)]
Controller
! (t) : learning rate, 0<! (t)<1
x k +1 = !x k + "C ( x k # x k *)
ˆ
• Various algorithms for computing best control value
Estimator
ubest ( t ) = arg max Q [ x(t + 1), u ]
u
x k = !x k "1 " #C ( x k "1 " x k "1 *) + K ( z k " H x x k "1 )
ˆ ˆ ˆ ˆ
Q-Learning Snail Q-Learning, Ball on Plate

3. LQG Control Optimizes Discrete- Structuring an Efficient
Time LTI Markov Process Decision Tree (Off-Line)
• Choose most important attributes first
• Recognize when no result can be
deduced
• Exclude irrelevant factors
! S, A, Pam ( x k , x ') , Ram ( x k , x ') #
" $
where
• Iterative Dichotomizer*: the ID3 Algorithm
S : infinite set of states, x1 , x 2 ,…, x K – Build an efficient decision tree from a fixed
A : infinite set of actions, a1 , a2 ,…, aM set of examples (supervised learning)
Pam ( x k , x ') = Pr ! x k ( ti +1 ) = x ' | x k ( ti ) = x k , a ( ti ) = am #
" $
Ram ( x k , x ') = Expected immediate reward for transition from x k to x ' *Dichotomy: Division into two (usually contradictory)
parts or opinions
Fuzzy Ball-Game Training Set Parameters of the ID3 Algorithm
Attributes Decisions
Case # Forecast Temperature Humidity Wind Play Ball?
1 Sunny Hot High Weak No
2 Sunny Hot High Strong No
3 Overcast Hot High Weak Yes
4 Rain Mild High Weak Yes
5 Rain Cool Low Weak Yes
6 Rain Cool Low Strong No
7 Overcast Cool Low Strong Yes
8 Sunny Mild High Weak No
9 Sunny Cool Low Weak Yes • Decisions, e.g., Play ball or
10 Rain Mild Low Weak Yes
11 Sunny Mild Low Strong Yes don!t play ball
12 Overcast Mild High Strong Yes
13 Overcast Hot Low Weak Yes – D = Number of possible decisions
14 Rain Mild High Strong No • Decision: Yes, no

4. Parameters of Parameters of
the ID3 Algorithm the ID3 Algorithm
• Attributes, e.g., Temperature, humidity, • Training trials, e.g., all the
wind, weather forecast
– M = Number of attributes to be considered in games played last month
making a decision – N = Number of training trials
– Im = Number of values that the ith attribute can
take – n(i) = Number of examples with
• Temperature: Hot, mild, cool ith attribute
• Humidity: High, low
• Wind: Strong, weak
• Forecast: Sunny, overcast, rain
Example: Probability Spaces for Example: Decision, given
Three Attributes Values of Three Attributes
• Probability of an attribute value
represented by area in diagram
Attribute #1 Attribute #2 Attribute #3 Attribute #1 Attribute #2 Attribute #3
2 possible values 6 possible values 4 possible values 2 possible values 6 possible values 4 possible values

5. Accurate Detection of Events Depends
Accurate Detection of Events Depends
on Their Probability of Occurence
on Their Probability of Occurence
! noise = 0.1
! noise = 0.2
! noise = 0.4
Entropy Measures Information Entropy of Two Events with Various
Content of a Signal Frequencies of Occurrence
• Pr(i) log2Pr(i) represents the channel capacity
(i.e., average number of bits) required to portray
• S = Entropy of a signal encoding I distinct events the ith event
I • Frequencies of occurrence estimate
S = ! " Pr(i) log 2 Pr(i) 0 " Pr(.) " 1
log2 Pr(.) " 0 probabilities of each event (#1 and #2)
i =1 – Pr(#1) = n(#1)/N
log2 Pr(#1 or #2) " 0
– Pr(#2) = n(#2)/N = 1 – n(#1)/N
• i = Index identifying an event encoded by
a signal
• Pr(i) = Probability of ith event
S = S# 1 + S# 2
• log2Pr(i) = Number of bits required to = ! Pr(#1) log 2 Pr(#1) ! Pr(# 2) log 2 Pr(# 2)
characterize the probability that the ith
event occurs

6. Best Decision is Related to Entropy
Entropy of Two Events with Various and the Probability of Occurrence
Frequencies of Occurrence • High entropy
Entropies for 128 Trials – Signal provides high coding I
S = !" Pr(i) log 2 Pr(i)
Pr(#1) - # of Bits(#1) Pr(#2) - # of Bits(#2) Entropy
precision of distinct events
n n/N log2(n/N) 1 - n/N log2(1 - n/N) S
1 0.008 -7 0.992 -0.011 0.066 – Differences coded with few bits
2 0.016 -6 0.984 -0.023 0.116 i=1
4 0.031 -5 0.969 -0.046 0.201 • Low entropy
8 0.063 -4 0.938 -0.093 0.337
16 0.125 -3 0.875 -0.193 0.544 – Lack of distinction between
32 0.25 -2 0.75 -0.415 0.811 signal values
64 0.50 -1 0.50 -1 1
96 0.75 -0.415 0.25 -2 0.811 – Detecting differences requires
112 0.875 -0.193 0.125 -3 0.544
120 0.938 -0.093 0.063 -4 0.337 many bits
124 0.969 -0.046 0.031 -5 0.201
126 0.984 -0.023 0.016 -6 0.116 • Best classification of events
127 0.992 -0.011 0.008 -7 0.066 when S = 1...
– but that may not be achievable
Case # Forecast Temperature Humidity Wind Play Ball?
1
2
3
4
Sunny
Sunny
Overcast
Rain
Hot
Hot
Hot
Mild
High
High
High
High
Weak
Strong
Weak
Weak
No
No
Yes
Yes
Decision-Making Decision Tree Produced by
5 Rain Cool Low Weak Yes
6
7
Rain
Overcast
Cool
Cool
Low
Low
Strong
Strong
No
Yes
ID3 Algorithm
Parameters for ID3
8 Sunny Mild High Weak No
9 Sunny Cool Low Weak Yes
10 Rain Mild Low Weak Yes
11 Sunny Mild Low Strong Yes
12
13
Overcast
Overcast
Mild
Hot
High
Low
Strong
Weak
Yes
Yes
• Root Attribute gains, Gi
14 Rain Mild High Strong No
– Forecast: 0.246
– Temperature: 0.029
• SD = Entropy of all possible decisions –
–
Humidity: 0.151
Wind: 0.048
D
SD = !" Pr(d) log 2 Pr(d)
d =1
• Gi = Information gain of ith attribute
Im D
Gi = SD + ! Pr(i) ! Pr(id ) log 2 Pr(id )
i=1 d =1
• Pr(id) = n(id)/ N(d) = Probability that ith • Temperature is inconsequential and
attribute correlates with dth decision is not included in the decision tree

7. Decision Tree Produced by Search
ID3 Algorithm
• Typical AI textbook problems
– Prove a theorem
• Sunny Branch
– Solve a puzzle (e.g., Tower of
Attribute gains, Gi Hanoi)
– Temperature: 0.57 – Find a sequence of moves that
– Humidity: 0.97 wins a game (e.g., chess)
– Wind: 0.019 – Find the shortest path
connecting a set of points (e.g.,
Traveling salesman problem)
– Find a sequence of symbolic
transformations that solve a
calculus problem (e.g.,
Mathematica)
• The common thread: search
– Structures for search
– Strategies for search
Curse of Structures for Search
Dimensionality
• Feasible search paths may • Trees
grow without bound
– Possible combinatorial – Single path between root and any node
explosion
– Checkers: 5 x 1020 possible
– Path between adjacent nodes = arc
moves – Root node
– Chess: 10120 moves
– Protein folding: ? • no precursors
• Limiting search complexity – Leaf node
– Redefine search space
– Employ heuristic (i.e., pragmatic)
• no successors
rules • possible terminator
– Establish restricted search range
– Invoke decision models that
have worked in the past

8. Structures for Search Directions of Search
• Forward chaining
• Graphs –Reason from premises to actions
–Multiple paths –Data-driven: draw conclusions
between root from facts
and some • Backward chaining
nodes
–Reason from actions to premises
–Trees are
subsets of –Goal-driven: find facts that
graphs support hypotheses
Strategies for Search Blind Search
• Search forward from opening?
• Node expansion
• Search backward from end game? – Find all successors to that node
• Realistic assessment • Both?
• Depth-first forward search
– Not necessary to consider all 10120 possible moves – Expand nodes descended from most recently
to play good chess expanded node
– Playing excellent chess may require much forward – Consider other paths only after reaching a node
and backward chaining, but not 10120 evaluations with no successors
– Most applications are more procedural
• Breadth-first forward search
• Search categories – Expand nodes in order of proximity to the start node
– Blind search – Consider all sequences of arc number n (from root
– Heuristic search node) before considering any of number (n + 1)
– Probabilistic search – Exhaustive, but guaranteed to find the shortest path
– Optimization to a terminator

9. AND/OR Graph Search
Blind Search
• Bidirectional search
– Search forward from root node and
backward from one or more leaf nodes
– Terminate when search nodes coincide • A node is “solved” if
• Minimal-cost forward search – It is a leaf node with a satisfactory goal
– Each arc is assigned a cost state
– Expand nodes in order of minimum cost – It has solved AND nodes as successors
– It has OR nodes as successors, at least
one of which is solved.
• Goal: Solve the root node
Heuristic Search Heuristic Optimal Search
• For large problems, blind search typically
leads to combinatorial explosion
• Employ heuristic knowledge about the
quality of possible paths
– Decide which node to expand next
– Discard (or prune) nodes that are unlikely to
be fruitful
• Search for feasible (approximately
optimal) rather than optimal solutions
• Ordered or best-first search
– Always expand “most promising” node

10. Mechanical Control System
Heuristic Dynamic
Programming: A* Search
k kf
Jk f = ! Ji +
ˆ ! J (arc )
ˆ
i i
i=1 i= k +1
• Each arc bears an incremental cost
• Cost, J, estimated at kth instant =
– Cost accrued to k
– Remaining cost to reach final point, kf
• Goal: minimize estimated cost by choice of
remaining arcs
• Choose arck+1, arck+2 accordingly
• Use heuristics to estimate remaining cost
Inferential Fault Analyzer for
Helicopter Control System Local Failure Analysis
• Local failure analysis • Frames store facts and facilitate search and inference
– Set of hypothetical models of specific failure – Components and up-/downstream linkages of control system
• Global failure analysis – Failure model parameters
– Forward reasoning assesses failure impact – Rule base for failure analysis (LISP)
– Backward reasoning deduces possible causes
Aft Rotor
Forward Rotor
Cockpit Controls

11. Heuristic Search Global Failure Analysis
• Global failure analysis
– Determination based on aggregate of
local models
• Heuristic score based on
– Criticality of failure
– Reliability of component
– Extensiveness of failure
– Implicated devices
– Level of backtracking
– Severity of failure
– Net probability of failure model
Shortest Path Problems
• Find the shortest (or • Simulated annealing solution
Next Time:
least costly) path that
• Genetic algorithm solution
visits all selected cities
just once • Neural network solution
Knowledge
– Traveling Saleman
– MapQuest/GPS/GIS
Representation
Modified Dijkstra
Algorithm