This is lecure 3 note for JAIST summer school on computational motor control (Hirokazu Tanaka & Hiroyuki Kambara). Lecture video: https://www.youtube.com/watch?v=dtpgJLRt90M

1. Computational Motor
Control Summer School
03: State space models
for motor adaptation.
Hirokazu Tanaka
School of Information Science
Japan Institute of Science and Technology

2. State-space modeling of motor adaptation.
In this lecture, we will learn:
• Motor adaptation paradigms
• Continuous-time state-space models
• Discrete-time state-space models
• Controllability
• Observability
• State-space description for motor adaptation
• Multi-rate models
• Motor memory of errors
• Mirror reversal (non-error based learning)

3. Motor adaptation paradigms to dynamical perturbations: Force-field adaptation.
Shadmehr & Mussa-Ivaldi (1994) J Neurosci
Baseline (no field) Initial exposures
adaptation catch trials

4. Motor adaptation paradigms to kinematical perturbations: Visuomotor rotation.
Krakauer et al. (2000) J Neurosc; Krakauer (2009) Progress in Motor Control

5. Adaptation to prism displacements.
Martin et al. (1996) Brain ;Kitazawa et al. (1995) J Neurosci

6. Adaptation to prism displacements.
Kitazawa et al. (1995) J Neurosci
1 1n n ne e ke  1
1
1
n
in
i
e e k e


  

7. Continuous-time state-space models.
F ma mx 
x v
Fv a
m


 
x
v
 
  
 
x
Newton’s equation of dynamics
0 1 0
0 0 1/
x x
F
v v m
      
          
      
x Ax Bu
 x Ax Bu
State-space representation
A x B u
State-space vector

8. Discrete-time state-space models.
Discrete-time representation
 k k t x x
 
 
 
1 ( 1)k
k k
k k k
k k
k t
t
t
t t
   
  
   
    
x x
x x
x Ax Bu
I A x B u
1
ˆ ˆ
k k k  x Ax Bu  
 
2
2ˆ
ˆ t
t
t t
te t
e
t


  
  
 


 
A
A
A I A
B BB
t
Δt 2Δt 3Δt (k-1)Δt kΔt (k+1)Δt0
k-1 k k+10 1 2 3
time (continuous)
time steps
(discrete)

9. Deterministic and stochastic state-space models.
1k k k
k k
  


x Ax Bu
z Cx
1
k
kk k
kk
k   

 
w
v
x Ax Bu
z Cx
Deterministic Stochastic
XkXk-1 Xk+1
zk-1 zk zk+1
uk-1 uk uk+1

10. Linear time-variant and time-invariant state-space models.
1k k k
k k
k k
k
  


x x u
xCz
A B
Time-variant model
1k k k
k k
  


x x u
xCz
A B
Time-invariant model
Throughout these lectures, we will use linear time-invariant (LTI) models
for mathematical simplicity.

11. State-space models in an explicit component form.
1k k k  x Ax Bu
k kz Cx
1, 1
2, 1
, 1
k
k
N k
x
x
x



1,
2,
,
k
k
N k
x
x
x
11 12 1
21 21
11 1
N
N
a a a
a a
a a
11
21
1
1
L
N NL
b b
b
b b
1
L
u
u
= +
1,
2,
,
k
k
N k
x
x
x
1
M
z
z
11 12 1
1 112
N
MM
c c c
c c c
=
Process equation
Measurement equation
N vector N×N matrix N vector N×L matrix L vector
M vector M×N matrix N vector

12. Controllability: the ability of driving a system into desired final state.
1k k k  x Ax Bu
, , ,N L N N
k k
N L 
   x u A B
Controllability is the ability of external inputs {uk} to drive a state from any
initial condition to any final condition in a finite time. A state-space model
is controllable if the N×NL controllability matrix has full row rank:
2 1n
   B AB A B A B
Sketch of proof:
0
1 1
2
2 2 1
1
1
0
2
N N N
N N
NN
N
N
N
 
  


 
  
 
 
       
 
 
x Ax Bu
A x ABu Bu
u
u
A x B AB A B
u
Kalman (1963) SIAM J Contr

13. Observability: determining hidden state from measurements.
k kz Cx
, ,N M N
k
M
k

  x z C
Observability is the ability to determine a (latent) state from a sequence
of measurements {zk}. A state space model is called observable if the
MN×N observability has full rank N:
Kalman (1963) SIAM J Contr
1N 
 
 
 
 



 
C
CA
C A

14. State-space models for dynamic (force-field) motor adaptation.
Thoroughman & Shadmehr (2000) Nature; Donchin et al. (2003) J Neurosci
1n n n
n n n
  
 
x Ax Bu
z Cx Du

15. State-space models for dynamic (force-field) motor adaptation.
Thoroughman & Shadmehr (2000) Nature; Donchin et al. (2003) J Neurosci
1n n n
n n n
  
 
x Ax Bu
z Cx Du

16. State-space models for kinematic (visual rotation) motor adaptation.
Tanaka et al. (2006) J Neurophysiol
T
1k k k k
k k k
z
z
   

x Ax BH
H x

17. Trial-by-trial generalization width reflects directional tuning width.
   
 
 
 
1
1i i N
i
N
g
g
g

 

 
 
   
  
r r r r Rg
 T
k k k  rR g
         1 1 1
T
k k k k k k k k        R R g R g gr g
Suppose that, for target direction θ, the motor output is a weighted sum of
population activity {gi(θ)} multiplied with preferred directions {ri}:
A gradient descent learning rule specifies the change of preferred directions
according to the movement error Δrk and the population activity {g(θk)} :
This change affects the motor output at the next trial as:

18. Two-rate model of motor adaptation: fast and slow learners.
Smith et al. (2006) PLoS Biol
1n n nu  x Ax B
 
 
 
 
 
 
 
 
f ff f
1
s ss s
1
0
0
n n
n
n n
x xa b
u
x xa b


      
       
      
      
n nz  Cx
 
 
 
   
f
f s1
1 1s
1
1 1 n
n n n
n
x
z x x
x

 

 
   
 
 
       f s s f
,a a b b 
There are two learners in the brain; the fast learner (x(f)) learns
quickly but forgets quickly, while slow learner (x(s)) learns slowly
but maintains its memory longer.
Motor output is a sum of the fast
and slow learners.
State vector consists of fast
(x(f)) and slow learners (x(s)).

19. The model explains savings, spontaneous recovery.
Smith et al. (2006) PLoS Biol
Savings Spontaneous recovery

20. The prediction of spontaneous recovery is confirmed in humans.
Smith et al. (2006) PLoS Biol

21. The slow process contributes to motor memory consolidation.
Joiner & Smith (2008) J Neurophysiol
The slow process, but not the fast process, contributes to motor memory consolidation.

22. Explicit (strategic) and implicit (error-based) learning.
Mazzoni & Krakauer (2006) J Neurosci
Strategy (aiming the adjacent target) cancels the “error” without
any adaptation!

23. Explicit (strategic) and implicit (error-based) learning.
Mazzoni & Krakauer (2006) J Neurosci

24. Explicit (strategic) and implicit (error-based) learning.
Mazzoni & Krakauer (2006) J Neurosci

25. What is “motor error?”: Aiming error and target error.
Taylor & Ivry (2011) PLoS Comp Biol; Taylor & Ivry (2014) Prog Brain Res

26. State-space model for strategic and error-based learning.
Taylor & Ivry (2011) PLoS Comp Biol; Taylor & Ivry (2014) Prog Brain Res
yn: target direction
rn: rotation angle
xn: adaptation variable
sn: strategy variable
yn
sn
sn-rn+xn
 aiming
n n n nn n ne s s r x r x    
     target
n n n n n n nn ne y s r x y sx r     
aiming
netarget
ne

27. State-space model for strategic and error-based learning.
Taylor & Ivry (2011) PLoS Comp Biol; Taylor & Ivry (2014) Prog Brain Res
yn
sn
sn-rn+xn
 aiming
n n n nn n ne s s r x r x    
     target
n n n n n n nn ne y s r x y sx r     
aiming
netarget
ne
aiming
targ
1
e
1
t
n n n
nn n
x ax be
s cs de


 
 
a=0.99, b=0.015,
c=0.999, d=0.022

28. Steepest descent learning rule for optimization.
Lecture 6, in Neural Networks for Machine Learning, Geoff Hinton
E
( 1) ( )n n E
 
 

w w
w
optimum
E
w
E
w
Descent learning rule:
RPROP: Adjustment of learning rate.
E
w
gradient
learning rate
     
1  1 

29. Motor memory of experienced errors.
Herzfeld et al. (2014) Science
( ) ( ) ( )
( 1) ( ) ( ) ( )
ˆ
ˆ ˆ
n n n
n n n n
e y y
x ax e
  

 
( )
( )
( )
( )
( )
: perturbation
ˆ : estimated perturbation ("belief")
: sensory consequence
ˆ : predicted sensory consequence
: control signal
n
n
n
n
n
x
x
y
y
u
State-space model: memory of environments
Population-coding model: memory of errors
 ( ) ( )n n
i i
i
w g e    
 
2
2
exp
2
i
i
e e
g e

 
  
 
 
 
 
   
( 1)
( 1) ( 1) ( 1) ( )
T ( 1) ( 1)
sgn
n
n n n n
n n
e
e e
e e


  
 
 
g
w w
g g error
activity
w1 w2 w3 wn
η

30. Motor memory of experienced errors.
Herzfeld et al. (2014) Science

31. Displacement and left-right reversal: Why so different?
Martin et al. (1996) Brain; Sekiyama et al. (2000) Nature
Displacement prism
… takes only few dozen trials. … takes a few weeks.
Left-right reversed prism
Day 3
Day 34

32. Mirror reversal: a distinct form of motor adaptation?
Taglen et al. (2014) J Neurosci; Lilicrap et al. (2013) Exp Brain Res
Movement number Movement number
Absoluteerror
Absoluteerror
Visual rotation Mirror reversal

33. Summary
• A state-space model consists of a process equation
(temporal transition) and an observation equation
(measurement).
• Humans are flexible to a novel environment, known as
motor adaptation, such as perturbations of force fields
and visual transformation.
• State-space modeling has been very successful in
describing trial-by-trial adaptation processes in humans.

34. References
• Thoroughman, K. A., & Shadmehr, R. (2000). Learning of action through adaptive combination of motor primitives. Nature,
407(6805), 742-747.
• Donchin, O., Francis, J. T., & Shadmehr, R. (2003). Quantifying generalization from trial-by-trial behavior of adaptive systems
that learn with basis functions: theory and experiments in human motor control. The Journal of Neuroscience, 23(27),
9032-9045.
• Tanaka, H., Sejnowski, T. J., & Krakauer, J. W. (2009). Adaptation to visuomotor rotation through interaction between
posterior parietal and motor cortical areas. Journal of Neurophysiology, 102(5), 2921-2932.
• Smith, M. A., Ghazizadeh, A., & Shadmehr, R. (2006). Interacting adaptive processes with different timescales underlie
short-term motor learning. PLoS Biol, 4(6), e179.
• Joiner, W. M., & Smith, M. A. (2008). Long-term retention explained by a model of short-term learning in the adaptive
control of reaching. Journal of Neurophysiology, 100(5), 2948-2955.
• Inoue, M., Uchimura, M., Karibe, A., O'Shea, J., Rossetti, Y., & Kitazawa, S. (2015). Three timescales in prism adaptation.
Journal of Neurophysiology, 113(1), 328-338.
• Mazzoni, P., & Krakauer, J. W. (2006). An implicit plan overrides an explicit strategy during visuomotor adaptation. The
Journal of Neuroscience, 26(14), 3642-3645.
• Taylor, J. A., & Ivry, R. B. (2011). Flexible cognitive strategies during motor learning. PLoS Comput Biol, 7(3), e1001096-
e1001096.
• Taylor, J. A., & Ivry, R. B. (2014). Cerebellar and prefrontal cortex contributions to adaptation, strategies, and reinforcement
learning. Progress in Brain Research, 210, 217.
• Taylor, J. A., Krakauer, J. W., & Ivry, R. B. (2014). Explicit and implicit contributions to learning in a sensorimotor adaptation
task. The Journal of Neuroscience, 34(8), 3023-3032.
• Telgen, S., Parvin, D., & Diedrichsen, J. (2014). Telgen, S., Parvin, D., & Diedrichsen, J. (2014). Mirror reversal and visual
rotation are learned and consolidated via separate mechanisms: Recalibrating or learning de novo?. The Journal of
Neuroscience, 34(41), 13768-13779.?. The Journal of Neuroscience, 34(41), 13768-13779.
• Lillicrap, T. P., Moreno-Briseño, P., Diaz, R., Tweed, D. B., Troje, N. F., & Fernandez-Ruiz, J. (2013). Adapting to inversion of
the visual field: a new twist on an old problem. Experimental Brain Research, 228(3), 327-339.
• Ostry, D. J., Darainy, M., Mattar, A. A., Wong, J., & Gribble, P. L. (2010). Somatosensory plasticity and motor learning. The
Journal of Neuroscience, 30(15), 5384-5393.

35. Exercise
• Simulate the state-space model proposed by Taylor and
Ivry.
• Mirror reversal is different from most adaptation
paradigms in that learning from error worsens the
performance. Can we consider a state-space model for
mirror reversal?