Galilean Differential Geometry Of Moving Images

Slide 1: Galilean Differential Geometry of Moving Images Daniel Fagerström CVAP/NADA/KTH danielf@nada.kth.se

Slide 2: Differential Structure of Movies • How can we describe the local structure of an image sequence? • We will assume that a movie is a smooth function of 2+1 dimensional space-time • Looking for generic properties

Slide 3: Approaches for Motion Analysis • Optical flow • Spatio-temporal texture • Spatio-temporal differential invariants

Slide 4: Optical Flow • Geometry of the projected motion of particles in the observers field of view • Binding hypothesis needed to use it on image sequences • “Top down” • Local formulation, but non-local, due to binding hypothesis • Undefined when particles appears and disappears, e.g. motion boundaries • Does not use image structure

Slide 5: Spatio-Temporal Differential Invariants • Local geometry of spatio-temporal images • ”Bottom Up”, low level • No binding hypotheses, connection to the environment considered as a higher level problem • Well defined everywhere • Does use image structure, extension of low level vision for still images

Slide 6: Overview • Galilean geometry • Moving frames • Image geometry • 1+1 dimensional Galilean differential invariants • 2+1 dimensional Galilean differential invariants • What is required for a more realistic movie model

Slide 7: Galilean Geometry • Spatial and temporal translation a, spatial rotation R and spatio-temporal shear v  x   R v  x       a, a, v   n , R  SO n   t    0 1  t       x Shear x t t

Slide 8: Galilean Geometry • Insensitive to constant relative motion for parallel projection, approximately otherwise • Simplest meaning full model • Assumed implicitly when one talk about optical flow invariants: div, rot, dev, i.e. first order flow • Shape properties from the environment can be derived from relative motion • (Newton physics describe Galilean invariants)

Slide 9: Galilean Invariants • Planes of simultaneity (constant t) are invariant and has Euclidean geometry: distances and angles are invariants – i.e. an image sequence • The temporal distance between planes of simultaneity is an invariant p T  t, p   x, y , t  p S   x, y  , if t  0

Slide 10: Galilean ON-System • An n+1 dimensional Galilean ON-system (e1,e2,,e0) is s.t. (e1,e2,,en) is an Euclidean ON-system and ||e0||T=1

Slide 11: Moving Frames • Galilean geometry has no metric • We will use Cartan's method of moving frames, that does not require a metric • Moving frame: e:M! G½ GL(n) • Attach a frame that is adapted to the local structure in each point • Differential geometry: the local change of the frame: de

Slide 12: Moving Frames e  Ai de  dAi  dAA1e  C  A e C  AB   C  A  AC  B  A 1 C(A) contains the differential geometric invariants expressed in the global frame i

Slide 13: Image Geometry • Image space: E2­ - trivial fiber bundle with I Euclidian base space and log intensity as fiber (Koenderink 02) z=f(x,y) • f is smooth • Image geometry – Global gray level transformations – Lightness gradients

Slide 14: Gradient Gauge • For points where rf0 we can choose an adapted ON-frame {u,v} s.t. fu=0  u  1  fy  f x   x     f  f       Ai f y   y   v  x   • All functions over iujv f, i+j¸ 1 becomes invariants w.r.t. rotation in space and translation in intensity

Slide 15: Gray Level Invariants   0,  0 c12   0 C  A    c ,  12 0  c12   f x f xy  f y f xx  dx   f x f yy  f y f xy  dy , f x2  f y2   c12  u - isophote curvature   c12  v - gradient curvature   0, f uu f uv c12   du  dv   du   dv  0 fv fv invariant w.r.t. rotation in space and monotonic intensity transforma tions

Slide 16: Hessian Gauge • For points whererf0 we can choose an ON- frame {p,q} s.t. fpq=0 and |fpp|>|fqq|   p   cos   sin    x  f xy      Ai, tan 2      sin  cos    y  f yy  f xx  q    • All functions over ipjq f, i+j¸ 2 becomes invariants w.r.t. rotation in space, translation in intensity and addition of a linear light gradient (Koenderink 02)

Slide 17: Galilean 1+1 D • Two cases: – Isophotes cut the spatial line, motion according to the constant brightness assumption – Isophotes are tangent to the spatial line (along curves), creation, annihilation

Slide 18: Tangent Gauge • Let {t,x} be a global Galilean ON-frame, for points where fx0 we can define an adapted Galilean ON-frame {s,x} s.t. fs=0.  s  t    x , 0  f s  ft   f x ,   s   1  f t f x   t      0    Ai.  x  1   x   

Slide 19: Isophote Invariants  0 c01  C  A    0 0 ,  a  0,  0   f ss f sx c01   ds  dx fx fx  a ds   dx a  0,  0 a  c01 s - acceleration   c01 x - divergence

Slide 20: Hessian Gauge • Let {t,x} be a global Galilean ON-frame, for points where fxx0 we can define an adapted Galilean ON-frame {r,x} s.t. frx=0. r  t    x , 0  f rx   r f x  f tx   f xx ,   r   1  f tx f xx   t      0    Ai.     x  1  x 

Slide 21: Hessian Invariants  0 c01  C  A    0 0 ,  a  0,  0   f rrx f rxx c01   ds  dx f xx f xx  a dr   dx a  0,  0 a  c01 r - acceleration   c01 x - divergence

Slide 22: Galilean 2+1 D • General case • Also here are two different main cases – Isophote surfaces transversal to the spatial plane. Motion of isophote curves in the image – Isophote surfaces tangent to the plane. Creation, annihilation and saddle points

Slide 23: Invariants in the General Case    x y ∂s 1 v v ∂t ∂u = 0 cos −sin  ∂ x = A i , ∂v 0 sin  cos ∂ y   u u u v v v 0 a ds du dv a ds du dv C  A= 0 0  dsu duv dv u v 0 − ds du dv 0 au, av - acceleration u, v - divergence u, v - skew of the ”flow field”  - rotation of the plane in the temporal direction u, v - flow line curvature in the plane

Slide 24: More Descriptive Invariants a= a  a  , u 2 v 2 v u a =arctan a / a  , D=  u  u  v v  = 2 −1 0   2       u− v 0 1 u  v 1 0 1  u− v  u v 0 1 2  u v  v − u  = 2 −1 0  curlD 0 1 divD 1 0 defD  2 0 1  2 Q  −1 1 0 0 −1  Q    • D - rate of strain tensor for the spatio-temporal part of the frame field • a, curl D, div D, def D - are flow field invariants • a, , , u, v - are not flow field invariants

Slide 25: Tangent Gauge • Let {t, x, y} be a global Galilean ON-frame, for points where ||{fx,fy}||0 we can define an adapted Galilean ON-frame {s,u,v} s.t. fs=fu=fsu=0 • Principal acceleration extrema • Direction of u constant along s – used in Guichard (98)

Slide 26: Tangent Gauge  t  1 0 0   t   0 c01 c02          u   1 0 fy  f x   x   Ai C  BA   0 0 c12  fx  fy  2 2 0  c 0    v 0 fx f y   y     12   s  t   u   v c01  au ds   u du   dv 0  f s  ft   fv c02  av ds   v dv 0  f su  f tu   f uu   f uv c12   du   dv f ss f uv f ssu  f t f uv f tu f  au     t   s   1 f v f uu f uu    f v f uu f uu f v  t     u    0 1 0   u   BAi av   f ss f v   0 0  1   v   u   f suu f uu  v       v   f sv f v f f f   sv uv  suv C  BA  C  B   BC  A B 1 f v f uu f uu

Slide 27: Hessian Gauge • Let {t, x, y} be a global Galilean ON- frame, we define an adapted Galilean ON- frame {r, p, q} s.t. fpq= frp= frq=0. • Also defined when the spatial tangent disappears, e.g. for creation and disappearance of structure • r is the same vector field as when the optical flow constraint equation is solved for the spatial image gradient

Slide 28: Hessian Gauge  r  t    x    y 0 c01 c02    0  f rx  f tx   f xx   f xy C  BA   0 0 c12  0  c12 0  0  f ry  f ty   f xy   f yy    r  1 f ty f xy  f tx f yy f tx f xy  f ty f xx   t    1    c01  a p dr   p dp   p dq x   2  0 1 0   x   Ai   f xx f yy  f xy  c02  aq dr   q dp   q dq     y 0 0 1  y  c12   p dp   q dq  r  1 0 0   r  a p   f rrp f pp   1     p    0 cos   sin    x   BAi aq   f rrq f qq   fx  fy  2 2  q  0 sin  cos    y     p   f rpp f pp f xy  q   f rqq f qq tan 2  f yy  f xx  p   f rpq f pp  q   f rpq f qq C  BA  C  B   BC  A B 1   f rpq /( 2 f pp  2 f qq )

Slide 29: Real Image Sequences • Localized filters are not invariant w.r.t. Galilean shear, velocity adapted (s – directed) filters are needed • More generic singular cases for imprecise measurements

Slide 30: Conclusion • Theory about differential invariants for smooth Galilean spatio-temporal image sequences • Local operators • “Bottom up” • Contains more information about the image sequence than optical flow • Extension of methods for still images