Augmented	
  reality	
  techniques
Sergi	
  Grau	
  Carrión	
  
http://sergigrau.cat	
  
@sergi_grau
Adapted	
  milgram's	
  VR-­‐AR	
  
continuum

http://en.wikipedia.org/wiki/File:Adapted_milgrams_VR-­‐AR_continuum.png
Virtual	
  reality	
  -­‐	
  Immersive

http://www.youtube.com/watch?v=yf0sllpZx3w
Virtual	
  reality	
  -­‐	
  Immersive

http://www.youtube.com/watch?v=WYG7xGFByJ8
Virtual	
  reality	
  -­‐	
  Semi	
  immersive

http://www.youtube.com/watch?v=7MVyT1lgGGQ
Adapted	
  milgram's	
  VR-­‐AR	
  
continuum

http://en.wikipedia.org/wiki/File:Adapted_milgrams_VR-­‐AR_continuum.png
Augmented	
  reality

Started	
  in	
  1957	
  (Roosevelt	
  home)
Augmented	
  reality

http://www.youtube.com/watch?v=DjevdT5XpFw
Augmented	
  reality
Augmented	
  reality
Augmented	
  reality
Augmented	
  reality

See-­‐through
Augmented	
  reality

Monitor	
  based
Augmented	
  reality
Augmented	
  reality
Augmented	
  reality
Adapted	
  milgram's	
  VR-­‐AR	
  
continuum

http://en.wikipedia.org/wiki/File:Adapted_milgrams_VR-­‐AR_continuum.png
TUI
TUI

http://www.ted.com/talks/david_merrill_demos_siftables_the_smart_blocks.html
TUI

https://vimeo.com/82277562
Adapted	
  Virtuality-­‐Reality	
  
continuum

http://en.wikipedia.org/wiki/File:Adapted_milgrams_VR-­‐AR_continuum.png
Augmented	
  Reality
Capture	
  real	
  world	
  
Computer	
  Vision

Capture	
  virtual	
  world	
  
Computer	
  Graphics

http://wearables.unisa.edu.au
Diminished	
  reality

http://www.youtube.com/watch?v=FgTq-­‐AgYlTE
Diminished	
  reality

http://tangible.media.mit.edu/project/focalspace/
Ingredients	
  for	
  successful	
  AR	
  
http://www.youtube.com/watch?v=yDjHbLjtL-­‐c
Environment	
  analysis
+
Realistic	
  augmentation
Environment	
  analysis

Real-­‐time	
  3D	
  tracking	
  of	
  user	
  viewpoint	
  w.r.t.	
  World
•
•
•
•
•

Geolocalization	
  
Electro-­‐magnetic	
  
Acoustic	
  
Inertial	
  
Vision-­‐based	
  (active	
  or	
  passive)
Image-­‐based	
  3D	
  tracking

Inside-­‐out
Image-­‐based	
  3D	
  tracking

Outside-­‐in
OpenCV

http://www.youtube.com/watch?v=Z43_hCM74rU
Color	
  tracking
Hand	
  tracking

http://www.youtube.com/watch?v=B4dwu3si9x0
Hand	
  Tracking

http://www.youtube.com/watch?v=CYJjQgV0mZY
Optical	
  Flow

Optical	
  Flow
http://www.youtube.com/watch?v=5rR_9YIcg_s
Optical	
  Flow

http://www.youtube.com/watch?v=1D93RmW_eN4
Optical	
  Flow

http://www.youtube.com/watch?v=C95bngCOv9Q
Optical	
  Flow

http://www.youtube.com/watch?v=0L34HsXXU00
Face	
  tracking

http://www.youtube.com/watch?v=JyBMxeVCQkc
Head	
  tracking
Face	
  tracking

http://www.youtube.com/watch?v=h9kPI7_vhAU
Face	
  tracking
Face	
  tracking
Face	
  substitution

http://auduno.github.io/clmtrackr/examples/facesubstitution.html
Image-­‐based	
  3D	
  reconstruction
Computer	
  vision	
  is	
  good	
  at	
  location	
  features:	
  
Points	
  
Regions	
  
Textures	
  
Contours	
  
...
Image-­‐based	
  3D	
  reconstruction
Geometry
3D	
  reconstruction:	
  what	
  is	
  where?
Semantic:	
  augmentation	
  of	
  context	
  
Object	
  recognition	
  
Interpretation	
  of	
  content
3D	
  reconstruction	
  of	
  regions
Based	
  on	
  planar	
  equations.	
  
– Hypothesis:	
  world	
  is	
  piece-­‐wise	
  planar

3D	
  registration	
  by	
  Iterative	
  Closest	
  Point	
  (ICP)
ARToolkit

http://www.youtube.com/watch?v=VR12e7WnkxU
ARToolkit

http://www.youtube.com/watch?v=YWG4nhdsySE
ARToolkit
ARToolkit
Fast	
  and	
  cheap	
  6D	
  marker	
  tracking	
  
Easy	
  calibration	
  routine:	
  
– Measure	
  camera	
  properties:	
  
• Center	
  point	
  of	
  the	
  camera	
  image.	
  
• The	
  lens	
  distortion.	
  
• Camera	
  focal	
  length.	
  

!
!
!

Virtual	
  objects	
  appear	
  only	
  when	
  complete	
  markers	
  are	
  
visible.
Image	
  processing
Image	
  processing
Image	
  processing
Image	
  processing
Image	
  processing
Pattern	
  matching
Feature	
  tracking:	
  SIFT
Markerless	
  tracking
Depth	
  cameras

Infer	
  depth	
  for	
  ALL	
  pixels
Depth	
  cameras

Zcam	
  (3DV	
  Systems)	
  2007	
  –	
  1.3Megapixel	
  60fps
ZCam
Kinect

http://www.youtube.com/watch?v=YTBvjLGDluY
Kinect

http://www.youtube.com/watch?v=NqjopQmqWAE
Kinect
Kinect
Kinect
Motion	
  capture

http://people.csail.mit.edu/rywang/handtracking/handtracking.mov
Motion	
  capture
Markless	
  motion	
  capture

http://vimeo.com/33323400
http://structure.io
http://structure.io
Depth	
  Cameras

Solves	
  the	
  “where”	
  but	
  not	
  the	
  “what”:	
  	
  
objects	
  must	
  still	
  be	
  identified	
  (segmented)
Realistic	
  augmentation
Environment	
  analysis
+
Realistic	
  augmentation
Realistic	
  augmentation
!

2	
  components:	
  	
  
	
   Geometric	
  coherency	
   	
  
	
   Lighting	
  coherency	
  
	
  
CG	
  Blending
Approximate	
  real	
  world	
  to	
  simulate	
  interaction

Virtual	
  in	
  front	
  of	
  real

Virtual	
  behind	
  real
Light	
  coherency

http://www.kevinkarsch.com/publications/sa11.html
Light	
  coherency
Light	
  coherence

http://www.youtube.com/watch?v=P3roYysI6pM
Light	
  probe

Aim:	
  Capture	
  light	
  coming	
  from	
  light	
  sources	
  in	
  real	
  world
Light	
  probe
Light	
  probe

Half-­‐life	
  2
Light	
  coherence

http://www.youtube.com/watch?v=ihhKLex2Na8
Light	
  coherence

http://handheldar.icg.tugraz.at/
Camera	
  effects
AR	
  Hardware
Lumus	
  DK-­‐40	
  AR	
  Glasses

Oculus	
  Rift	
  ‘Crystal	
  Cove’

Heapsylon	
  Sensoria	
  Fitness	
  socks

LG	
  Lifeband	
  Touch

Edison-­‐	
  Intel‘s	
  Development	
  Board

Sony’s	
  ‘wearable	
  HDTV’

Sony	
  SmartBand

Pebble
Google	
  Glasses
ORA	
  Glasses
http://optinvent.com/
Epson	
  Moverio	
  BT-­‐100
Meta	
  Pro	
  Glasses
Lumus	
  HUD	
  Display
Telepathy	
  One
NTT	
  DOCOMO
http://www.reconinstruments.com
vrAse
Reevu	
  Intelligent	
  Helmet’s	
  heads	
  up	
  display
Intel	
  RealSense	
  3D	
  camera
Creative	
  Senz3D
SeeSpace	
  InAir
 i-­‐Air	
  Touch	
  Technology
Interaction	
  and	
  Collaboration
http://www.youtube.com/watch?v=RlHOhYF_jqM
Interaction	
  and	
  Collaboration
http://www.youtube.com/watch?v=Y3TzhXYF0Lg
Augmented	
  reality	
  techniques
Sergi	
  Grau	
  Carrión	
  
http://sergigrau.cat	
  
@sergi_grau

Ar techniques@sergi grau