Taubin CIARP 2012


Published on

Keynote lecture given by Gabriel Taubin at CIARP 2012 (http://www.ciarp.org) on Wednesday, September 5, 2012, in Buenos Aires, Argentina

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Taubin CIARP 2012

  1. 1. Smooth  Signed  Distance  Surface  Reconstruc4on  and  Applica4ons   Gabriel  Taubin   Brown  University     CIARP  05  Sep  2012  
  2. 2. Typical  Surface  Reconstruc3on  Pipeline  Oriented   Reconstruc4on   Surface   Points     Method   Representa4on  Posi4ons  &  Normals   Implicit  Surface                              Polygon  Mesh      
  3. 3. Numerous  Applica3ons   •  Industry   –  Reverse  engineering   –  Fast  metrology   –  Physical  simula4ons  •  Entertainment     –  Anima4ng  digital  clays  for   movies  or  games  •  Archeology  and  Art   –  Digi4za4on  of  cultural     heritage  and  ar4s4c  works  •  Medical  Imaging   –  Visualiza4on   –  Segmenta4on  •  …  
  4. 4. Methods  to  Capture  Oriented  Points   Laser  Scanning     Oriented   Points     Structured  Ligh4ng   Mul4-­‐View  Stereo     Posi4ons,  Normals,  Colors  
  5. 5. Data  Acquisi3on  Laser  range  scanning   Mul4-­‐View  Stereo   Structured  Ligh4ng  
  6. 6. Gray  Code  Structured  Ligh3ng  Results  
  7. 7. hBp://mesh.brown.edu/byo3d/  
  8. 8. hBp://mesh.brown.edu/byo3d  
  9. 9. Surface  Reconstruc3on  from  Mul3-­‐View  Data  
  10. 10. SSD:  Smooth  Signed  Distance  Surface  Reconstruc3on   F.  Calakli,  G.  Taubin,  Computer  Graphics  Forum,  2011.    •  Con4nuous  mathema4cal  formula4on  •  Par4cular  discre4za4on  and  algorithm  •  Reconstructs  water4ght  surface  as  polygon  mesh  •  From  a  sta4c  oriented  point  cloud  
  11. 11. hBp://mesh.brown.edu/ssd  
  12. 12. Related  Papers  &  Projects  •  Vector  Field  Isosurface-­‐Based  Reconstruc4on  From  Oriented  Points,  by   Sibley  &  Taubin,  Siggraph  2005  (Sketch).  •  Smooth  Signed  Distance  Surface  Reconstruc4on,  by  Calakli  &  Taubin,  PG   2011  &  Computer  Graphics  Forum  2011.  •  Smooth  Signed  Distance  Colored  Surface  Reconstruc4on,  by  Calakli  &   Taubin,  chapter  in  State-­‐of-­‐the-­‐Art  Volume  on  Computer  Graphics,   Visualiza4on,  Visual  Analy4cs,  VR  and  HCI,  2012.  •  Accurate  3D  Footwear  Impression  Recovery  from  Photographs,  by  Andalo,   Calakli,  Taubin,  and  Goldenstein,  Proceedings  of  the  4th.  Interna4onal   Conference  on  Imaging  for  Crime  Detec4on  and  Preven4on  (ICDP-­‐2011).  •  High  Resolu4on  Surface  Reconstruc4on  from  Mul4-­‐view  Aerial  Imagery,   Calakli,  Ulusoy,  Restrepo,  Mundy  &  Taubin,  3DIMPVT  2012  •  REVEAL  Digital  Archaeology  Project    •  Cuneiform    Automa4c  Transla4on  Project  
  13. 13. Par3cularly  Good  at  Extrapola3ng  Missing  Data  
  14. 14. Implicit  func3on  framework   <  0   0   Z( f ) >  0   Oriented  Points,  D     Computed  Implicit  Surface,  S’  (samples  from  unknown  surface  S)   Find    a  scalar  valued  func4on    f    :    D    →    ℜ  ,  whose  zero                     level  set  Z(f)  =  S’  is  the  es4mate  for  true  surface  S  
  15. 15. Implicit  Curve  and  Surface  Reconstruc3on  •  Input:  oriented  point  set:        D  =  {  (  pi,  ni  )  i=1,…,N}          contained  in  a  bounding  volume  V  •  Output:  implicit  surface      S  =  {  x  |  f  (x)  =  0  }          with  the  func4on  defined  on  V,  such  that              f  (pi)  =  0      and      ∇f  (pi)  =  ni    ∀(pi,ni)  ∈  D  •  A  family  of  implicit  func4ons  with  a  finite  number  of   parameters  has  to  be  chosen  •  Parameters  must  be  es4mated  so  that  the  condi4ons   stated  above  are  sa4sfied,  if  not  exactly,  then  in  the   least-­‐squares  sense  
  16. 16. Challenges   Uniform  sampling  Non-­‐uniform  sampling   Noisy  data   Misaligned  scans  
  17. 17. General  Approaches  •  Interpola4ng  polygon  meshes   Boissonnat  [1984],  Edelsbrunner  [1984]   Amenta  et  al.  [1998],  Bernardini  et  al.  [1999]   Dey  et  al.  [2003][2007],  …  •  Implicit  func4on  fijng   Taubin  [1991],  Hoppe  et  al.  [1992],  Curless  et  al.  [1996]   Whitaker  [1998],  Carr  et  al.[2001],  Davis  et  al.  [2002],     Ohtake  et  al.  [2004],  Turk  et  al.  [2004],  Shen  et  al.  [2004]   Sibley-­‐Taubin  [2005]    
  18. 18. Poisson  Surface  Reconstruc3on      Kazhdan  et  al.  [2006]                          Manson  et  al.  [2008]    
  19. 19. Poisson  Surface  Reconstruc3on  1.  Extend  oriented  points  to  con4nuous  vector   field    v(  p)    defined  on  the  whole  volume,  so               that   v(p i ) ≈ ni2.  Integrate  vector  field,  by  minimizing   2 ∫ V ||∇f ( p) − v( p) || dp3.  Determine  isolevel,  by  minimizing   N ∑ ( f (p i ) − f0 )2 i=1
  20. 20. Main  problem  with  this  approach  
  21. 21. Main  problem  with  this  approach  
  22. 22. Main  problem  with  this  approach  
  23. 23. Main  problem  with  this  approach  
  24. 24. What  kind  of  implicit  func3on?  0   1   0   Indicator  Func4on   Smooth  Signed  Distance  Func4on  
  25. 25. Vectorfield  Isosurface-­‐Based  Reconstruc3on  From  Oriented  Points   P.  Sibley  and  G.  Taubin  [Siggraph  2005  Sketch]  •  Surface  reconstruc4on  from  cloud  of  oriented  points  •  Implicit  representa4on  can  deal  with  missing  data  •  Rather  than  fijng  analy4c  func4on  (RBFs,  etc),  and  then   extract  isosurface  for  visualiza4on,  fit  isosurface  directly  to  data  
  26. 26. Vectorfield  Isosurface-­‐Based  Reconstruc3on  From  Oriented  Points   P.  Sibley  and  G.  Taubin  [Siggraph  2005  Sketch]  •  Surface  reconstruc4on  from  cloud  of  oriented  points  •  Implicit  representa4on  can  deal  with  missing  data  •  Rather  than  fijng  analy4c  func4on  (RBFs,  etc),  and  then   extract  isosurface  for  visualiza4on,  fit  isosurface  directly  to  data  •  Ini4al  approach:  3  steps   1.  Fit  con4nuous  vector  field  to  data  normal  vectors   2.  Integrate  vector  field  on  regular  grid   3.  Determine  Isolevel  
  27. 27. VFIso:  Representa3on  •  Regular  voxel  grid    •  Implicit  func4on  defined  by  values  at  ver4ces  •  And  tri-­‐linear  interpola4on   f ( p ) =∑ f ϕ ( p) α α α•  The  output    is:   { fα } fα ϕα (p)
  28. 28. VFIso:  Varia3onal  Approach  •  Extend  data  to  vector  field  on  the  volume  minimizing   2 2 E (v ) = ∑ v ( p) − n + λ ∑ v α − v β   D   (  α ,β )   data regulariza tion where : v ( p) = ∑ v α ϕα ( p) α•  Then  integrate  vector  field  ?    
  29. 29. VFIso:  Details  •  Problem:  Vector  field  v(p)  not  integrable!  •  Instead:  Fit  scalar  field    {    f  α  }    directly  by  minimizing               2 2 E ({ fα }) = ∑ ∇f ( p ) − n + λ ∑ ∇fα − ∇f β D (α , β )•  Where   ∇f ( p) = ∑α ∇fα ϕα ( p)•  And      ∇  f  α      is  discre4zed  using  central  differences        •  This  is  a  standard  (Laplacian)  Least  Squares  problem  •  Minimize  to  find  isolevel   E ( f 0 ) = 0 2 ∑ f ( p) − f D
  30. 30.          Squirrel(9K)                Angel(24K)                  Bunny(35K)            Ram(678K)  
  31. 31. Some  Methods  to  Capture  3D  Point  Clouds  Multi-Flash Camera Shadow Multi-Flash Backdrop Attachment Turntable8 Megapixel Camera
  32. 32. Beyond  SilhoueBes:  Surface  Reconstruc3on   using  Mul3-­‐Flash  Photography  D.  Crispell,  D.  Lanman,  P.  Sibley,  Y.  Zhao  and  G.  Taubin  [3DPVT  2006]  
  33. 33. Multi-Flash RecoveredMulti-Flash 3D Photography: Turntable Sequence: Estimated Shape: 3D Point Cloud Appearance:Capturing the Shape and Appearance Input Image Phong BRDF Modelof 3D ObjectsA new approach for reconstructing 3D objectsusing shadows cast by depth discontinuities, asdetected by a multi-flash camera. Unlike existingstereo vision algorithms, this method works evenwith plain surfaces, including unpainted ceramicsand architecture.Data Capture: A turntable and a digital cameraare used to acquire data from 670 viewpoints. Foreach viewpoint, we capture a set of images usingillumination from four different flashes. Futureembodiments will include a small, inexpensivehandheld multi-flash camera. Mul3-­‐Flash  Camera   Shadow   Turntable   Rota3on   Recovering a Smooth Surface Backdrop   Mul3-­‐Flash   ABachment   The reconstructed point cloud can possess errors, including gaps and noise. To minimize these effects, we find an implicit surface which interpolates the Turntable   3D points. This method can be applied to any 3D point cloud, including those 8  Megapixel   Camera   generated by laser scanners. 33  
  34. 34. VFIso  Results  [2006  110x110x110  grid]  
  35. 35. SSD  Con3nuous  Formula3on  •  Oriented  point  set:        D  =  {  (  pi,  ni  )  }    sampled  from  a  surface  S  •  Implicit  surface:      S  =  {  x  |  f  (x)  =  0  }  such  that              f  (pi)  =  0      and      ∇f  (pi)  =  ni    ∀(pi,ni)  ∈  D  •  Least  squares  energy:     N N N N 2 p 2 +λ || Hf (x) || 2 dx 2 E( f ) = ∑ f (E(i f)) = ∑1f (p i )∇+ λp i ) − n(p i ) + n i2 2 f ( 1 ∑ ∇f i − λ ∑ ∫ i=1 i=1 V i =1 i =1
  36. 36. What  does  the  regulariza3on  term  do  ?  •  Near  data  points:  since  the  data  terms  dominate,  the   func4on  approximates  the  signed  distance  •  Away  from  data  points:  the  regulariza4on  term   dominates  and  forces  the  gradient  to  be  smooth  and   close  to  constant  
  37. 37. Role  of  each  energy  term   N N 2E ( f , v, M ) = ∑ f (p i ) + λ1 ∑ v(p i ) − n i + λ2 ∫ || M (x) ||2 dx 2 V i =1 i =1 Quadra4c  energy  in  f,  v,  and  M  If  f,  v,  and  M  are  linear  func3ons  of  the  same  parameters,   then  the  minimiza4on  reduces  to  a  least  squares  problem    
  38. 38. Linear  families  of  func3ons  •  Popular  Smooth  Basis  Func4ons   –  Monomials  [Taubin’91]   –  Radial  basis  func4ons  [Carr  et  al.,  ‘01],     –  Compactly  supported  basis  func4ons  [Othake  et  al.  ‘04],   –  Trigonometric  polynomials  [Kazhdan  et  al.  ‘05],   –  B-­‐splines  [Kazhdan  et  al.,  06],     –  Wavelets  [Manson  et  al.  ‘08],       t t Non-­‐homogenous,     Quadra4c  energy   E ( F ) = F AF − 2b F + c Global  minimum   AF = b
  39. 39. We  can  use  Independent  Discre3za3ons  •  Hybrid  FE/FD  discre4za4on     –  Trilinear  interpolant  for  the  func4on  f(x)   –  Primal  finite  differences  for  the  gradient  ∇f(x)   –  Dual  finite  differences  for  the  Hessian  Hf(x)  •  As  long  as  f(x),  ∇f(x),  and  Hf(x)  are  wrixen  as  a   linear  combina4ons  of  the  same  parameter  vector  F   Non-­‐homogenous,     t t Quadra4c  energy   E ( F ) = F AF − 2b F + c Global  minimum   AF = b
  40. 40. Implementa3on  •  Primal-­‐Dual  octree  data  structure  •  Cascading  mul4-­‐grid  itera4ve  solver   (conjugate  gradient):   Solve  the  problem  on  a  much  coarser  level   –  Use  the  solu4on  at  that  level  to  ini4alize  the   solu4on  at  the  next  level     –  Refine  with  the  itera4ve  solver  •  Iso-­‐surface  extrac4on  (crack-­‐free)   –  Dual  marching  cubes  [Schaefer  2005]  
  41. 41. Marching  Cubes  on  Octrees  •  Non-­‐conforming  hexahedral  mesh    •  Results  in  crack  problem.  •  Problem  solved  by  Dual  Marching  Cubes  
  42. 42. Input  point   MPU     Poisson   D4  Wavelets   SSD  cloud   [Othake  ‘03  ]    [Kazhdan  ‘06  ]    [Manson  ‘08  ]   [Calakli  ’11]  
  43. 43. SSD  Surface  Reconstruc3on  •  Theore4cal  contribu4ons:   –  Oriented  point  samples  regarded  as  samples  of   Euclidean  signed  distance  func4on   –  Reconstruc4on  as  global  minimiza4on  problem   –  Yet  sparse  system  of  linear  equa4ons  •  Empirical  advantages:   –  Robust  to  noise  and  uneven  sampling  density  •  Future  work:   –  Streaming  out-­‐of-­‐core  implementa4ons   –  Parallel/Mul4-­‐core/GPU  implementa4ons   –  Dynamic  shapes  
  44. 44. REVEAL  Digital  Archaeology  Project   Cooper,  Kimia,  Taubin,  Galor,  Sanders,  Willis  •  The  main  goal  is  to  automate  the  tedious  processes  of   data  collec4on  and  documenta4on  at  the  excava4on   site,  as  well  as  to  provide  visualiza4on  tools  to  explore   the  data  collected  in  the  database  •   Also  to  solve  specific  problems  in  Archaeology  using   computer  vision  techniques.  •  We  first  used  a  network  of  cameras  to  capture  the   ac4vity  at  the  excava4on  site,  to  reconstruct  the  shape   of  the  environment  as  it  is  being  excavated,  to   reconstruct  layers,  and  to  locate  finds  in  3D  •  Then  we  switched  to  mul4-­‐view  stereo  with  photos   captured  with  hand-­‐held  cameras  
  45. 45. REVEAL Archaeological Data AcquisitionAssisted Data Acquisition, Algorithmic Reconstruction, Integrated multi-format analysis Data  Acquisi3on   Advanced  Algorithms   Improve   Automa3cally  Convert  Photos  to  3D  Models   speed  and   accuracy  with   computer   assisted  data   entry   Import  photos,   videos,  and  laser   Semi-­‐automa3cally       scans  and   Assemble       connect  them  to     Fragments     database   Into  Ar3facts   objects   Import  External  Data   Laser  Scanned   Site  Plans   Models   REVEAL   Database   Objects:   Data:   Ar4facts   Text   Excava4ons   Photos   Areas   Video   Rule-­‐based     Sites   3D  Models   Reconstruc3ons   hBp://sourceforge.net/projects/revealanalyze  
  46. 46. REVEAL Archaeological Analysis Data  integrated  and  synchronized  in  tabular,  plan  drawing,  3D  spa3al,  image,  and  video  formats  Typical Activity Sequence Examine  Rela3onship  of  Ar3facts  in-­‐situ  in  auto-­‐ generated  3D  Excava3on  Model     Display  Photos  of   Selected  Ar3facts   Ar4facts   Export   Excava4ons   FormaBed  Select  Ar3facts   Areas   Ar3fact  Data  for  on  Site  Plan   Sites   inclusion  in  Site   Publica3on  
  47. 47. [Snavely    et.  al.  2006]  hxp://phototour.cs.washington.edu/bundler/   MVS     soaware  Patch-­‐based  Mul4-­‐View  Stereo  (PMVS)  hxp://grail.cs.washington.edu/soyware/pmvs/   [Furukawa  and  Ponce  2008]  
  48. 48. Accurate  3D  Footwear  Impression  Recovery  From  Photographs,   F.  A.  Andalo,  F.  Calakli,  G.  Taubin,  and  S.  Goldenstein,   Interna4onal  Conference  on  Imaging  for  Crime  Detec4on  and   Preven4on  (ICDP-­‐2011).     Comparable  to  3D  Laser  Scanner  
  49. 49. Handheld  Interac4ve,  Incremental  3D  Object  Scanning   Kim,  Honors  Thesis,  Brown  University,  May  2012  •  Based  on  MS  Kinect   sensor  •  Con4nuous  coarse  pose   es4ma4on  from  color   camera  using  PTAM  •  3D  snapshots  captured   from  different  points  of   view  using  depth  camera  •  Alignment  improved  with   Itera4ve  Closest  Points   (ICP)  algorithm  •  SSD  is  run  on  aligned  3D   snapshots  
  50. 50. Real-­‐Time  High-­‐Defini3on  Stereo  on  GPGPU   using  Progressive  Mul3-­‐Resolu3on  Adap3ve  Windows  Y.  Zhao,  and  G.  Taubin,  Image  and  Vision  Compu4ng  2011.     Screen  shots  of  our  real-­‐3me  stereo  system  working  on  the  field  
  51. 51. Real-­‐Time  High-­‐Defini3on  Stereo  on  GPGPU   using  Progressive  Mul3-­‐Resolu3on  Adap3ve  Windows   Y.  Zhao,  and  G.  Taubin,  Image  and  Vision  Compu4ng  2011.     Stereo  Frame   Stereo   Mul3-­‐Resolu3on   Grabbing   Rec3fica3on   Pyramid  Genera3ng   Resolu3on  Scan  from  Low  to   Background  Modeling  With  Shadow  Removal     Foreground  Detec3on  with  Dila3on  &  Erosion     High   Stereo  Matching  Using  Adap3ve  Window  with   Cross-­‐Checking   Disparity  Refinement  ¼ Resolution ½ Resolution Full Resolution Coarse-­‐to-­‐fine  matching  on  mul;ple  resolu;ons   Processing  Pipeline  
  52. 52. High  Resolu3on  Surface  Reconstruc3on  from  Mul3-­‐view  Aerial  Imagery   by  Calakli,  Ulusoy,  Restrepo,  Mundy  &  Taubin,  3DIMPVT  2012  
  53. 53. Microscopic  3D  Shape  Capture  Liberman  &  Taubin  (work  in  progress)   ~  4  mm  
  54. 54. Ques4ons?  This  material  is  based  upon  work  supported  by  the  Na4onal  Science  Founda4on  under  Grants  CCF-­‐0729126,  IIS-­‐0808718,  CCF-­‐0915661,  and   IIP-­‐1215308.