Image Processing and Cartography with the NASA Vision Workbench

2,519 views
2,301 views

Published on

These are the slides from a talk I gave about the NASA Vision Workbench at the FOSS4G conference at the end of 2007. For a more up-to-date discussion of the Vision Workbench, see this presentation instead: http://www.slideshare.net/mdhancher/the-nasa-vision-workbench-reflections-on-image-processing-in-c-presentation

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,519
On SlideShare
0
From Embeds
0
Number of Embeds
25
Actions
Shares
0
Downloads
126
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Image Processing and Cartography with the NASA Vision Workbench

  1. 1. Image Processing and Cartography with the NASA Vision Workbench Matthew D. Hancher Intelligent Systems Division NASA Ames Research Center September 26, 2007 Intelligent Systems Division NASA Ames Research Center
  2. 2. Talk Overview • Who We Are • Introduction to the Vision Workbench • Example Applications • FOSS and NASA Intelligent Systems Division NASA Ames Research Center
  3. 3. NASA Ames Research Center • NASA’s Silicon Valley research center • Small spacecraft • Supercomputers • Intelligent Systems • Human Factors • Thermal protection systems • Aeronautics • Astrobiology Intelligent Systems Division NASA Ames Research Center
  4. 4. GIS & Imaging at Ames MASTER NASA World Wind (MODIS/ASTER simulator) NASA/Google Western States Planetary Content Fire Monitoring Mission Intelligent Systems Division NASA Ames Research Center
  5. 5. IRG & ACES Intelligent Robotics Adaptive Control & Group Evolvable Systems Group Intelligent Systems Division NASA Ames Research Center
  6. 6. Intro to the Vision Workbench Intelligent Systems Division NASA Ames Research Center
  7. 7. NASA Vision Workbench • Open-source image processing and machine vision library in C++ • Developed as a foundation for unifying raster image processing work at NASA Ames • A “second-generation” C++ image processing library, drawing on lessons learned by VXL, GIL, VIGRA, etc. • Designed for easy, expressive coding of efficient image processing algorithms Intelligent Systems Division NASA Ames Research Center
  8. 8. Open-Source VW Modules • Core: Low-level types & platform support • Math: General-purpose mathematical tools VW “Foundation” • Image: Basic image operations, filters, etc. Modules • FileIO: Simple, flexible image file IO layer • Camera: Camera models & related tools • Cartography: Geospatial image manipulation • Mosaic: Image mosaicing & multi-band blending • HDR: High-dynamic-range imaging (Open source as of now) Intelligent Systems Division NASA Ames Research Center
  9. 9. VW Modules Underway • InterestPoint: Interest point detection & matching • Stereo: Stereo correlation & 3D reconstruction • Python: Python bindings to many VW capabilities • GPU: GPU-accelerated image operations • Texture: Texture analysis & matching • Display: Image display and user interaction (The first four to be released later this year) Intelligent Systems Division NASA Ames Research Center
  10. 10. Design Goals & Approach • A simple, clean API for easy hacking • Simple syntax: Write what you mean! • Easy to manipulate arbitrarily large images • Automatic memory management • Generates high-performance code • Optimized processing via lazy evaluation • Function inlining via “generic” (template-based) C++ style Intelligent Systems Division NASA Ames Research Center
  11. 11. API Philosophy • Simple, natural, mathematical, expressive • Treat images as first-class mathematical data types whenever possible • Example: IIR filtering for background subtraction background += alpha * ( image - background ); • Direct, intuitive function calls • Example: A Gaussian smoothing filter result = gaussian_filter( image, 3.0 ); Intelligent Systems Division NASA Ames Research Center
  12. 12. Image Module Basics Intelligent Systems Division NASA Ames Research Center
  13. 13. Under the Hood: Image Views • The core “image view” concept: • Can be evaluated at a location to return a pixel value • Has a width and height in pixels • Cannonical example: the ImageView class • ImageView<PixelRGB<uint8> > image(1024,768); • Data processing represented as views • image2 = gaussian_filter(image1, 3.0); • Lazy container for arbitrary views • ImageViewRef<PixelRGB<uint8> > image3 = gaussian_filter(image1, 3.0); Intelligent Systems Division NASA Ames Research Center
  14. 14. Image Views II • Eliminates unnecessary temporaries • background += alpha * ( image - background ); • Supports procedurally generated images • image2 = fixed_grid(10,10,white,black,1024,768); • Allows greater control over processing • image2 = block_rasterize( gaussian_filter(image1, 3.0) ); • Views of images on disk • DiskImageView<PixelRGB<uint8> > disk_image(filename); Intelligent Systems Division NASA Ames Research Center
  15. 15. Applications & Modules Intelligent Systems Division NASA Ames Research Center
  16. 16. GigaPan Panorama Stitcher (As featured in the GigaPan layer in Google Earth.) Intelligent Systems Division NASA Ames Research Center
  17. 17. Mosaic Module • ImageComposite • Composite an arbitrary number of arbitrarily large images • It’s “just another image view” • Supports multi-band blending for seamless composites • QuadTreeGenerator • Generates a tiled pyramid representation of an arbitrary image view on disk • Great for building e.g. KML superoverlays or TMS maps Intelligent Systems Division NASA Ames Research Center
  18. 18. Cartographic Reprojection (As seen in the newly updated Google Moon.) Intelligent Systems Division NASA Ames Research Center
  19. 19. Cartography Module • GeoReference • Uses PROJ.4 for standard projections, GDAL to read/write • GeoTransform • Reprojects image data between GeoReferences • Makes “just another image view” • OrthoImageView • Ortho-rectifies an aerial or satellite image against an arbitrary DEM (in conjunction with the Camera module). • Also “just another image view” Intelligent Systems Division NASA Ames Research Center
  20. 20. Automated Image Alignment • Problem: Given two images, find and align the overlap region. Intelligent Systems Division NASA Ames Research Center
  21. 21. Image Alignment w/ Interest Points Point correspondencesto be aligned image Locate interest points inin first image Locate interest points second alignment Images determine image Intelligent Systems Division NASA Ames Research Center
  22. 22. Interest Point Module • Interest point detectors, descriptors, and matching ScaledInterestPointDetector<LoGInterest> detector; InterestPointList ip1 = interest_points( image1, detector ); InterestPointList ip2 = interest_points( image2, detector ); PatchDescriptor descriptor; compute_descriptors( image1, ip1, descriptor ); compute_descriptors( image2, ip2, descriptor ); DefaultMatcher matcher(threshold); InterestPointList matched1, matched2; matcher.match( ip1, ip2, matched1, matched2 ); Matrix2x2 homography = ransac( matched1, matched2, SimilarityFittingFunctor(), InterestPointErrorMetric() ); Intelligent Systems Division NASA Ames Research Center
  23. 23. The Ames Stereo Pipeline Fast, high quality, automated stereogrammetric surface reconstruction originally developed for Mars Pathfinder science operations Disparity Now a Vision Workbench application. Intelligent Systems Division NASA Ames Research Center
  24. 24. The Ames Stereo Pipeline Primary Image Secondary Image Ephemeris or Registration Automated Interest Points Mask / Sign of Laplacian of Gaussian Fast Stereo Correlation Outlier Rejection / Hole Filling / Smoothing Disparity Map Camera Model (e.g. Linear Pushbroom) Point Cloud/DTM Mesh Generation 3D Mesh Surprise: It’s all just Vision Workbench image views! Intelligent Systems Division NASA Ames Research Center
  25. 25. Mars Stereo: MOC NA MGS MOC-Narrow Angle • Malin Space Science Systems • Altitude: 388.4 km (typical) • Line Scan Camera: 2048 pixels • Focal length: 3.437m • Resolution: 1.5-12m / pixel • FOV: 0.5 deg Intelligent Systems Division NASA Ames Research Center
  26. 26. NE Terra Meridiani !% quot;quot; #$ !! $ # quot; quot;quot; quot;quot; !% #$ !!quot;quot;quot; $ !%quot;quot;#$ Upper Left: This DTM was generated from MOC images E04-01109 and M20-01357 (2.38°N, 6.40°E). The contour lines (20m spacing) overlay an ortho-image generated from the 3D terrain model. Lower Right: An oblique view of the corresponding VRML model. Intelligent Systems Division NASA Ames Research Center
  27. 27. Preliminary MOLA Comparison Elevation at boresight pixel (m) Scanline Capture Time (s) Intelligent Systems Division NASA Ames Research Center
  28. 28. Lunar Stereo: Apollo Orbiter Cameras ITEK Panoramic Camera • Focal length: 610 mm (24”) • Optical bar camera • Apollo 15,16,17 Scientific Instrument Module (SIM) • Film image: 1.149 x 0.1149 m • Resolution: 108-135 lines/mm Intelligent Systems Division NASA Ames Research Center
  29. 29. Apollo 17 Landing Site Top: Stereo reconstruction Right: Handheld photo taken by an orbiting Apollo 17 astronaut Intelligent Systems Division NASA Ames Research Center
  30. 30. Public Outreach: Hayden Planetarium Intelligent Systems Division NASA Ames Research Center
  31. 31. Public Outreach: Hayden Planetarium Intelligent Systems Division NASA Ames Research Center
  32. 32. Application: Image Matching • Problem: Given an image, find others like it. Example database: Apollo Metric Camera images Intelligent Systems Division NASA Ames Research Center
  33. 33. Texture-Based Image Matching Model image Texture bank filtering Filtering (Gaussian 1st derivative and LOG) Grouping to remove orientation Output Representation Energy in a window E-M Gaussian mixture model Segmentation Iterative tryouts, MDL Max vote Post-processing Grouping Summarization Mean energy in segment Euclidian distance Vector Comparison Matched image Intelligent Systems Division NASA Ames Research Center
  34. 34. Image Matching: Results Intelligent Systems Division NASA Ames Research Center
  35. 35. FOSS and NASA Intelligent Systems Division NASA Ames Research Center
  36. 36. The NOSA • The NASA Open Source Agreement, an OSI-approved non-viral open source license • Intended to protect users from contributor patent licensing issues. • Yes, we know: The current version (1.3) has several well-known peculiarities. Intelligent Systems Division NASA Ames Research Center
  37. 37. U.S. Contractor Rights • The University and Small Business Patent Procedures Act of 1980, a.k.a. “Bayh-Dole”. • A university, small business, or non-profit can claim patent ownership of a federally-funded invention before the government. • The government must actively promote and attempt to commercialize the invention. • Severely complicates open-source initiatives within the government that involve universities, small businesses, or non-profits. Intelligent Systems Division NASA Ames Research Center
  38. 38. The Open Source Process • Open-source approval stages include: • Invention disclosure • Copyright assignment (all parties) • Legal review (copyright & patent issues) • Export control review (e.g. ITAR) • Computer security review • more.... Intelligent Systems Division NASA Ames Research Center
  39. 39. Signs of Improvement • The old model: (e.g.VW 1.0) • Seek approvals after code completion • Long, slow, high-latency release cycle • The new model: ?? (e.g. WV 2.0 ??) • Seek periodic approval for upcoming development • Allows regular updates within prescribed bounds • On the horizon: ?? • User contribution process ? • Publicly-accessible subversion repository ??? Intelligent Systems Division NASA Ames Research Center
  40. 40. Free and Open Data • Free and open data has received much less attention than free and open software. • The National Aeronautics & Space Act: • The Administration, in order to carry out the purpose of this Act, shall... provide for the widest practicable and appropriate dissemination of information concerning its activities and the results thereof. • Alas, NASA does not own much of what is often imagined to be “NASA data”. Intelligent Systems Division NASA Ames Research Center
  41. 41. Outreach: Google Earth Astronaut Photography MODIS Coverages • Make more datasets publicly available as KML (and soon WMS) for mash-ups. • Increase the visibility of existing public repositories of NASA data and imagery. Intelligent Systems Division NASA Ames Research Center
  42. 42. Outreach: Google Moon Data coming soon via KML and WMS from NASA. Intelligent Systems Division NASA Ames Research Center
  43. 43. Obtaining the Vision Workbench • VW version 1.0.1 available now. • VW version 2.0 coming this fall! http://ti.arc.nasa.gov/visionworkbench/ • To contact me: Matthew.D.Hancher@nasa.gov Intelligent Systems Division NASA Ames Research Center

×