Estlin aegissoyajpl 2012
Upcoming SlideShare
Loading in...5

Like this? Share it with your network

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Autonomous Exploration for Gathering Increased Science AEGIS Tara EstlinBenjamin Bornstein, Daniel Gaines, David R. Thompson, Rebecca Castano, Robert C. Anderson,  Michael Burl, Charles de Granville and Michele Judd 2011 NASA Software of the Year
  • 2. Consider the following problem…• You are a robotic explorer                                            on another planet • You only talk to Earth                                                            once a day• You are in a hostile                                       environment• You have limited power and computing abilities• You are constantly on the move exploring different  terrains• As you move, you need to quickly determine if you  see objects that are interesting to scientists• If you do, you want to acquire data on these  objects, before the rover moves past 2
  • 3. This is why we developed AEGIS AEGIS:• Is a new paradigm for in‐situ                     science using onboard autonomy• Provides intelligent targeting and  data acquisition by – analyzing images of the rover scene – identifying high‐priority science targets  (e.g., rocks), and – taking high quality data of these targets  completely autonomously with no  ground interaction required 3
  • 4. AEGIS Video 4
  • 5. How is AEGIS being used? AEGIS:• Is in regular operational use onboard                                    the Mars Exploration Rover (MER) Mission                                Opportunity rover for the past  two years• Excels in automated targeting with narrow field‐of‐ view (FOV) remote sensing instruments, such as: – MER Panoramic Cameras (in current use) – MER Mini‐Thermal Emission Spectrometer  – Mars Science Laboratory (MSL) Rover                                               ChemCam Spectrometer Mini-TES Mosaic – Before AEGIS, had to manually select                                    targets, based on ground analysis 5
  • 6. AEGIS Process for MER Process fully automated! Advanced image Navcam acquisition processing technique enables reliable, rapid identification of candidate targets. Target detectionAlgorithms quantify key Target feature Scientists canintuitive target extraction prioritizeproperties such as importantbrightness, size, and propertiesshape. for each run Target prioritization Target pointing determination Robust approach to Top score for large size pointing selection maximizes data of target. Pancam pointing High-quality, 13 color filter, quarter-frame Pancam acquisition Panoramic camera image
  • 7. Benefit of AEGIS for Rover Drive Autonomously‐targeted remote sensing taken end of drive by AEGIS X X Autonomously‐targeted remote  sensing taken mid‐drive by AEGIS Manually‐targeted remote sensing  as specified by science team (taken before drive) Targeted data Targeted data selected manually with AEGIS 7
  • 8. AEGIS identifies rock targetsScientist target profile: large rocks of high reflectance.Markers show top ten prioritized targets. 8
  • 9. AEGIS identifies rock targetsArea covered by MER Panoramic Camera. 9
  • 10. AEGIS identifies rock targetsArea covered by MER Panoramic Camera. 10
  • 11. AEGIS delivers 13F Pancam image after autonomously targeting the top priority rock 11
  • 12. MER before AEGIS Sol 1 Sol 2 Sol 3 Perform  MANUALLY TARGETED remote  Pancam Sol 2sensing of current  Perform  Perform  rover area untargeted   untargeted  Drive rover 100m  remote  remote to new stopping  sensing of  sensing of  point local area local area Acquire wide‐angle images of new terrain area Pancam Sol 3 Multi-Sol Plan 12
  • 13. MER after AEGIS Sol 1 Sol 2 Sol 3 Perform  MANUALLY TARGETED remote  AEGIS Pancam 1sensing of current  Perform  Perform  rover area autonomously  autonomously targeted targetedDrive rover 100m  remote  remote to new stopping  sensing of  sensing of  point local area local area Acquire wide‐angle images of new terrain area AEGIS Pancam 2 Multi-Sol Plan 13
  • 14. SIGNIFICANCE Scientists trust AEGIS to make intelligent  decisions about collecting new science.
  • 15. Significance: Aerospace• Fully operational and used                                 Opportunity Today regularly on MER mission – Saves valuable time every targeted                                                  data collection – Has been used more than any other                                                new technology on the Opportunity rover• Currently infusing into Mars Science  Laboratory (MSL) Mission• Attracting strong interest for  – 2018 Mars rover and other future in‐situ                             missions (e.g., Titan, Venus, Europa) – Military applications (e.g., UAVs) 15
  • 16. Significance: Science/Technology• Enables the collection of science data that would  otherwise not be possible – During or right after drives – Different times of day and temps• Saves significant time and cost                                for   targeted data collection – Gets data into the hands of scientists twice as fast                  (or more) than standard operations• Enables scientists to easily use and                                    interact with autonomy software  – Parameters chosen after significant                                      consultation with scientists• 25+ science and technology publications• Application to large number of problems in  industry and academia (e.g., underwater robotics) 16
  • 17. Significance: Humanitarian• Directly contributing to humanity goal of  finding life on other planets – Mars program theme of “Follow the water…”• Significant outreach vehicle; over 35 media  articles since release in 2010 – “Mars Rover Getting Smarter As It Gets Older” – “NASA upgrades Mars rover brain” 17
  • 18. Significance: Humanitarian Inspiring the next generation in STEM:Science, Technology, Engineering and Mathematics Over 46 Amateur Astronomy Clubs, Australia Schools and Teacher Organizations around United Kingdom the world featured our “rock hound” software in their newsletters 18
  • 19. DEVELOPMENT AEGIS is flight proven and fully operational
  • 20. Development Status • TRL Level 9: Flight Proven – Software fully operational  – In regular use on MER Opportunity rover • MER and MSL Scientists have already asked for  extensions, which are in progress: – Enabling multiple targeted observations – Triggering on single filter color images – Identifying novel targets – Identifying representative targets – “Soil only” detector“AEGIS is a true success story for the Mars Technology Program” Dr. Samad Hayati Manager of Mars Technology Program 20
  • 21. ASSESSMENT of USEAEGIS technology is being applied in a wide range of applications.
  • 22. NASA Use – MER AEGIS is considered every multi‐sol plan. Sol 2138 Sol 2172 Sol 2313 Sol 2221 Sol 2247 Sol 2278 Sol 2290 Sol 2304 Sol 2312“AEGIS is a significant enhancement for the mission and the scientific community. MER is the first mission to implement the capability that AEGIS provides – and it has really paid off.” Dr. John Callas 22 Mars Exploration Rover Mission Project Manager
  • 23. NASA Use – MSL Rover• The MSL Rover ChemCam Team has requested  AEGIS (PI: Roger Wiens)• AEGIS is ideal for ChemCam’s narrow field‐of‐view Laser‐Induced Breakdown Spectrometer (LIBS) – Samples rocks from a distance of 1 to 7 meters  – Able to rapidly identify rock elemental composition• AEGIS enables multiple autonomously targeted ChemCam measurements throughout the day• MSL flight software integration in progress 23
  • 24. NASA Use – MSLVideo clip from Dr. Roger WiensPrincipal InvestigatorMSL Rover Mission Chemcam Instrument 24
  • 25. NASA Use – MSL 25
  • 26. NASA Use – Mars 2018 Rover• 2018 Rover Mission will have                                                limited time to core and store                                               up to 30 rock samples  – Will need to drive up to                                                        20 kilometers – Will need to consider targets                                                            from distinct areas• Strong interest from Mars 2018 Mission Program  Office (Charles Whetsel, Chris Salvo) in using  AEGIS to collect data on potential targets – Get data to science team faster – More targets could be considered  26
  • 27. Future Use – Other Missions• AEGIS system can enable a wide spectrum of  missions:  – Collect valuable science more often – Enhance onboard autonomy capabilities• Strong application to in‐situ missions to Titan,  Europa, Venus, Mars, the Moon, and small  bodies• Science autonomy listed as critical capability  in Titan Prebiotic Explorer Mission Study – Helps address challenges such as extremely limited  communication, high platform mobility, etc.“Onboard science algorithms will analyze the imagedata to detect trigger conditions such as scienceevents, interesting features, changes relative toprevious observations, …” TiPEx mission study team 27
  • 28. Industry, Government, Research Use Autonomous  Underwater  Vehicles (AUVs) Unmanned  Lunar  Aerial  Exploration VehiclesUnmanned  Sea Surface  AEGIS AEGIS is transferable to Multi‐core  Processor  Vehicles  a wide range of Benchmarking (USSVs) application domains Search and  Commercial  Rescue  Spectroscopy Robotics 28
  • 29. Industry, Government, Research Use Autonomous  Underwater  Vehicles (AUVs) Unmanned Aerial Vehicles (UAVs) Unmanned • Developing Lunar  automated Aerial  Exploration cueing capability for UAV Vehicles surveillance platforms.• Lower-resolution wide area imagery used to trigger on selected areas. Sea Surface  AEGIS higher-resolution follow-up Unmanned   Multi‐core  Processor • Proof-of-concept completed for ships using satellite imagery Vehicles  Benchmarking (USSVs)• Evaluating for use identifying ground vehicles on imagery from AFRL "Angel Fire" aerial asset Search and  Commercial  Rescue  Spectroscopy Robotics 29
  • 30. Industry, Government, Research Use Autonomous  Underwater  Vehicles (AUVs) Unmanned  Lunar  Aerial  Exploration “Moon Express is developing a lunar lander and Vehicles mobility system for exploration of platinum group metals on the surface of the moon as well as compete for the Google Lunar X-Prize.Unmanned   AEGIS AEGIS could be a great asset to this quest by autonomously recognizing rocks from iron-richSea Surface  Multi‐core  Processor  Vehicles  asteroids that might contain Benchmarking (USSVs) platinum.” -- Moon Express, Inc. Search and  Commercial  Rescue  Spectroscopy Robotics 30
  • 31. Industry, Government, Research Use Autonomous  Underwater  Robotic Underwater Vehicles Vehicles (AUVs) • WHOI Nereus vehicle Lunar  Unmanned  Aerial  Exploration – Performs deep ocean scientific Vehicles survey and sampling – Used to locate hydrothermal systems, volcanic processes, etc.Unmanned   AEGIS • CMU/Pittsburgh Aquarium Reefbot – Automatically detect, classify, and count fishSea Surface  Multi‐core  Processor  Vehicles in their natural habitat Benchmarking (USSVs) • AEGIS could save days/weeks of exploration time through autonomous data collection Search and  Commercial  Rescue  Spectroscopy Robotics 31
  • 32. Industry, Government, Research Use Autonomous  Underwater  Robotic Underwater Vehicles Vehicles (AUVs) • WHOI Nereus vehicle important advances in automatic Lunar  “AEGIS makes Unmanned  Aerial  – Performs deep ocean.. It has direct relevance to work Explorationdata analysis. scientific Vehicles at CMU in underwater vehicles for detecting survey and sampling – Usedand cataloging fish in deep water reefs.” to locate hydrothermal systems, volcanic processes, etc. D. WettergreenUnmanned   AEGIS • CMU/Pittsburgh Aquarium Reefbot – Automatically detect, CMU Robotics Institute classify, and count fishSea Surface  Multi‐core  Processor  Vehicles in their natural habitat Benchmarking (USSVs) • AEGIS could save days/weeks of exploration time through autonomous data collection Search and  Commercial  Rescue  Spectroscopy Robotics 32
  • 33. IMPACT“AEGIS usage on the MER Opportunity Rover showcases how it could be extremely beneficial for the Mars 2018 Mission.” Charles Whetsel,  Manager, Advanced Concepts 2018 Mars Program Office
  • 34. MER Impact: Increased Science• Before AEGIS, all targeted data required: – Manual evaluation of images – One to several communication cycles  – The rover to remain stationary and sometimes backtrack By the time the “Block Island” meteorite was noticed in an image, the Opportunity rover was already 200 meters past. The rover had to turn around and backtrack (costing 25 additional sols).• After AEGIS, targeted data can be collected: – Without ground analysis of context images – Without communication cycles – Any time during a rover drive  – Any time of day 34
  • 35. MER Impact: Increased ScienceVideo clip from Professor Steve SquyresPrincipal InvestigatorMars Exploration Rover Mission 35
  • 36. MER Impact: Increased Science 36
  • 37. MSL 2011 Rover Impact MSL Impact: Increased Science• 2008 study on ChemCam                                             9m target selection using 65                                          MER Panoramic camera                                                images • Top 5 targets evaluated – If random sampling, 10% chance of being rock – If chosen by AEGIS, 92% chance of being rock 37
  • 38. 2018 Rover ImpactMars 2018 Rover Impact: Increased Science• Previous study on AEGIS application to 2018 Mission• AEGIS can be used to collect additional targeted  science data – Increased close‐contact measurements by 50% – Increased remote‐sensing measurements by 500% – Provides scientists more targets to choose from            for coring2018 Rover Impact: Cost Benefit• Study showed number of sols required to investigate  and core a target could be decreased from 7 to 4 sols 38
  • 39. CREATIVITYAEGIS is a pioneering flight software system that provides scientists with sophisticated control over targeted data collection
  • 40. Creativity: Innovation• AEGIS provides new paradigm for                                                Original Optimized surface data acquisition 62 Memory Usage (megabytes) – Scientist provides description of target – System can collect data whenever target                                                      detected • Flight challenges 3.75 – Image processing performed on RAD6000  (orders of magnitude slower than standard  Memory reduced 16x desktop machine) 62 MB to 3.75 MB – AEGIS limited to < 4 MB of memory – Large performance optimizations made! Runtime (seconds)• Inventive approach to flight  software change – Full flight software upload not possible Benchmark Images – AEGIS uploaded as standalone module – Loaded into memory whenever want to use  Performance improved 7X (< 30 secs) 40
  • 41. Creativity: Usability• Parameters defined through collaboration with scientists – Describe attributes of candidate targets – Express diverse and evolving science goals • System did not require extensions to                                            Albedo, shape, and size MER command dictionary or telemetry• Training materials  – Web interface for creating commands – User’s guide for software usage and                                                     sequencing – Standard terrain profiles available – Result message (EVR) interpreter“One of the key aspects that has made the AEGIS team successful is their long track record of working with the scientists.” Dr. Jack Stocky New Millennium Program Manager 41
  • 42. Creativity: Quality Factors• Reliable target detection – Find rock targets in diverse terrain – Is resilient to dust‐covered or shadowed rocks – Works under strict computation constraints• Risk control through resource limits and time  deadlines• Validation and Verification – Extensive MER testing procedure and code reviews – Nightly build, static analysis, unit and regression tests 42
  • 43. SUMMARY“This autonomous science breakthrough is really changing expectations for future science mission operations.” Raymond E. Arvidson Deputy Principal Investigator Mars Exploration Rover
  • 44. Summary• Significance:  Far Reaching – Aerospace: Routinely used on MER – Science/Technology: • Enables science that could not be previously collected • Applications in military, commercial and research fields – Humanitarian: • Contributing to goal of finding life on other planets • Inspiring next generation in STEM areas• Development:  Flight Proven• Assessment of use:   – Planned for Infusion into New Missions and Applications• Creativity:  Pioneering/Deeply Innovative – Innovation: New paradigm for in‐situ data acquisition – Usability: System parameters designed through direct  collaboration with scientists – Quality: Reliable target detection under strict computation  44 constraints
  • 45. For more information and surface results, visit  the AEGIS website: 45
  • 46. For more information…Questions? We’d like to acknowledge our sponsors: New Millennium Program, Mars Technology Program, JPL Research and Technology Development Program, and the IND Technology Program and thank you to the: The Mars Exploration Rover Mission
  • 47. EXTRA
  • 48. AEGIS Target Detection Contours Flood fill + Morphology ops Edge detection 
  • 49. Contours can be further filtered Operator  Filtering  Rules Rover Body Masking 49
  • 50. Target Feature ExtractionReflectance – Mean – Variance – Skew – KurtosisSize Light Dark – Inscribed circle – Pixel areaShape – Eccentricity – Ellipse fit error – Roundness – Ruggedness – Angularity Rounded Angular
  • 51. Target Prioritization / Top Target Selection Images from MER field trial• Scientists can  Near the prioritize different  top of the feature values and  list of combinations of two “round” – e.g., prefer large, high rocks albedo rocks – Can also support MER  cobble campaign, outcrop  finder, soil finder, etc.• Priority specification is  Near the part of command  bottom of the list of sequencing “round” rocks• Can be easily changed  as rover enters  different terrain areas
  • 52. AEGIS Code Details• AEGIS is 7968 SLOC (C)• Limited to less than 4 MB of memory• Requires only 232 KB of disk space• Regular static analysis using Coverity PreventTM• Formal code reviews – Internal AEGIS Team – Other JPL AI/machine-learning developers not members of AEGIS Team – MER Team 52
  • 53. Meteorite Detector Devin Island
  • 54. Meteorite Detector Marquette Island
  • 55. AEGIS Target Detections• Target detections are consistent with AEGIS selection  profiles  – 90% of top targets meet the selection profile  – Confirmed by evaluation of context Navcam imagery – All results reviewed with MER Science Team• The MER Science Team is very happy with AEGIS and  continues to request it regularly 55
  • 56. OASIS Framework • OASIS: Onboard Autonomous Science Investigation System• Objective: Maximize science returned on surface mission – Identify and respond to science opportunities – Data prioritization for downlink – Maximize utilization of onboard resources• Approach – Data segmentation and feature extraction for multiple instruments – Science Data Analysis • Prioritize targets and/or data • Summarize data – Automated Planning and Scheduling • Adjust rover activities to collect new data • Ensure operation within rover resource and operation constraints
  • 57. AEGIS in the OASIS Framework • AEGIS is a flight software system derived from the larger OASIS framework• Developed by same team of people• AEGIS includes a subset of OASIS capabilities selected for MER Relevant - Instruments available on MER Desired - Requested by scientists Feasible - Fit within memory and time limits
  • 58. AEGIS Results from B Sol 2138
  • 59. AEGIS Result from B Sol 2221