Estlin aegissoyajpl 2012

Autonomous Exploration for Gathering Increased Science AEGIS Tara EstlinBenjamin Bornstein, Daniel Gaines, David R. Thompson, Rebecca Castano, Robert C. Anderson, Michael Burl, Charles de Granville and Michele Judd 2011 NASA Software of the Year

Consider the following problem…• You are a robotic explorer on another planet • You only talk to Earth once a day• You are in a hostile environment• You have limited power and computing abilities• You are constantly on the move exploring different terrains• As you move, you need to quickly determine if you see objects that are interesting to scientists• If you do, you want to acquire data on these objects, before the rover moves past 2

This is why we developed AEGIS AEGIS:• Is a new paradigm for in‐situ science using onboard autonomy• Provides intelligent targeting and data acquisition by – analyzing images of the rover scene – identifying high‐priority science targets (e.g., rocks), and – taking high quality data of these targets completely autonomously with no ground interaction required 3

AEGIS Video 4

How is AEGIS being used? AEGIS:• Is in regular operational use onboard the Mars Exploration Rover (MER) Mission Opportunity rover for the past two years• Excels in automated targeting with narrow field‐of‐ view (FOV) remote sensing instruments, such as: – MER Panoramic Cameras (in current use) – MER Mini‐Thermal Emission Spectrometer – Mars Science Laboratory (MSL) Rover ChemCam Spectrometer Mini-TES Mosaic – Before AEGIS, had to manually select targets, based on ground analysis 5

AEGIS Process for MER Process fully automated! Advanced image Navcam acquisition processing technique enables reliable, rapid identification of candidate targets. Target detectionAlgorithms quantify key Target feature Scientists canintuitive target extraction prioritizeproperties such as importantbrightness, size, and propertiesshape. for each run Target prioritization Target pointing determination Robust approach to Top score for large size pointing selection maximizes data of target. Pancam pointing High-quality, 13 color filter, quarter-frame Pancam acquisition Panoramic camera image

Benefit of AEGIS for Rover Drive Autonomously‐targeted remote sensing taken end of drive by AEGIS X X Autonomously‐targeted remote sensing taken mid‐drive by AEGIS Manually‐targeted remote sensing as specified by science team (taken before drive) Targeted data Targeted data selected manually with AEGIS 7

AEGIS identifies rock targetsScientist target profile: large rocks of high reflectance.Markers show top ten prioritized targets. 8

AEGIS identifies rock targetsArea covered by MER Panoramic Camera. 9

AEGIS identifies rock targetsArea covered by MER Panoramic Camera. 10

AEGIS delivers 13F Pancam image after autonomously targeting the top priority rock 11

MER before AEGIS Sol 1 Sol 2 Sol 3 Perform MANUALLY TARGETED remote Pancam Sol 2sensing of current Perform Perform rover area untargeted untargeted Drive rover 100m remote remote to new stopping sensing of sensing of point local area local area Acquire wide‐angle images of new terrain area Pancam Sol 3 Multi-Sol Plan 12

MER after AEGIS Sol 1 Sol 2 Sol 3 Perform MANUALLY TARGETED remote AEGIS Pancam 1sensing of current Perform Perform rover area autonomously autonomously targeted targetedDrive rover 100m remote remote to new stopping sensing of sensing of point local area local area Acquire wide‐angle images of new terrain area AEGIS Pancam 2 Multi-Sol Plan 13

SIGNIFICANCE Scientists trust AEGIS to make intelligent decisions about collecting new science.

Significance: Aerospace• Fully operational and used Opportunity Today regularly on MER mission – Saves valuable time every targeted data collection – Has been used more than any other new technology on the Opportunity rover• Currently infusing into Mars Science Laboratory (MSL) Mission• Attracting strong interest for – 2018 Mars rover and other future in‐situ missions (e.g., Titan, Venus, Europa) – Military applications (e.g., UAVs) 15

Significance: Science/Technology• Enables the collection of science data that would otherwise not be possible – During or right after drives – Different times of day and temps• Saves significant time and cost for targeted data collection – Gets data into the hands of scientists twice as fast (or more) than standard operations• Enables scientists to easily use and interact with autonomy software – Parameters chosen after significant consultation with scientists• 25+ science and technology publications• Application to large number of problems in industry and academia (e.g., underwater robotics) 16

Significance: Humanitarian• Directly contributing to humanity goal of finding life on other planets – Mars program theme of “Follow the water…”• Significant outreach vehicle; over 35 media articles since release in 2010 – “Mars Rover Getting Smarter As It Gets Older” – “NASA upgrades Mars rover brain” 17

Significance: Humanitarian Inspiring the next generation in STEM:Science, Technology, Engineering and Mathematics Over 46 Amateur Astronomy Clubs, Australia Schools and Teacher Organizations around United Kingdom the world featured our “rock hound” software in their newsletters 18

DEVELOPMENT AEGIS is flight proven and fully operational

Development Status • TRL Level 9: Flight Proven – Software fully operational – In regular use on MER Opportunity rover • MER and MSL Scientists have already asked for extensions, which are in progress: – Enabling multiple targeted observations – Triggering on single filter color images – Identifying novel targets – Identifying representative targets – “Soil only” detector“AEGIS is a true success story for the Mars Technology Program” Dr. Samad Hayati Manager of Mars Technology Program 20

ASSESSMENT of USEAEGIS technology is being applied in a wide range of applications.

NASA Use – MER AEGIS is considered every multi‐sol plan. Sol 2138 Sol 2172 Sol 2313 Sol 2221 Sol 2247 Sol 2278 Sol 2290 Sol 2304 Sol 2312“AEGIS is a significant enhancement for the mission and the scientific community. MER is the first mission to implement the capability that AEGIS provides – and it has really paid off.” Dr. John Callas 22 Mars Exploration Rover Mission Project Manager

NASA Use – MSL Rover• The MSL Rover ChemCam Team has requested AEGIS (PI: Roger Wiens)• AEGIS is ideal for ChemCam’s narrow field‐of‐view Laser‐Induced Breakdown Spectrometer (LIBS) – Samples rocks from a distance of 1 to 7 meters – Able to rapidly identify rock elemental composition• AEGIS enables multiple autonomously targeted ChemCam measurements throughout the day• MSL flight software integration in progress 23

NASA Use – MSLVideo clip from Dr. Roger WiensPrincipal InvestigatorMSL Rover Mission Chemcam Instrument 24

NASA Use – MSL 25

NASA Use – Mars 2018 Rover• 2018 Rover Mission will have limited time to core and store up to 30 rock samples – Will need to drive up to 20 kilometers – Will need to consider targets from distinct areas• Strong interest from Mars 2018 Mission Program Office (Charles Whetsel, Chris Salvo) in using AEGIS to collect data on potential targets – Get data to science team faster – More targets could be considered 26

Future Use – Other Missions• AEGIS system can enable a wide spectrum of missions: – Collect valuable science more often – Enhance onboard autonomy capabilities• Strong application to in‐situ missions to Titan, Europa, Venus, Mars, the Moon, and small bodies• Science autonomy listed as critical capability in Titan Prebiotic Explorer Mission Study – Helps address challenges such as extremely limited communication, high platform mobility, etc.“Onboard science algorithms will analyze the imagedata to detect trigger conditions such as scienceevents, interesting features, changes relative toprevious observations, …” TiPEx mission study team 27

Industry, Government, Research Use Autonomous Underwater Vehicles (AUVs) Unmanned Lunar Aerial Exploration VehiclesUnmanned Sea Surface AEGIS AEGIS is transferable to Multi‐core Processor Vehicles a wide range of Benchmarking (USSVs) application domains Search and Commercial Rescue Spectroscopy Robotics 28

Industry, Government, Research Use Autonomous Underwater Vehicles (AUVs) Unmanned Aerial Vehicles (UAVs) Unmanned • Developing Lunar automated Aerial Exploration cueing capability for UAV Vehicles surveillance platforms.• Lower-resolution wide area imagery used to trigger on selected areas. Sea Surface AEGIS higher-resolution follow-up Unmanned Multi‐core Processor • Proof-of-concept completed for ships using satellite imagery Vehicles Benchmarking (USSVs)• Evaluating for use identifying ground vehicles on imagery from AFRL "Angel Fire" aerial asset Search and Commercial Rescue Spectroscopy Robotics 29

Industry, Government, Research Use Autonomous Underwater Vehicles (AUVs) Unmanned Lunar Aerial Exploration “Moon Express is developing a lunar lander and Vehicles mobility system for exploration of platinum group metals on the surface of the moon as well as compete for the Google Lunar X-Prize.Unmanned AEGIS AEGIS could be a great asset to this quest by autonomously recognizing rocks from iron-richSea Surface Multi‐core Processor Vehicles asteroids that might contain Benchmarking (USSVs) platinum.” -- Moon Express, Inc. Search and Commercial Rescue Spectroscopy Robotics 30

Industry, Government, Research Use Autonomous Underwater Robotic Underwater Vehicles Vehicles (AUVs) • WHOI Nereus vehicle Lunar Unmanned Aerial Exploration – Performs deep ocean scientific Vehicles survey and sampling – Used to locate hydrothermal systems, volcanic processes, etc.Unmanned AEGIS • CMU/Pittsburgh Aquarium Reefbot – Automatically detect, classify, and count fishSea Surface Multi‐core Processor Vehicles in their natural habitat Benchmarking (USSVs) • AEGIS could save days/weeks of exploration time through autonomous data collection Search and Commercial Rescue Spectroscopy Robotics 31

Industry, Government, Research Use Autonomous Underwater Robotic Underwater Vehicles Vehicles (AUVs) • WHOI Nereus vehicle important advances in automatic Lunar “AEGIS makes Unmanned Aerial – Performs deep ocean.. It has direct relevance to work Explorationdata analysis. scientific Vehicles at CMU in underwater vehicles for detecting survey and sampling – Usedand cataloging fish in deep water reefs.” to locate hydrothermal systems, volcanic processes, etc. D. WettergreenUnmanned AEGIS • CMU/Pittsburgh Aquarium Reefbot – Automatically detect, CMU Robotics Institute classify, and count fishSea Surface Multi‐core Processor Vehicles in their natural habitat Benchmarking (USSVs) • AEGIS could save days/weeks of exploration time through autonomous data collection Search and Commercial Rescue Spectroscopy Robotics 32

IMPACT“AEGIS usage on the MER Opportunity Rover showcases how it could be extremely beneficial for the Mars 2018 Mission.” Charles Whetsel, Manager, Advanced Concepts 2018 Mars Program Office

MER Impact: Increased Science• Before AEGIS, all targeted data required: – Manual evaluation of images – One to several communication cycles – The rover to remain stationary and sometimes backtrack By the time the “Block Island” meteorite was noticed in an image, the Opportunity rover was already 200 meters past. The rover had to turn around and backtrack (costing 25 additional sols).• After AEGIS, targeted data can be collected: – Without ground analysis of context images – Without communication cycles – Any time during a rover drive – Any time of day 34

MER Impact: Increased ScienceVideo clip from Professor Steve SquyresPrincipal InvestigatorMars Exploration Rover Mission 35

MER Impact: Increased Science 36

MSL 2011 Rover Impact MSL Impact: Increased Science• 2008 study on ChemCam 9m target selection using 65 MER Panoramic camera images • Top 5 targets evaluated – If random sampling, 10% chance of being rock – If chosen by AEGIS, 92% chance of being rock 37

2018 Rover ImpactMars 2018 Rover Impact: Increased Science• Previous study on AEGIS application to 2018 Mission• AEGIS can be used to collect additional targeted science data – Increased close‐contact measurements by 50% – Increased remote‐sensing measurements by 500% – Provides scientists more targets to choose from for coring2018 Rover Impact: Cost Benefit• Study showed number of sols required to investigate and core a target could be decreased from 7 to 4 sols 38

CREATIVITYAEGIS is a pioneering flight software system that provides scientists with sophisticated control over targeted data collection

Creativity: Innovation• AEGIS provides new paradigm for Original Optimized surface data acquisition 62 Memory Usage (megabytes) – Scientist provides description of target – System can collect data whenever target detected • Flight challenges 3.75 – Image processing performed on RAD6000 (orders of magnitude slower than standard Memory reduced 16x desktop machine) 62 MB to 3.75 MB – AEGIS limited to < 4 MB of memory – Large performance optimizations made! Runtime (seconds)• Inventive approach to flight software change – Full flight software upload not possible Benchmark Images – AEGIS uploaded as standalone module – Loaded into memory whenever want to use Performance improved 7X (< 30 secs) 40

Creativity: Usability• Parameters defined through collaboration with scientists – Describe attributes of candidate targets – Express diverse and evolving science goals • System did not require extensions to Albedo, shape, and size MER command dictionary or telemetry• Training materials – Web interface for creating commands – User’s guide for software usage and sequencing – Standard terrain profiles available – Result message (EVR) interpreter“One of the key aspects that has made the AEGIS team successful is their long track record of working with the scientists.” Dr. Jack Stocky New Millennium Program Manager 41

Creativity: Quality Factors• Reliable target detection – Find rock targets in diverse terrain – Is resilient to dust‐covered or shadowed rocks – Works under strict computation constraints• Risk control through resource limits and time deadlines• Validation and Verification – Extensive MER testing procedure and code reviews – Nightly build, static analysis, unit and regression tests 42

SUMMARY“This autonomous science breakthrough is really changing expectations for future science mission operations.” Raymond E. Arvidson Deputy Principal Investigator Mars Exploration Rover

Summary• Significance: Far Reaching – Aerospace: Routinely used on MER – Science/Technology: • Enables science that could not be previously collected • Applications in military, commercial and research fields – Humanitarian: • Contributing to goal of finding life on other planets • Inspiring next generation in STEM areas• Development: Flight Proven• Assessment of use: – Planned for Infusion into New Missions and Applications• Creativity: Pioneering/Deeply Innovative – Innovation: New paradigm for in‐situ data acquisition – Usability: System parameters designed through direct collaboration with scientists – Quality: Reliable target detection under strict computation 44 constraints

For more information and surface results, visit the AEGIS website: http://aegis.jpl.nasa.gov/ 45

For more information…Questions?Tara.Estlin@jpl.nasa.gov We’d like to acknowledge our sponsors: New Millennium Program, Mars Technology Program, JPL Research and Technology Development Program, and the IND Technology Program and thank you to the: The Mars Exploration Rover Mission

AEGIS Target Detection Contours Flood fill + Morphology ops Edge detection

Contours can be further filtered Operator Filtering Rules Rover Body Masking 49

Target Feature ExtractionReflectance – Mean – Variance – Skew – KurtosisSize Light Dark – Inscribed circle – Pixel areaShape – Eccentricity – Ellipse fit error – Roundness – Ruggedness – Angularity Rounded Angular

Target Prioritization / Top Target Selection Images from MER field trial• Scientists can Near the prioritize different top of the feature values and list of combinations of two “round” – e.g., prefer large, high rocks albedo rocks – Can also support MER cobble campaign, outcrop finder, soil finder, etc.• Priority specification is Near the part of command bottom of the list of sequencing “round” rocks• Can be easily changed as rover enters different terrain areas

AEGIS Code Details• AEGIS is 7968 SLOC (C)• Limited to less than 4 MB of memory• Requires only 232 KB of disk space• Regular static analysis using Coverity PreventTM• Formal code reviews – Internal AEGIS Team – Other JPL AI/machine-learning developers not members of AEGIS Team – MER Team 52

Meteorite Detector Devin Island

Meteorite Detector Marquette Island

AEGIS Target Detections• Target detections are consistent with AEGIS selection profiles – 90% of top targets meet the selection profile – Confirmed by evaluation of context Navcam imagery – All results reviewed with MER Science Team• The MER Science Team is very happy with AEGIS and continues to request it regularly 55

OASIS Framework • OASIS: Onboard Autonomous Science Investigation System• Objective: Maximize science returned on surface mission – Identify and respond to science opportunities – Data prioritization for downlink – Maximize utilization of onboard resources• Approach – Data segmentation and feature extraction for multiple instruments – Science Data Analysis • Prioritize targets and/or data • Summarize data – Automated Planning and Scheduling • Adjust rover activities to collect new data • Ensure operation within rover resource and operation constraints

AEGIS in the OASIS Framework • AEGIS is a flight software system derived from the larger OASIS framework• Developed by same team of people• AEGIS includes a subset of OASIS capabilities selected for MER Relevant - Instruments available on MER Desired - Requested by scientists Feasible - Fit within memory and time limits

AEGIS Results from B Sol 2138

AEGIS Result from B Sol 2221

Estlin aegissoyajpl 2012

by NASAPMC on Mar 19, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Estlin aegissoyajpl 2012 Presentation Transcript