CLIM Program: Remote Sensing Workshop, Distributed Access and Analysis: NASA - Mike Little, Feb 12, 2018
•
0 likes•106 views
Data systems in NASA?s Earth Science Division are primarily focused on providing stewardship of the products of remote sensing and are manifested as Digital Active Archive Systems. Each Instrument Team has a related Science Team which defines the algorithms and monitors the processing of the output of the instruments to produce the related data products and in an format and standards compliance of them. These teams are influenced also by the research and applied sciences components of the programs, but the primary focus is on proving the ongoing validity of the products. Across the distributed system, every product is different. However, this is not conducive to analytics. NASA?s Advanced Information Systems Technology (AIST) program is developing an entirely new approach to creating Analytic Centers which focus on the scientific investigation and harmonize the data, computing resources and tools to enable and to accelerate scientific discovery. Stay tuned to _nd out how. A major element, in today’s science interests, is the comparison of multi-dimensional datasets; this warrants considerable experimentation in trying to understand how to do so meaningfully and quantitatively; asked another way, \What do you mean by similar?" Uncertainty quantification has evolved considerably in the arenas of data reduction and full physics models; however, the emerging demand for machine learning and other artificial intelligence techniques has failed to keep uncertainty quantification and error propagation in mind and there is considerable work to be done.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to CLIM Program: Remote Sensing Workshop, Distributed Access and Analysis: NASA - Mike Little, Feb 12, 2018
Similar to CLIM Program: Remote Sensing Workshop, Distributed Access and Analysis: NASA - Mike Little, Feb 12, 2018 (20)
More from The Statistical and Applied Mathematical Sciences Institute
More from The Statistical and Applied Mathematical Sciences Institute (20)
Recently uploaded
Recently uploaded (20)
CLIM Program: Remote Sensing Workshop, Distributed Access and Analysis: NASA - Mike Little, Feb 12, 2018
- 1. NASA Data Access and Analysis SAMSI Theory of Data Systems Michael.M.Little@nasa.gov AIST Program Manager February 10, 2018 2/11/18 NASA Overview of Data Systems 1
- 2. Overview of Overview • NASA Data Repositories • NASA Data Analytic Support • Considerations in Solving the ToDS Problem 2/11/18 NASA Overview of Data Systems 2
- 3. NASA Sources of Remote Sensing Data • Primary Respository: EOS-DIS Digital Active Archive Centers (DAAC) • Thematic Orientation • Common Metadata Repository (CMR) • Other Data not at DAACs • Aircraft data • In situ data • Biodiversity data 2/11/18 NASA Overview of Data Systems 3
- 4. Repositories: NASA Data Collection Systems 2/11/18 NASA Overview of Data Systems 4
- 5. Repositories: Data Sources at NASA 2/11/18 NASA Overview of Data Systems 5
- 6. Repositories: Current and Estimated Archive Growth Today Cloud Evolution (ExCEL) Project 62/11/18 NASA Overview of Data Systems
- 7. Repositories: EOS-DIS Cloud Technical Foundational Capabilities Analytics: Cloud based on-the-fly data analysis and API access to analytic optimized cloud storage Data Distribution: Distribution includes raw data, sub-setting, visualization, and more Data Management Data Archive * Analytics Data Distribution Data Ingest NGAP AWS Networking IT Security * Important, although not considered foundational in ExCEL project Data Ingest: Multi-mission, multi- disciplines data ingest into the cloud Data Management: Development/execution of information lifecycle needs of NASA mission based earth science data sets Data Archive: Preserve and protect NASA mission based earth science data assets Platform: Cloud infrastructure capabilities on Amazon Web Services (AWS) and the EOSDIS NASA Compliant General Application Platform, including networking and IT Security 72/11/18 NASA Overview of Data Systems
- 8. Repositories: Other NASA Sources • Aircraft-based Observations • Science Data: Work in progress to integrate with DAACs • Engineering Test Flights: Held by PI or Instrument Engineers • In situ Observations • Science Data Held by PI 2/11/18 NASA Overview of Data Systems 8
- 9. NASA Tools for Data Analysis • EOS-DIS Cloud Analytics • AIST Analytic Center Development 2/11/18 NASA Overview of Data Systems 9
- 10. Background: Key Thrusts for NASA AIST • Analytic Center as a framework for tool integration • Focus on Science Investigators and the Investigation • What tools are needed and how to plug into AIST18 • Sensor Web Observing Strategy • Constellations/swarms of sensors (In situ, airborne, orbit) • Complete architecture • Command and control structures and tools • Study areas • Alternative OSSE technologies • Reduce cost and turnaround time • Computer aided discovery • What do you mean by similar? • How to compare multi-dimensional datasets. 2/11/18 NASA Overview of Data Systems 10
- 11. Missions DATA LIFE CYCLE Models&Model Output SensorControl SensorMgt HW Onboard Plan MissionMgt Mission Mgt Space Ground HW Movement DataModels Storage Containers HW Storage Mgt Characterizing Product Generation Clean/Groom ENHMetadata CAL/VAL ModelInput ScienceAnalysis ApplicUsage OpUse Disposal/ Repository Generation Data Mgt 1 2 3 4 5 6 7 8 9 Env. Demo Space Demo Actual Gnd Test Flight Proof 0 NASA NESSF, DEVELOP, Fellowships, NSF, DoE, NASA STMD SBIR OI, DHS, DARPA TRL Pre OA Missions COM DAACs, HEC DAACs, HEC Mission & GMAO, GISS, LIS Commercial DAACs Flight Mission LIS, GMAO, RCME R&A Applied Sci NARA NASA EARTH SCIENCE DIVISION INFORMATION TECHNOLOGY PROGRAMS PURPOSE- DECONFLICT ITR&D PROGRAMS FOR AIST 2016 AIST AIST FltMissions& Measures CMAC Measures& Access Commercial AIST Demos & Infusions Principles Concept Crit Proof of Concept Lab Val Env. Breadbd FEMA, NOAA, NWS, USGS AIST Demo ESDIS, HEC SCAN Missions Commercial MAP Data ExploitationData ManagementData Generation PrePhaseA TRL Data Life Cycle Background AIST Program Technical Overview 2/11/18 11 AIST Responsibility
- 12. Solutions: What is an Analytic Center • An environment for conducting a Science investigation • Enables the confluence of resources for that investigation • Tailored to the individual study area • Harmonizes data, tools and computational resources to permit the research community to focus on the investigation • Reduce the data preparation time to something tolerable • Catalog of optional resources (think HomeDepot shopping) • Semantic-enabled catalog of resources (think Yelp) with help • Relevant publications • Provide established training data sets of varying resolution • Provide effective project confidentiality, integrity and availability • Single sign-on and unified financial tracking • Publication support for new templates from AGU, Elsevier, ACM, IEEE, etc. • Concept of Operations • Retrieve and Condition data into a local storage system • Data handling tools are pre-configured to interface with local storage system 2/11/18 NASA Overview of Data Systems 12
- 13. Project Work Environment Tools • Discovery & Catalog • Work Management • Data Interfaces • Analytic Tools • Modeling • Collaboration • Visualization • Sharing/ Publication • Local/custom Computational Infrastructure • Computing • Capacity • Capability • Storage • Communications Data • Catalog • NASA DAAC • Other US Govt • Non-US • Local or non-public User • Project Definition • Plan for Investigation Computing • Local systems • High End Computing • Cloud Computing Capability • Quantum Computing • Neuromorphic Computing Storage • Data Containers • Thematic model • Metadata/Ontology • Resulting Products • Published data • Provenance Solutions: Analytic Center as a Framework 2/11/18 NASA Overview of Data Systems 13
- 14. Solutions: OceanWorks as an Analytic Center 2/11/18 NASA Overview of Data Systems 14 Project Work Environment -OceanWorks- Tools • EDGE • Services • Area Averaged Time series • Time averaged map • Correlation Map • Daily Differences • Matchup (single satellite-multiple in situ) • Workflow: • Jupyter Notebook • Zeppelin Notebook • Visualization • OnEarth • CMC (GIS) • Collaboration • Confluence, JIRA, GIT, Bamboo • Apache wiki • Smartsheet and Google Office • Slack • Expert help - MUDROD Computational Infrastructure Data • EDGE • Earthdata CMR • nonCMR DAAC • PI data ingest • GRACE (CSR) • ECCO (MIT) • Altimetry (OSU) • DOMS • ICOADS • SAMOS • SPURS • ASCAT Winds • Aquarius • MODIS SST • MUR-SST Physical Oceanographers • Project Definition • Plan for Investigation Computing • Local systems • AMCE Cloud Computing • JPL on premises Cloud Storage • Nexus POC: Thomas.huang@nasa.gov
- 15. Solutions: Tropical Cycle Information System (TCIS) 2/11/18 NASA Overview of Data Systems 15 Project Work Environment -JPL- Tools • Integrated through web portal • Re-use requires TCIS Team • NEOS3 instrument simulator • Statistical evaluation • Storm Structure • Environment vertical structure • Visualization • Models Computational Infrastructure Tropical Cyclone Research • Project Definition • Plan for Investigation Computing • Local systems • AWS through AMCE Data • Integrated web portal • Hurricane List & Tracks • HS3 Field Campaign • SMAP • SST • TPW • WIND • ECMWF output • HAMSR • GOES-East • GOES-West • Microwave Rain Signature Storage • Local Storage POC: svetla.m.hristova-veleva@nasa.gov https://tropicalcyclone.jpl.nasa.gov/
- 16. Solutions: Earth Data Analytics System (EDAS) as an Analytic Center 2/11/18 NASA Overview of Data Systems 16 Project Work Environment -NCCS- Tools • Earth System Grid Federation Framework • Discovery & Catalog • Work Management – Jupyter Notebook • Data Interfaces - OpenDAP • Analytic Tools: Min, max, sum, diff, average, rms,anomaly, std deviation • Visualization using time series or spaceplot plot routines Computational Infrastructure Data • MERRA • ECMWF ERA • NOAA NCEP CFSR • JMA JRA • UA ERA Climate Impact • Project Definition • Plan for Investigation Computing • Local systems • High End Computing (DISCOVER) • GSFC Science Cloud Storage • Local Sparc • SGI Data Migration Facility POC: laura.carriere@nasa.gov https://www.nccs.nasa.gov/services/Analytics
- 17. Solutions: NASA Earth Exchange (NEX) as an Analytic Center 2/11/18 NASA Overview of Data Systems 17 Project Work Environment -NEX- Tools • Models • Tops • Biome-BGC • LPJ Dynamic Global Model • Sandbox for small scale experiments • Analysis • R and python based tools • Matlab VIIRS HDF5 swath conversion • Workflow: Jupyter Notebook Computational Infrastructure Data • Built into website • Dataset Sources • Landsat • Sentinel 1A • Modis • ASTER • TRMM • AVHRR • Climate Datasets • Landcover • Digital Elevation Map • STATSGO Soils • USDA Aerial NAIP • And others Land Change/Use Community • Project Definition • Plan for Investigation Computing • NASA Advanced Supercomputing (NAS) • Amazon Web Services (AWS) Public Cloud • Ames Quantum Computer (D-wave) Storage • Data Containers • Thematic model • Metadata/Ontology • Resulting Products • Published data • Provenance POC: rama.nemani@nasa.gov https://nex.nasa.gov/nex
- 18. Solutions: Recent AIST Development Projects • Data Wrangling • Data Container Studies reported at AGU2015, AGU2016 • Ontology-based metadata (AIST14-0095) • Science Enterprise Service Bus (AIST14-0014) • In Situ-Satellite matchup service (AIST14-0086) • SciSpark (AIST14-0034) • Quantum Computing tools (AIST14-0096, AIST16-0091) • Multi-instrument data integration (AIST11-0077, AIST16-0048) • Workflow tools (AIST16-0092) • Analytic Tools • Regression analysis on Global Vegetation (AIST14-0115) • Pattern-based content analysis in GIS (AIST14-0027) • Uncertainty Quantification (AIST11-0011, AIST16-0030) • Computer-aided Discovery (AIST14-0036, AIST16-0048) • Discipline Specific Tools • EcoSis Tools (AIST16-0118) • Hyperspectral image analysis tools (AIST16-0138) • Modeling • Community re-organization biodiversity models (AIST16-0052) 2/11/18 NASA Overview of Data Systems 18
- 19. Solutions: Future AIST Tool Needs 2/11/18 NASA Overview of Data Systems 19 • Integration • Re-usable framework and relevant human factors engineering • Geographic information Systems (GIS) harmonization with data • Analytic Tool Needs • Analytic Tool Development and harmonization • Leverage other US Government investments in tools • Uncertainty assessment and quantification • Online training and expert help systems • Linkage to publishing industry • Leverage visualization advances (VR, AR, etc)
- 20. So… What is a sensor web? A candidate definition: A sensor web is a distributed system of sensing nodes that are interconnected by a communications fabric and that functions as a single, highly coordinated, virtual instrument. It autonomously detects and dynamically reacts to events, measurements, and other information from constituent sensing nodes and from external nodes (e.g., predictive models) by modifying its observing state so as to optimize science information return. - Steve Talabac, 2003 2/11/18 NASA Overview of Data Systems 20
- 21. FY16-18 Studies • Observing System Simulation Experiments (OSSE) • Current approach is slow and expensive • Need a faster, cheaper • Most theory focuses on just weather • What do you mean by similar? • Comparison of multi-dimensional data sets • Simplistic interpretations create misconceptions 2/11/18 NASA Overview of Data Systems 21
- 22. Considerations in Theory of Data Systems • Many scientific investigations report that 80% of their time is spent on cleaning up the data • NASA Mission Design focuses on Stewardship and Cal/Val • Format is unique to the instrument science team and the data product • Most Science Investigations need data from MULTIPLE SOURCES • Each Data Supplier focuses on their own data • Different Geolocation reference point in each data product • Different Flags for no-observation • Different grid references • Different statements/estimates of uncertainty and error • Large volumes of data from disparate sources require modern analytic techniques, tools and computing capacity • Regression techniques lack techniques to quantify uncertainty 2/11/18 NASA Overview of Data Systems 22