Hnilo RDAP11 Data Archives in Federal Agencies


Published on

National Climate Model Portal, Jay Hnilo, NOAA NOMADS; Data Archives in Federal Agencies; RDAP11 Summit

The 2nd Research Data Access and Preservation (RDAP) Summit
An ASIS&T Summit
March 31-April 1, 2011 Denver, CO
In cooperation with the Coalition for Networked Information

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • “ What to Archive Process”
  • A Vision we helped create – working on NOMADS/NCMP to implement our part. Rutledge
  • NCMP is a Service within NOMADS. Our primary mission: distributed format neutral access to NOAA’s model data & LEVERAGE COMMUNITY RESOURCES and APPLICATIONS: iRODS, ESG, TDS, LAS, CDAT, … We are open to new collaborations…
  • How to move into the Future?
  • Hnilo RDAP11 Data Archives in Federal Agencies

    1. 1. The National Climate Model Portal Dr. Jay Hnilo NCMP Senior Scientist NOAA’s Cooperative Institute for Climate and Satellites (CICS-NC) National Climatic Data Center (NCDC) Asheville, NC 28801 Overview & NCDC Archive and Access Challenges ASIS&T Research Data Access and Preservation Summit Denver CO. March 31, 2011 M. Sutton 1995 The National Oceanic and Atmospheric Administration
    2. 2. <ul><li>Background: NOMADS </li></ul><ul><li>- A Data Access System </li></ul><ul><li>NCMP and NOMADS </li></ul><ul><li>- Goals and Motivation </li></ul><ul><li>NCDC Archive Processes </li></ul><ul><li>- Archive Processes </li></ul><ul><li>- Distributed Access Philosophy </li></ul>Outline
    3. 3. <ul><li>Until 2002 there existed no long-term archive for Climate and Weather models in NOAA. </li></ul><ul><li>Retrospective analysis and model inter-comparison are necessary to verify and improve short term NWP models, seasonal forecasts, climate simulations, assessments and detection. </li></ul><ul><li>University and Institutional research goes largely untapped by NOAA scientists. Effort is wasted on data receipt and format issues with no infrastructure to collaborate. </li></ul>Background NOMADS Data Access System
    4. 4. <ul><li>In 2002 to overcome a deficiency in model data access, some of the Nations top scientists actively engaged in a grass-roots framework to share data and research findings over the Internet. </li></ul><ul><li>NCDC, NCEP and GFDL initiated the NOAA Operational Model Archive and Distribution System. </li></ul><ul><li>NOMADS is a distributed data service providing format independent access to climate and weather models and associated data. </li></ul>Background NOMADS Data Access System
    5. 5. <ul><li>foster research within the geo-science </li></ul><ul><li>communities (ocean, weather, and climate) </li></ul><ul><li>to study multiple earth systems using </li></ul><ul><li>collections of distributed data, </li></ul><ul><li>promote model evaluation and product development </li></ul><ul><li>develop institutional partnerships via distributed open technologies. </li></ul><ul><li>provide distributed access to models and associated data. Begin to scale to petabyte. </li></ul>Background Project Goals
    6. 6. <ul><li>Pare down large file sizes of high resolution data and products- and provide flexible inter-operable access. </li></ul><ul><li>(re-) Group different data sets to create needed products – such as initialization files for model development, analysis, or by forecast projection. </li></ul><ul><li>Subset and aggregate the data: </li></ul><ul><li> - in parameter space </li></ul><ul><ul><li> - in physical space </li></ul></ul><ul><ul><li> - in temporal space </li></ul></ul>Background Motivation: Tools for Users
    7. 7. NCDC Archive Motivation for Archive Stewardship
    8. 8. <ul><li>NOAA-wide procedure to identify, appraise, and decide what scientific records are preserved in a NOAA Facility. Then a Submission Agreement (SA) outlines details of dataset. </li></ul><ul><li>Reviewed by a cross-NOAA working group- the Environmental Data Management Committee (EDMC). Long-term stewardship the goal. </li></ul><ul><li>Criteria developed using guidelines from National Archives and Records Administration (NARA), and National Research Council (NRC) reports on NOAA data management, and from other related reviews. </li></ul><ul><li>Technology to provide access and value-added products to deep archive an on-going NCDC activity (NOMADS-NCMP etc.). </li></ul>NCDC Archive Archive Procedures
    9. 9. <ul><li>The Submission Agreement and “What to Archive Process” allows data center to make informed planning decisions </li></ul><ul><li>Provides a formal way to be selective about how data are supported </li></ul><ul><li>Documents the justification for allocating archive support for the data </li></ul><ul><li>Data Reduction policies and recommendations now underway with NOMADS (e.g., remove fcsts > 5 years). </li></ul>NCDC Archive Benefits of Archive Procedures
    10. 10. Private or published results Search tools Metadata Ontologies Public and private catalogue Workflow generation tools Private virtual workspaces Shared virtual workspaces Monitoring & control services Workflow orchestration engine Observing System Simulation Experiments Other (e.g. Unique Instrumentation) Modeling Systems Analyses Datasets Event detection M Earth Systems Modeling Framework National Climate Model Portal NCMP Observing Systems Real-time data streams Middleware, access protocols, secure data transport User authentication, access control logic Metadata vocabularies, ontology standards Users Education and training, user support Compute servers NCDC Archive Digital libraries Task Reanalysis and Climate Clearing-house Community vetted observational database GEO US-GEO Adaptive DOE Earth System Grid & iRODS Q A Pre / Post Processing Rutledge/Meacham/Fontaine 2006 U.S. GEO Modeling Infrastructure Vision Access D D D D data Q Q/A M model A analysis NOAA Climate Services Portal
    11. 11. National Climate Model Portal Web Based Data Services Community <ul><li>Priority Technologies & Partners </li></ul><ul><li>GO-ESSP Community </li></ul><ul><li>NCSP National Climate Service Portal </li></ul><ul><li>- NCPP National Climate Prediction and </li></ul><ul><li>Projections Center (ESRL prototype) </li></ul><ul><li>OPeNDAP, OGC, CF, NetCDF, TDS… </li></ul><ul><li>iRODS Renaissance Computing Institute </li></ul><ul><li>ESGF Earth System Grid Federation </li></ul><ul><li>IPCC & LLNL/PCMDI Archive </li></ul>The NOMADS-NCMP System Data Ingest NCDC Archive NOMADS Archive Interface
    12. 12. <ul><li>British Atmospheric Data Centre </li></ul><ul><ul><li>Bryan Lawrence – Director, British Atmospheric Data Centre </li></ul></ul><ul><li>Geophysical Fluid Dynamics Laboratory </li></ul><ul><ul><li>V. Balaji, Head, Modeling Group, Princeton/GFDL </li></ul></ul><ul><li>The German Climate Computing Centre </li></ul><ul><ul><li>Michael Lautenschlager (NeRC Grid) </li></ul></ul><ul><li>Lawrence Livermore National Laboratory </li></ul><ul><ul><li>Dean Williams, PCMDI, Chief Archive Services/CMIP5 , ESGF </li></ul></ul><ul><li>National Center for Atmospheric Research </li></ul><ul><ul><li>Don Middleton, Senior Manager, Enabling Technologies, ESGF </li></ul></ul><ul><li>Pacific Marine Environmental Laboratory </li></ul><ul><ul><li>Steve Hankin (Unified Access Framework, DMIT) </li></ul></ul><ul><li>NOAA/Earth Systems Research Laboratory </li></ul><ul><ul><li>Cecelia Deluca (National Climate Projection and Prediction NCPP prototype) </li></ul></ul><ul><li>NOAA/National Climatic Data Center </li></ul><ul><ul><li>Glenn Rutledge, (Program Manager NOMADS/NCMP) </li></ul></ul>Global Organization for Earth Systems Science Portals Related Workshop: 2011 GO-ESSP Workshop NCDC hosts the 2011 GO-ESSP Workshop May 9-10, Asheville NC
    13. 13. Questions? [email_address] [email_address] NCDC Asheville, NC M. Sutton 1995 NCMP NOAA National Climate Model Portal Thank you