The ESA SCIENTIFIC
TESTBED in CASPAR

S. ALBANI & L. FUSCO (ESA)
     Sergio.Albani@esa.int, Luigi.Fusco@esa.int




DPE, ...
SUMMARY
•   ESA EO data handling
•   ESA data: benefits, risks and policies
•   The ESA scientific testbed in CASPAR
•   C...
ESA EO Data Handling


• ESA, through worldwide
receiving ground stations, acquires
data from Earth Observation (EO)
satel...
The EO data avalanche…

                                                      Evolution of ESA's EO Historical Data Archiv...
Benefits, risks and policies (1)


• EO archives ensure a global coverage of the Earth with the
  following important char...
Benefits, risks and policies (2)


• We have to preserve data against changes in:
  – hardware,
  – software,
  – environm...
Benefits, risks and policies (3)


• So ESA is currently proposing to Member States to establish a
  viable European-wide ...
ESA TESTBED ACTIVITIES


The ESA testbed is covering:
  • the setup of the framework in ESA-ESRIN;
  • the definition and ...
Testbed: HW deployment in ESRIN
Testbed: the GOME dataset (1)


•   The ESA selected dataset as a suitable demonstration case in
    the framework of the ...
Testbed: the GOME dataset (2)

      Level 0: raw data
      Level processors
      Auxiliary data
      Documents and met...
ESA TESTBED ACTIVITIES


The ESA testbed is covering:
  •   the setup of the framework in ESA-ESRIN;
  •   the definition ...
Testbed: going to OAIS


                                   Content Information
                                          ...
ESA TESTBED ACTIVITIES


The ESA testbed is covering:
  •   the setup of the framework in ESA-ESRIN;
  •   the definition ...
Testbed: preservation scenario (1)


                                         We want to preserve the
          Auxiliary ...
Testbed: preservation scenario (2)


                                                      We have to preserve the
       ...
Testbed: preservation scenario (3)



GOME L1 dataset       L1BL1C processor

 General documents     Gome L1 products    ...
CONCLUSIONS


• The preservation of EO data is of vital importance
  for the scientific and operational user communities.
...
THANKS!
Upcoming SlideShare
Loading in …5
×

Esa Scientifics Testbed Sergio Albani

795
-1

Published on

3rd Annual WePreserve Conference Nice 2008

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
795
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Esa Scientifics Testbed Sergio Albani

  1. 1. The ESA SCIENTIFIC TESTBED in CASPAR S. ALBANI & L. FUSCO (ESA) Sergio.Albani@esa.int, Luigi.Fusco@esa.int DPE, Planets and CASPAR Third Annual Conference Nice 29/30 October 2008
  2. 2. SUMMARY • ESA EO data handling • ESA data: benefits, risks and policies • The ESA scientific testbed in CASPAR • Conclusions
  3. 3. ESA EO Data Handling • ESA, through worldwide receiving ground stations, acquires data from Earth Observation (EO) satellites and archives/processes them at Processing and Archiving Facilities. • ESA-ESRIN is the largest European EO data provider and operates as the reference European centre for EO Payload Data Exploitation. • At present, several thousands ESA worldwide users have online access to EO missions related metadata (10 million references), data (about 3 PB) and derived information.
  4. 4. The EO data avalanche… Evolution of ESA's EO Historical Data Archives between 1986-2006 ESA EO data archives include data from ESA missions LANDSAT 2-4 MSS (75-Dec 93) (ERS and ENVISAT) 3000 and Third Party missions 2850 AQUA Modis (April 03-today) (Landsat, SPOT, ALOS, TIROS, 2700 2550 ENVISAT LR (March 02-today) AQUA&TERRA, etc.). 2400 ENVISAT HR (March 02-today) 2250 2100 TERRA Modis (June 01-today) The Total Archive in TerraBytes (TB) 1950 1800 1650 QUICK SCATT (01-today) /PROBA (May 02- today) LANDSAT 7 ETM (April 99-Dec 03) evolution 1500 1350 1200 SEA STAR SeaWifs (Apr 98-today) follows a 1050 900 750 ERS 2 HR (May 95-today) ERS 2 LBR (May 95-today) similar trend 600 450 300 JERS SAR/OPS VNIR (92-Sep 98) for all 150 0 ERS 1 HR (Jul 91-Mar 00) ERS 1 LBR (Jul 91-Mar 00) national 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 Year SPOT 1-4 HRV (87-today) archives. 14 Upcoming ESA missions and GMES pre-operations. 12 Data (Petabytes) 10 8 The plans of new missions indicates 5-10 times 6 more data to be archived in next 10-15 years 4 2 0 1980 1985 1990 1995 2000 2005 2010 2015 2020
  5. 5. Benefits, risks and policies (1) • EO archives ensure a global coverage of the Earth with the following important characteristics: – multi-sensor data (from optical to active radar sensors); – long series (time-span that extends from a few years to decades); – variable geometrical resolution (from few meters to few hundreds meters); – variable geographical coverage (local, regional, global); – variable temporal resolution (from few days up to months). • The requirements for accessing ESA historical archives is strongly increased in the last ten years and the trend is likely to increase in the future mainly for long term science and environmental monitoring. • Therefore, the prospect of losing the digital records of science (and with the specific unique data, information and publications managed by ESA) is very alarming.
  6. 6. Benefits, risks and policies (2) • We have to preserve data against changes in: – hardware, – software, – environment, – knowledge base of the scientific community… • But current funds for ESA missions cover preservation and access to data for a baseline period limited to only 10 years after acquisition! • The issues concern: – type and amount of data to be preserved; – location of archives and their replication for security reasons; – technical choices (e.g. formats, media); – availability of adequate funds. • Decisions has to be taken in coordination with other data owners and with the support/advice of the user community.
  7. 7. Benefits, risks and policies (3) • So ESA is currently proposing to Member States to establish a viable European-wide infrastructure for permanent access to the records of science (data and publications) by the definition of: – an overall strategy for the EO data long term preservation; – an harmonized European archives management policy for ESA and Member States national EO data holdings . • Meanwhile ESA: – is active in developing appropriate techniques and strategies by promoting and participating to projects related to long term data preservation; – is participating to the CASPAR project playing the role of both user and infrastructure provider for the scientific data testbed using new proposed standards, specialized infrastructures and open GRID environment.
  8. 8. ESA TESTBED ACTIVITIES The ESA testbed is covering: • the setup of the framework in ESA-ESRIN; • the definition and collection of a significant sample of a whole processing chain dataset (viewers, converters, processors and data of different level, revision and format); • the conversion of the data from the native format to a OAIS compliant format; • the generation of appropriate Representation Information, Descriptive Information, Knowledge Modules and Scientific Community profiles; • the analysis of ontologies to describe and preserve scientific workflows (e.g. the applicability of CIDOC CRM on scientific data); • the ingestion of data in the CASPAR system; • the coping with some long term data preservation problems by using the CASPAR components, methodology and tools.
  9. 9. Testbed: HW deployment in ESRIN
  10. 10. Testbed: the GOME dataset (1) • The ESA selected dataset as a suitable demonstration case in the framework of the scientific testbed of the CASPAR project consists of data from GOME (Global Ozone Monitoring Experiment), a sensor on board the ESA ERS-2 (European Remote Sensing) satellite. • The GOME dataset: – has a big total amount of information distributed with a high level of complexity; – is unique because it provides more than 11 years global worldwide coverage; – is very important for the scientific community and the Principal Investigators (e.g. KNMI and DLR) that on a routine basis receive GOME data for their research projects (e.g. concerning ozone depletion or climate change); – is just a test-case because similar issues involve many other Earth Observation instrument datasets.
  11. 11. Testbed: the GOME dataset (2) Level 0: raw data Level processors Auxiliary data Documents and methods Level 1: radiances/reflectances Level 2: geophysical data as trace gas amounts Level 3: a mosaic composed by several level 2 data with interpolation of data values to fill the satellite gaps Data viewers Examples of GOME science applications Format converters
  12. 12. ESA TESTBED ACTIVITIES The ESA testbed is covering: • the setup of the framework in ESA-ESRIN; • the definition and collection of a significant sample of a whole processing chain dataset (viewers, converters, processors and data of different level, revision and format); • the conversion of the data from the native format to a OAIS compliant format; • the generation of appropriate Representation Information, Description Information, Knowledge Modules and Scientific Community profiles; • the analysis of ontologies to describe and preserve scientific workflows (e.g. the applicability of CIDOC CRM on scientific data); • the ingestion of data in the CASPAR system; • the coping with some long term data preservation problems by • using the CASPAR components, methodology and tools.
  13. 13. Testbed: going to OAIS Content Information The RepInfo provided are Descriptive contained in the manifest Data Object Representation Information Information file and in the schemas. The filename itself. Metadata is extracted by the product and contained Preservation Description Information Packaging in the SAFE manifest. Information Reference Provenance Empty. Context Fixity A Cyclical SAFE AIP Redundancy Check (CRC) code for a file. The principal investigator who recorded the data and the information concerning its storage, handling and migration. No packaging restrictions.
  14. 14. ESA TESTBED ACTIVITIES The ESA testbed is covering: • the setup of the framework in ESA-ESRIN; • the definition and collection of a significant sample of a whole processing chain dataset (viewers, converters, processors and data of different level, revision and format); • the conversion of the data from the native format to a OAIS compliant format; • the generation of appropriate Representation Information, Descriptive Information, Knowledge Modules and Scientific Community profiles; • the analysis of ontologies to describe and preserve scientific workflows (e.g. the applicability of CIDOC CRM on scientific data); • the ingestion of data in the CASPAR system; • the coping with some long term data preservation problems by using the CASPAR components, methodology and tools.
  15. 15. Testbed: preservation scenario (1) We want to preserve the Auxiliary data ability to process data from level 1b to level 1c. Level 0 data L0->L1b processor L1b data Raw signals Fully Levelplus data 1b L1b->L1c processor L1c data calibrated calibration data data Level 1b data L1b->L2 processor L2 data Docs & Knowledge
  16. 16. Testbed: preservation scenario (2) We have to preserve the GOME L1 products, processors, reference manuals, etc. GOME L1 L1BL1C L1BL1C products processor source code PSD.pdf readme_1st.doc license.doc readme.doc The C Bible disclaimer.pdf release_l01.doc The OS Bible user_manual.pdf howtouse_l01.doc ERS-Products.pdf The Ozone ProductSpecification.pdf The ERS-2 satellite The GOME sensor
  17. 17. Testbed: preservation scenario (3) GOME L1 dataset L1BL1C processor General documents Gome L1 products Scientific Community Profile 1: L1BL1C source code L1B->L1C documents Scientific Community Profile 2: Ozone; Data and processors ERS-2 satellite; Ozone; GOME sensor; are ingested ERS-2 satellite; L0 & L1 data; GOME sensor. L1b->c algorithms; in the CASPAR system The Preservation&Scenario: RepInfo PDI L1b->c processors. CASPAR (Provenance something changes and a new Events chain & Context) L1b products OS or gLib Version Change 1. plus the processor COMPONENTS (FIND, PDS, KM, POM…)to be developed/ingested processor has update and documents are returned to the are returned to the user with 2. Alert to preserve the ability to process user profile 1 to2 data from profile L1c. with L1b 3. Source Code Retrieve that asks for 4. Processor Recompiled corrected L1 data. 5. Processor Reingested 6. Docs&Links Updated 7. Changes Notify
  18. 18. CONCLUSIONS • The preservation of EO data is of vital importance for the scientific and operational user communities. • ESA is pursuing the objective to ensure the perpetual preservation of these data in coordination with institutions of its member states. • ESA data preservation initiatives will benefit of the results of CASPAR (and other similar EU sponsored projects) by adopting, when applicable, technical solutions and procedures developed in the framework of these cooperative projects.
  19. 19. THANKS!

×