More Related Content

Slideshows for you(20)

Similar to Phidias: Steps forward in detection and identification of anomalous atmospheric events(20)


Phidias: Steps forward in detection and identification of anomalous atmospheric events

  1. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS: Boosting the use of cloud services for marine data management, services and processing Webinar | October 13, 2020, 15:00 CEST
  2. Objectives 213.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc Develop a catalogue that will allow users to discover and access data, open-source software, public Application Programming Interfaces (APIs) and interactive processing services. Optimise and industrialize workflows to allow the largest degree of reusability of data as possible. Implement an end-user web common interactive processing service based on notebook and datacube technologies allowing new users to easily have open access to HPC capacities and develop new algorithms. Improve the FAIRisation of satellite and environmental datasets and preserve FAIR (Findable Accessible Interoperable Reusable) datasets into a Remote Data Access (RDA) certified repository. Develop new data pre-processing models coupled with HPC capabilities, by building a new and innovative on-the-fly computing service for smart processing of data, addressing the problem of efficient filtering and on-the-fly first diagnostics on incoming in-situ data. Deploy data post-processing methods as a service for several end-users, including scientific communities, public authorities, private entities and citizen scientists. Consortium of 13 committed partners from 5 different EU countries, led by CINES (French national Computing Centre for Higher Education)
  3. Webinar Series 313.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc Next Webinar will be on «Big Data Earth Observation» Use Case – All recorded Webinars are available on Phidias HPC website under the Webinar page
  4. Webinar Agenda 15:00 - 15:05 - Introduction of PHIDIAS project Francesco Osimanti, Trust-IT Services, PHIDIAS WP7 Leader 15:05 - 15:15 - Thematic and objective: proposed approaches and available Sentinel measurements Pascal Prunet, SPASCIA & PHIDIAS WP4 Leader 15:15 - 15:25 - HPC/HPDA Infrastrucuture and capacities for implementing intelligent screening in an operational context Nicolas Pascal, ICARE & SPASCIA Technical Partner 15:25 - 15:30 - Q&A Session 15:30 - 15:40 - Implementation of processing and products Dominique Jolivet, HYGEOS & SPASCIA Technical Partner 15:40 - 15:50 - ESCAPE: A dive into a Datalake for Open Science Xavier Espinal, CERN & ESCAPE project 15:50 - 16:00 - Q&A and Closing Remarks 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc 4
  5. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS: Steps forward in detection and identification of anomalous atmospheric events from space Webinar | October 13, 2020, 15:00 CEST Pascal PRUNET, CEO SPASCIA SPASCIA
  6. 6 Earth and environment challenges at local, regional and global scales : continuous increase of human activities modify the atmospheric composition Impact on environment (climate; soil, water and air quality; biodiversity) Impact on health, on economy Needs to understand and to monitor for acting European answers for better adressing these challenges : Europe is entering in the era of operational measurement of the air composition from space, for the analysis and forecasting of chemical weather and climate monitoring CONTEXT First measurements already available : IASI, GOME (Metop) Sentinel 5 precursor (S5P/TROPOMI) 4 Operational sounding instruments from 2022-2023 : Sentinel 4 and 5 onboard Metop-NG and MTG Copernicus Operational forecasting services for atmospheric chemistry (CAMS) and Climate (C3S) Development of the European capacity for the measurement of GHG for monitoring human emissions 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  7. 7 From 2022, European atmospheric sounding missions will deliver each day several TB (terabytes) of raw datacubes at high spatial/temporal/spectral resolutions. This represents an unprecedented amount of atmospheric data, with improved quality and coverage. How to comprehensively deal with all available information ? ➢ Key challenge is to provide the capacity of intelligent screening of large amounts of satellite data for targeting scenes or events of interest in view of their dedicated processing or exploitation. PHIDIAS addresses those needs by using HPC and HPDA capacities of intelligent screening approaches in an operational context, for detection and identification of atmospheric composition events. . CONTEXT Onboard geostationary MTG satellite, the Sentinel-4 mission comprises an UVN instrument and data InfraRed Sounder (IRS), completed by FCI Onboard polar-orbiting MetOp SG satellite, the Sentinel-5 mission comprises an UVNS instrument and data from IASI-NG, MetImage and 3MI Each 30 minutes over Europe Each day over the globe 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  8. 8 PHIDIAS ADDED VALUE Earth observation from space is underexploited Huge quantity of data, increasing information content and complexity Operational forecasting systems Efficient, real-time data assimilation in models, but could not use « extreme » data related with « extreme events » Reduced added value of the data Research groups, scientific institutes Innovative and well adapted to event studies but case by case exploitation of the data Not exhaustive exploitation, no real-time analysis Detection and Filtering in real time of relevant data Realtime Detection & analyse of extreme events, pollutant plumes, other targeted events Dedicated services Early warning, monitoring, decision support Added-value : Robust detection, objective filtering Added-value : Verification, improvement Innovation 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  9. 9 HOW ? Earth observation from space Sentinel 5 Precursor (S5P) L1B, L2 PHIDIAS Pilot users : On the flow, realtime detection & analysis of anomalous events Complementary dedicated services Early warning, monitoring, decision support PCA-based screening of L1B data AI-based Detection of gas plume and sources from L2 products On demand processing of the data for detection of plumes and sources over a region/period specified by the user Catalogue/Metadata of selected and characterised L1B data Processed L2 products with additional characteristics L1B data L2 product 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  10. 10 This use case propose to develop, test and prototype the approach with Sentinel 5 Precursor data/products : Level 1 data … DATA 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  11. 11 … and Level 2 products DATA 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  12. 12 WP4 WORK PLAN Objective : intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric composition events Two processing prototypes are addressed proposed for development and test : 1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods developed for IASI, implementation and consolidation of algorithms and tools for generic processing of atmospheric spectra recorded by S5P. Real-time, systematic processing of data 2. New AI methods for objective/automatic detection of plumes from L2 products (CO, CH4 NO2, SO2): dedicated AI methods based on Particle Swarm Optimisation approaches and exploiting similitude with animal behavior modelling -> comprehensive exploitation of spatial and temporal information for detecting plumes and sources. On demand processing : historical data or real time processing 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  13. 13 1. PCA-based screening of L1 data – Example of IASI Nominal case: denoising filter Extreme case: identification of the residual signal Reconstruction of "scores" maps: rapid detection of anomalous situations Analysis of "residuals" : interpretation of the anomalous event 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  14. 14 1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Status: ➢ 1 year, global scale of S5P Level 1 (L1) data available at ICARE ➢ PCA-screening processing under development (adapted from IASI-PCA developed with CNES) ▪ Processing already tested on IASI real data ▪ Development/adaptation at HYGEOS is ongoing ➢ Analysis and pre-processing of L1 ShortWave InfraRed (SWIR) spectra (bands 7 and 8) is ongoing, in order to be ingested by the PCA processing ▪ Issue: complex but necessary filtering of bad pixels for exploiting the whole spectrum ▪ Analysis of spectral micro-windows: indicators for detection of pollutants (air quality, greenhouse gases) Next steps : first tests of PCA processing on small subsets of S5P data; generation of global scale database for the learning phase; implementation at ICARE for test and prototyping WORK PROGRESS 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  15. 15 2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others?: CH4 NO2, SO2) Status: ➢ Global S5P Level 2 products since 2018 available at ICARE ➢ Preparation and analysis of S5P Level 2 product (NO2) ▪ Analysis of image over pollution sources (see side results in the next slide) ▪ Implementation of dedicated storage of the product, optimised for speedy retrievals given {geolocation, time} constraints over the whole dataset (2018 - present) ➢ Development of the AI engine for source and plume detection ▪ Neural network agents, stochastic optimization process ▪ Selection of data subsets for the learning phase ➢ Alternative method analyzing significant signal enhancements Next steps : learning phase and tests on S5P data subsets WORK PROGRESS 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  16. 16 Project objectives ANALYSIS OF S5P PRODUCT OF NO2 FOR DETECTING POLLUTION LEVEL AT CITY SCALE P. PRUNET, O. LEZEAUX, C CAMY-PEYRET, H. THEVENON, ACCEPTED IN CITY AND ENVIRONMENT INTERACTIONS ➢ From ESA S5P measurements of the air pollutant NO2 from space, we have assessed the impact of the human activity reduction on air pollution by comparing the first 4-months periods of 2019 and 2020 on a daily, weekly and monthly basis for 4 major cities in Europe PARIS 2019 monthly averages MILAN MADRID ATHENS 2020 Monthly averages Reductions in the pollution level (using NO2 tropospheric column as a proxy) have been observed from Mid March and for April 2020 (52% +/-9% for Paris; 28% +/-8% for Milan region; 54% +/-16% for Madrid; not significantly observed for Athens), as compared to the same periods in 2019 PARIS : Reduction rates of the NO2 plume mass fortnightly mean between 2019 and 2020 from mid-March to end of April, and ✓ Start dealing with the data ✓ Understand the information ✓ Show the added-value of S5P 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  17. 17 Space-based Earth observation will provide amazing amount of complex, informative data : How to extract and use comprehensively this increasing information ? Operational Models and systems have strong limitation to exploit « extreme » data, i.e., too far from the model Research face to the necessity to focus their analyses of the underlined information and science PHIDIAS will provide new tools and approaches to deal with the measurements and the information, allowing better and more efficient use of the data, and pave the ways for finding new paradigms AIM XCO2 IFS model simulation provided by ECMWF (from H2020 ECMWF CHE project) : 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  18. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS: Steps forward in detection and identification of anomalous atmospheric events Webinar | October 13, 2020, 15:00 CEST Nicolas Pascal, AERIS/ICARE technical director
  19. AERIS/ICARE ? A Data and Services Centre mostly dedicated to satellite data, but also ground-based and model data Part of the Data Terra Research Infrastructure 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc 19 • 6,7 PB of usable storage capacity, • ~1200 cores dedicated to production, • 10 Gb bidirectional connection to RENATER.
  20. Towards a interoperable infrastructure 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc 20 Credits: R. Moreno sedoo
  21. Why CINES and AERIS/ICARE ? Convergence needed for HPDA algorithms: processing of big data volume with big computing resources CINES: access to important computing resources, experts in HPC and data sharing techniques. AERIS/ICARE: access to big volume satellite database, close to the atmosphere science community, experienced in science prototypes industrialization. 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc 21
  22. High level implementation scheme 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc 22 S5-P data ~500 GB/day Data sharing/caching Trigger process Get result data Users Trigger process Get result data Systematic processing On-demand processing W e b P o r t a l Users Data discovery Get result data Data users
  23. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. PHIDIAS WP4 : Technical implementation Webinar | October 13, 2020, 15:00 CEST Dominique Jolivet, HYGEOS
  24. OUTLINE State of the art : IASI-PCA PCA processing of IASI level 1 data already tested (CNES, SPASCIA, HYGEOS, LATMOS) Existing approach and methodology Promising results On going work : implementation for Sentinel 5P : Generation of a reference basis L1B reader for S5P data PCA analysis Challenges with S5P L1 data 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc 24
  25. PROCESSING SCHEME 13.10.2020 25PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  26. 26 CO NH3 This method has been successfully applied to IASI on Metop (SPASCIA, HYGEOS, LATMOS collaboration, CNES funding) More than 60 indicators (scores) of extreme atmospheric events (concerning more than 10 molecules: CO, NH3, CH4, SO2, HCl, ….). IASI experience Detection of fire events : exemple of IASI, 10/2017
  27. S5P - Reference basis Has to be done once. Can be seen as an ancillary data Composed of more than 75 000 spectra random – oriented selection Covering a long temporal period (typically one year): day 1, 6, 11, 16, 21 and 26 of each month are selected About 75 spectra per orbit: orientated random selection to extract a representative set of : Scanline and ground pixel (IFOV) number Solar and viewing zenith angles (SZA) Latitude/longitude 13.10.2020 27PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  28. S5P-L1B reader 13.10.2020 28 For one scanline Spectral channel Ground pixel Read L1B data measurements (scanline x ground pixel x spectrum) And Associated flags For one ground pixel into a scanline line (so for one spectrum): Detect and correct anomalous measurements Filter if neccessary (use or not) And normalization Temporary output: filtered normalized spectra Challenging for S5P PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  29. PCA 13.10.2020 29PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  30. S5P CHALLENGES (1) Code written in python (core API) Processing at the scale of an orbit (divided into chunks – arrays of data) S5P versus IASI: much more input data (but only SWIR bands BD7 and BD8), and only daytime data will be used S5P L1 data versus IASI Dealing with noise → the noise is radiance dependent hence scene dependent (different from IASI) Normalization → the instrument covariance matrix is to be established (more complicated than for IASI) Spectral indicators → to be defined (as for IASI) searching for the most appropriate spectral signatures of CH4 (first priority) and CO (second priority, since strongly perturbed by CH4 features). H2O features are also to be considered because of their ubiquity and large variability. Other species will be searched for in the discovery or extreme event mode. Day/night transition → since the recorded signal is the solar radiation reflected by the surface/atmosphere system, low sun scenes will be filtered out (threshold for high solar zenith angles, SZA) Does a huge amount of input data mean a huge amount of output data ? 13.10.2020 30PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  31. S5P CHALLENGES (2) Main output: logbook of atmospheric events detected (netcdf file). How to be efficient and useful for users and researchers ? Where is the L1B data: L1B filename (attribute), scanline, ground pixel Ground pixel quality flag (from L1B) Spatial geolocation: latitude/longitude Which indicators will provide an alert (detection flag coded in 8 or 16 bytes depending on the number of indicators) Values of the indicators (mean and standard deviation of the residual difference) Other outputs (to be investigated and validated by ICARE): Maps Post-processing for the whole day Tools to link to L1B or L2 data corresponding to detected events ? Additional difficulties: bad pixels always present in 2D SWIR focal arrays Need to establish the map of bad pixels in the spatial (ifov) and spectral (channel) plane using solar calibration views (onboard diffuser plate) acquired in the polar region for one orbit per day 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc 31
  32. The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854. Xavier Espinal (CERN) - ESCAPE WP2 leader ESCAPE - A dive into a Datalake for Open Science Webinar - Steps forward in detection and identification of anomalous atmospheric events 13 Oct 2020
  33. - Contributes to deliver Open Access and FAIR data services: trustable data repositories; enable data management policies; transparent data access layer The ESCAPE Data Infrastructure for Open Science - Define, integrate and commission an ecosystem of tools and services to build a data lake - Science projects to drive the services requirements most suitable to their needs PHIDIAS webinar 13/10/20 Steps forward in detection and identification of anomalous atmospheric events
  34. 36
  35. 37 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  36. 38 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  37. 39 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  38. 40 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  39. 41 13.10.2020 PHIDIAS Webinar | 13.10.2020 | | @PhidiasHpc
  40. Thank-you 13.02.2020 PHIDIAS Webinar | 13.02.2020 | | @PhidiasHpc 42