Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BETTER Session, Excercise 1 @ EO Joint Big Data Hackathon

42 views

Published on

The first exercise showcasing the developed BETTER pipelines at the BETTER project session at the EO Joint Big data Hackathon in Frascati, Italy, 7-8 November 2019. Read more here: https://www.ec-better.eu/pages/h2020-eo-big-data-hackathon

Published in: Technology
  • Be the first to comment

  • Be the first to like this

BETTER Session, Excercise 1 @ EO Joint Big Data Hackathon

  1. 1. Data pipelines for inter-communal conflict impact analysis Terradue BETTER - Nigeria change detection 7-8 Novembre 2019 - Frascati Italy H2020 EARTH OBSERVATION BIG DATA HACKATHON
  2. 2. Nigeria context https://www.wfp.org/countries/nigeria
  3. 3. Earth Observation dataset: Sentinel-1 Search parameters: productType: SLC start: 2018-09-09 end: 2018-09-21 box: 5.1,10.8,8.1,13.7 Sentinel-1 data discovery over Nigeria Goal: discover a pair of Sentinel-1 SLC for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service https://ellip.terradue.com/geobrowser/?id=better-eobd-hack/better-hack-nigeria-app#!
  4. 4. Select the product that “covers the area of interest” Platform: S1A Sensor: SAR Product Type: SLC Start Time: 2018-09-16T17:46:57.0540000Z End Time: 2018-09-16T17:47:24.0030000Z Orbit:23727 ASCENDING Track:30 Mode: IW_DP Swath: IW1 IW2 IW3 Sentinel-1 data discovery over Nigeria Goal: discover a pair of Sentinel-1 SLC for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service
  5. 5. Sentinel-1 data discovery over Nigeria Earth Observation dataset: Sentinel-1 Search parameters: productType: SLC start: 2019-09-09 end: 2019-09-21 orbitDirection: ASCENDING track: 30 Platform: S1A box: 5.1,10.8,8.1,13.7 Goal: discover a pair of Sentinel-1 SLC for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service
  6. 6. Coherence and backscatter change detection COIN Processing service Use the discovered pair of Sentinel-1 SLC procuts to fill the processing parameters: - Master - Slave Then set the parameter - Pixel spacing in meters to 10.0 Run the job Goal: process Sentinel-1 SLC pair for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service
  7. 7. Coherence and backscatter change detection COIN Processing service Use the discovered pair of Sentinel-1 SLC procuts to fill the processing parameters: - Master - Slave Then set the parameter - Pixel spacing in meters to 10.0 Run the job Goal: process Sentinel-1 SLC pair for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service
  8. 8. Change detection visualization
  9. 9. Hands-on: data discovery, processing and data pipelines Steps: 1. Set-up your credentials 2. Data discovery 3. COIN processing 4. COIN results 5. Sentinel-2 RGB composites data processing pipeline 6. SNAC data processing pipeline 7. COIN data processing pipeline 8. SNAC exploitation Hackathon kit available at: https://gitlab.com/ellip/training/wfp-nigeria-training
  10. 10. Hands-on: access and setup Open a Web Browser (best experience with Chrome) Type the address: https://notebooks.terradue.com You’ll be redirected to a login page. Use the credentials found on the card given to you
  11. 11. Hands-on: access and setup Open the folder: wfp-nigeria-training Double-click on the notebook: 1 Setup.ipybnb Run the cells and fill the information requested: username: same as in card API key: click on the link shown copy/paste the API key Goal: set-up the authentication/authorization information for all notebooks
  12. 12. Hands-on: data discovery Double-click on the notebook: 2 Data discovery.ipybnb Run the cells to repeat the data discovery steps done on the Geobrowser Goal: discover the pair of Sentinel-1 SLC acquisitions with the Ellip APIs following the same approach done using the Geobrowser
  13. 13. Hands-on: data discovery Goal: discover the pair of Sentinel-1 SLC acquisitions with the Ellip APIs following the same approach done using the Geobrowser
  14. 14. Hands-on: Sentinel-1 COIN processing Double-click on the notebook: 3 COIN processing.ipybnb Run the cells to submit the COIN processing request as done on the Geobrowser Goal: discover the COIN processing service and submit a processing request with the discovered pair of Sentinel-1 SLC acquisitions
  15. 15. Hands-on: RGB composites data processing pipeline Double-click on the notebook: 4 COIN results.ipybnb Run the cells to inspect the processing results Goal: explore the results of the COIN processing service
  16. 16. Data processing pipelines rely on the ATC model with: - Application - data driven application - Trigger - date or event driven application - Coordinator - schedule the execution of the Trigger instances periodically at fixed times, dates, or intervals Data processing pipelines
  17. 17. The ATC model starts with a data driven application taking as inputs one or more references to EO catalogue entries: - A single input reference - A pair of input references (e.g. interferometry) - A stack of input references A(pplication)TC
  18. 18. The Trigger is a date and/or event driven application that, when there’s new data available or an event: - Creates Data Items - Queues Data Items - Pipes Data Items AT(rigger)C
  19. 19. OWS Context (*) documents encapsulating all the information for processing the application and get it through the data processing pipeline AT(rigger)C - Data items (*) https://www.opengeospatial.org/standards/owc
  20. 20. The Coordinator submits Trigger instances periodically at fixed times, dates, or intervals ATC(oordinator)
  21. 21. Hands-on: RGB composites data processing pipeline Double-click on the notebook: 5 RGB composites data processing pipeline.ipybnb Run the cells to create and queue Data Items containing Sentinel-2 RGB Composites processing requests The data processing pipeline will produce, for a set of areas of interest, several RGB composites for all Sentinel-2 acquisitions during the period 2019-08-15 to 09-15. Goal: create a data processing pipeline for Sentinel-2 RGB composites over a few areas of interest in Nigeria
  22. 22. Hands-on: SNAC data processing pipeline Double-click on the notebook: 6 SNAC data processing pipeline.ipybnb Run the cells to create and queue Data Items containing SNAC processing requests Goal: create a data processing pipeline for Sentinel-1 SNAC over a large area in Nigeria
  23. 23. Hands-on: SNAC data processing pipeline
  24. 24. Hands-on: COIN data processing pipeline Double-click on the notebook: 7 COIN data processing pipeline.ipybnb Run the cells to create and queue Data Items containing COIN processing requests Goal: create a data processing pipeline for Sentinel-2 RGB composites over a few areas of interest in NigeriaGoal: create a data processing pipeline for Sentinel-1 COIN over a large area in Nigeria
  25. 25. Hands-on: COIN data processing pipeline
  26. 26. Hands-on: monitor a data processing pipeline Double-click on the notebook: 8 SNAC exploitation.ipybnb Monitor the Data Items status and access the associated results when done Goal: monitor and exploit the results of the Sentinel-1 SNAC data processing pipeline over a large area in Nigeria
  27. 27. ▪ Get in touch with Terradue: support(at)terradue.com www.terradue.com ▪ Become a BETTER data challenge promoter: https://www.ec-better.eu/ Way forward Horizon 2020 Research and Innovation Programme under grant agreement no 776280

×