Big data ET models & benchmarking
with distributed OSGEO tools

Yann Chemin
International Water Management Institute

Yann...
Contents

1
2

Introduction
Evapotranspiration Modeling
ET & Equity in Irrigation
Crop Water Consumption
River Basins Moni...
CGIAR
Consultative Group for International Agricultural Research
Ratified on October 2nd, 2013
Full Open Access & Open Sour...
Outline

1
2

Introduction
Evapotranspiration Modeling
ET & Equity in Irrigation
Crop Water Consumption
River Basins Monit...
Introduction

Evapotranspiration is the largest transiting quantity in the daily
hydrological cycle along with rain. It is...
Outline

1
2

Introduction
Evapotranspiration Modeling
ET & Equity in Irrigation
Crop Water Consumption
River Basins Monit...
Overview
There are several types of evapotranspiration modeling methods:
Reference ET: Hargreaves, Penman-Monteith
Potenti...
Equity of water use in irrigation systems

Irrigation water monitoring & management
Map: Uniform colour is equity of water...
Crop water consumption in irrigation systems
Actual evapotranspiration (mm/d, daily, 11 years)
for agricultural water perf...
Code for

Yann Chemin (IWMI)

HPC benchmarking ET models OSGEO

November 2nd, 2013

10 / 29
Water depletion in River Basins
Actual evapotranspiration (mm/d, daily, 7 years)
for the Murray-Darling Basin (1M Km2 )
Ja...
Water depletion in River Basins
Actual evapotranspiration (mm/d, daily, 7 years)
for the Murray-Darling Basin (1M Km2 )
Ja...
Evapotranspiration @ country level
Actual evapotranspiration (365 days integrated)
for water resources monitoring & manage...
ET models Benchmarking
Average ET for Sri Lanka, Daily 2000-2012 (mm/d, daily, 12.3 years)

Comparison
ETo & ETpot (rad) a...
Outline

1
2

Introduction
Evapotranspiration Modeling
ET & Equity in Irrigation
Crop Water Consumption
River Basins Monit...
Chain processing

Chain processing has a fundamental impact on remote sensing work:
Standardization limits bugs
Less prone...
Blueprint

GDAL+[C+OpenMP]
GRASSGIS+pyGRASS+[C+OpenMP]
Yann Chemin (IWMI)

HPC benchmarking ET models OSGEO

November 2nd,...
GDAL framework

GDAL+[C+OpenMP]

Yann Chemin (IWMI)

HPC benchmarking ET models OSGEO

November 2nd, 2013

18 / 29
GDAL framework

OpenMP Distribution

Yann Chemin (IWMI)

HPC benchmarking ET models OSGEO

Cores

November 2nd, 2013

19 /...
GRASS GIS framework

metaModule concept
pyGRASS: vertical integration of GRASS GIS modules
GRASS GIS modules: [C+OpenMP]
Y...
pyGRASS
Summary for Landsat 7 pyGRASS MetaModule

http://grasswiki.osgeo.org/wiki/Python/pygrass
Yann Chemin (IWMI)

HPC b...
Outline

1
2

Introduction
Evapotranspiration Modeling
ET & Equity in Irrigation
Crop Water Consumption
River Basins Monit...
Future: 128-cores from Tile-GX

64-core Tile-GX

Yann Chemin (IWMI)

Dual Tile-GX on 1/2 rack board

HPC benchmarking ET m...
Future: 120-cores from Xeon Phi

60-core Phi

Yann Chemin (IWMI)

Dual Phi in Gygabyte cage

HPC benchmarking ET models OS...
Outlooks

Multi-Cores Hardware (Tile-GX, Xeon Phi)
Multi-GPU distribution (OpenCL, CUDA)
Multi-CPU distribution (MPI)
MODI...
Outline

1
2

Introduction
Evapotranspiration Modeling
ET & Equity in Irrigation
Crop Water Consumption
River Basins Monit...
Conclusions

Distributed ET models benchmarking setup with OSGEO tools
GDAL: C+OpenMP
GDAL: Core-based scaling
GRASS GIS: ...
Collaboration call

We are looking for collaborators:
Having HPC capabilities
Interested in Global Water ET monitoring fro...
Thank You

Yann Chemin (IWMI)

HPC benchmarking ET models OSGEO

November 2nd, 2013

29 / 29
Upcoming SlideShare
Loading in …5
×

Big data ET models & benchmarking with distributed OSGEO tools

788 views

Published on

FOSS4G2013 TOKYO 基調講演:Big data ET models & benchmarking with distributed OSGEO tools
Dr. Yann Chemin (OSGeo Charter Member, International Water Management Institute (IWMI))

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
788
On SlideShare
0
From Embeds
0
Number of Embeds
35
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Big data ET models & benchmarking with distributed OSGEO tools

  1. 1. Big data ET models & benchmarking with distributed OSGEO tools Yann Chemin International Water Management Institute Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 1 / 29
  2. 2. Contents 1 2 Introduction Evapotranspiration Modeling ET & Equity in Irrigation Crop Water Consumption River Basins Monitoring Country Monitoring ET Models Benchmarking Yann Chemin (IWMI) 3 4 5 Frameworks Chain processing Blueprint GDAL GRASS GIS Future Conclusions HPC benchmarking ET models OSGEO November 2nd, 2013 2 / 29
  3. 3. CGIAR Consultative Group for International Agricultural Research Ratified on October 2nd, 2013 Full Open Access & Open Source research data and publication International Public Goods Public Domain Publications Open Access FOSS models and algorithms 2018: all 15 CG centres, already FOSS4G Lab: (gsl.worldagroforestry.org) Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 3 / 29
  4. 4. Outline 1 2 Introduction Evapotranspiration Modeling ET & Equity in Irrigation Crop Water Consumption River Basins Monitoring Country Monitoring ET Models Benchmarking Yann Chemin (IWMI) 3 4 5 Frameworks Chain processing Blueprint GDAL GRASS GIS Future Conclusions HPC benchmarking ET models OSGEO November 2nd, 2013 4 / 29
  5. 5. Introduction Evapotranspiration is the largest transiting quantity in the daily hydrological cycle along with rain. It is used by scientists and managers in: Irrigation systems performance Crop water productivity Water accounting Wetlands-agriculture interface Basin water uses quantification Climate change on water cycle & users Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 5 / 29
  6. 6. Outline 1 2 Introduction Evapotranspiration Modeling ET & Equity in Irrigation Crop Water Consumption River Basins Monitoring Country Monitoring ET Models Benchmarking Yann Chemin (IWMI) 3 4 5 Frameworks Chain processing Blueprint GDAL GRASS GIS Future Conclusions HPC benchmarking ET models OSGEO November 2nd, 2013 6 / 29
  7. 7. Overview There are several types of evapotranspiration modeling methods: Reference ET: Hargreaves, Penman-Monteith Potential ET: Priestley-Taylor, astronomical Actual ET: Thermodynamic/energy balance (mostly) OSGEO tools Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 7 / 29
  8. 8. Equity of water use in irrigation systems Irrigation water monitoring & management Map: Uniform colour is equity of water distribution Graph: Irrigation system equity in time (mm/d, daily, 12 years) Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 8 / 29
  9. 9. Crop water consumption in irrigation systems Actual evapotranspiration (mm/d, daily, 11 years) for agricultural water performance management. Irrigation System is 70,000 ha. Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 9 / 29
  10. 10. Code for Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 10 / 29
  11. 11. Water depletion in River Basins Actual evapotranspiration (mm/d, daily, 7 years) for the Murray-Darling Basin (1M Km2 ) Japan is 0.37M Km2 Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 11 / 29
  12. 12. Water depletion in River Basins Actual evapotranspiration (mm/d, daily, 7 years) for the Murray-Darling Basin (1M Km2 ) Japan is 0.37M Km2 Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 12 / 29
  13. 13. Evapotranspiration @ country level Actual evapotranspiration (365 days integrated) for water resources monitoring & management. Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 13 / 29
  14. 14. ET models Benchmarking Average ET for Sri Lanka, Daily 2000-2012 (mm/d, daily, 12.3 years) Comparison ETo & ETpot (rad) are similar, expected. ETa models are not so similar, expected. ETo & ETpot (rad) are higher than ETa models, expected. Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 14 / 29
  15. 15. Outline 1 2 Introduction Evapotranspiration Modeling ET & Equity in Irrigation Crop Water Consumption River Basins Monitoring Country Monitoring ET Models Benchmarking Yann Chemin (IWMI) 3 4 5 Frameworks Chain processing Blueprint GDAL GRASS GIS Future Conclusions HPC benchmarking ET models OSGEO November 2nd, 2013 15 / 29
  16. 16. Chain processing Chain processing has a fundamental impact on remote sensing work: Standardization limits bugs Less prone to human error Simpler parameterization access Permits to apply any number of modules to all target images Ensures maximum quality of generated images Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 16 / 29
  17. 17. Blueprint GDAL+[C+OpenMP] GRASSGIS+pyGRASS+[C+OpenMP] Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 17 / 29
  18. 18. GDAL framework GDAL+[C+OpenMP] Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 18 / 29
  19. 19. GDAL framework OpenMP Distribution Yann Chemin (IWMI) HPC benchmarking ET models OSGEO Cores November 2nd, 2013 19 / 29
  20. 20. GRASS GIS framework metaModule concept pyGRASS: vertical integration of GRASS GIS modules GRASS GIS modules: [C+OpenMP] Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 20 / 29
  21. 21. pyGRASS Summary for Landsat 7 pyGRASS MetaModule http://grasswiki.osgeo.org/wiki/Python/pygrass Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 21 / 29
  22. 22. Outline 1 2 Introduction Evapotranspiration Modeling ET & Equity in Irrigation Crop Water Consumption River Basins Monitoring Country Monitoring ET Models Benchmarking Yann Chemin (IWMI) 3 4 5 Frameworks Chain processing Blueprint GDAL GRASS GIS Future Conclusions HPC benchmarking ET models OSGEO November 2nd, 2013 22 / 29
  23. 23. Future: 128-cores from Tile-GX 64-core Tile-GX Yann Chemin (IWMI) Dual Tile-GX on 1/2 rack board HPC benchmarking ET models OSGEO November 2nd, 2013 23 / 29
  24. 24. Future: 120-cores from Xeon Phi 60-core Phi Yann Chemin (IWMI) Dual Phi in Gygabyte cage HPC benchmarking ET models OSGEO November 2nd, 2013 24 / 29
  25. 25. Outlooks Multi-Cores Hardware (Tile-GX, Xeon Phi) Multi-GPU distribution (OpenCL, CUDA) Multi-CPU distribution (MPI) MODIS and Landsat archives under close pipe distance? Online: automatic, PyWPS, SOS/network ? Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 25 / 29
  26. 26. Outline 1 2 Introduction Evapotranspiration Modeling ET & Equity in Irrigation Crop Water Consumption River Basins Monitoring Country Monitoring ET Models Benchmarking Yann Chemin (IWMI) 3 4 5 Frameworks Chain processing Blueprint GDAL GRASS GIS Future Conclusions HPC benchmarking ET models OSGEO November 2nd, 2013 26 / 29
  27. 27. Conclusions Distributed ET models benchmarking setup with OSGEO tools GDAL: C+OpenMP GDAL: Core-based scaling GRASS GIS: pyGRASS for metaModule GRASS GIS: pyGRASS finish =False GRASS GIS: C+OpenMP inside modules Targets: MODIS (Terra/Aqua), Landsat (all), Aster Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 27 / 29
  28. 28. Collaboration call We are looking for collaborators: Having HPC capabilities Interested in Global Water ET monitoring from Space Using distributed FOSS4G tools Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 28 / 29
  29. 29. Thank You Yann Chemin (IWMI) HPC benchmarking ET models OSGEO November 2nd, 2013 29 / 29

×