Adam Lewis–SPEDDEXES 2014

367 views

Published on

Unlocking Australia’s Landsat Archive: Lessons learned from Geoscience Australia’s journey towards transforming the way we use satellite data

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
367
On SlideShare
0
From Embeds
0
Number of Embeds
43
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Adam Lewis–SPEDDEXES 2014

  1. 1. Unlocking the Landsat Archive, the Australian Geoscience Data Cube & etc Adam Lewis, Geoscience Australia
  2. 2. Outline • Steps toward unlocking GA’s Landsat Archive • Putting data to use : the Australian Geoscience Data Cube (AGDC) • Where the data cube is heading - some future directions • Discrete Global Gridding System Business Systems Development NEO Team Brief
  3. 3. National Flood Risk Information Portal Value layer Delivery, storage and analysis layer Data acquisition and preparation layer Emergency managementWater Private sector Carbon accounting Climate and weather APS 200 and FOI reform Emergency management tools Water toolsCarbon accounting tools Climate and weather tools National framework datasets: Authoritative Base Image and Landcover of Australia Geometric correction Image generation Observation corrections Analysis of biophysical dynamics (Green/brown/water/s oil fraction and indices) Generation of landcover map Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels) -Commercial data (DMCii, SPOT, WV2, Geoeye, aerial) Scene storage Projected grid storage Un-projected grid storage Virtual compute Cloud compute Web services SecurityWMS WFS WCS WCPS Getting useful information out of data
  4. 4. Traditional remote sensing product process is too slow GA Wednesday Seminar 30/10/13 - Datacube Petabyte heirarchical archive: Millions of individual scenes Tape store accessed by robot. Orthorectification calibration, cloud Masking, atmospheric correction, mosaicing Client requests product Identify footprint of product in space or time Search catalog order scenes Product packaging and delivery Feature extraction, algorithm application spectral unmixing
  5. 5. National Flood Risk Information Portal Value layer Delivery, storage and analysis layer Data acquisition and preparation layer Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels) -Commercial data (DMCii, SPOT, WV2, Geoeye, aerial) Historical focus – collecting data Historical strength -1979 Australian Landsat Station - circa 600,000 Landsat scenes - unique archive over Australia (now largely repatriated to the USGS) - raw data, on tape, ~200 tB John MacDonald & Warren Serone, 2012
  6. 6. National Flood Risk Information Portal Data acquisition and preparation layer Image generation Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels) -Commercial data (DMCii, SPOT, WV2, Geoeye, aerial) Automate the production of images Automated systems ‘Process Management Application’ Baseline: 50 scenes per day, manual (3 fte) Target: 1000 scenes per day Actual: 2000+ scenes per day Driver: International Forest Carbon Initiative
  7. 7. National Flood Risk Information Portal Data acquisition and preparation layer Image generation Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels) -Commercial data (DMCii, SPOT, WV2, Geoeye, aerial) Deal with the storage problem Automated systems ‘Process Management Application’ Baseline: 50 scenes per day, manual (3 fte) Target: 1000 scenes per day Actual: 2000+ scenes per day Driver: International Forest Carbon Initiative Storing processed images - Earth Observation Data Store - Baseline: a few hundred scenes - Jan 2012: ~500,000 records - Dec 2012: >2,000,000 records Driver: International Forest Carbon Initiative
  8. 8. National Flood Risk Information Portal Data acquisition and preparation layer Image generation Geometric correction Observation corrections Analysis of biophysical dynamics (Green/brown/water/s oil fraction and indices) Generation of landcover map Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels) -Commercial data (DMCii, SPOT, WV2, Geoeye, aerial) Calibration to produce a measurement in x,y,z,t Calibration – Surface reflectance Baseline: empirical methods Target: physics-based method Actual: community acceptance, published method Driver: Data stewardship Correlation betw een Landsat 5 and 7 in Band 1 y = 0.9796x R2 = 0.986 0 1000 2000 3000 4000 5000 6000 7000 0 1000 2000 3000 4000 5000 6000 7000 Landsat 7 (Surface reflectance x 10000) Landsat5(Surfacereflectancex10000) Series1 Linear (Series1)
  9. 9. National Flood Risk Information Portal Data acquisition and preparation layer Image generation Geometric correction Observation corrections Analysis of biophysical dynamics (Green/brown/water/s oil fraction and indices) Generation of landcover map Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels) -Commercial data (DMCii, SPOT, WV2, Geoeye, aerial) Quality assessment / filters Calibration – Pixel quality assessment Baseline: accepted methods exist Target: implement an accepted method Actual: achieved Driver: Australian Space Research Program, Unlocking the Landsat Archive
  10. 10. National Flood Risk Information Portal
  11. 11. National Flood Risk Information Portal Data acquisition and preparation layer Image generation Geometric correction Observation corrections Analysis of biophysical dynamics (Green/brown/water/s oil fraction and indices) Generation of landcover map Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels) -Commercial data (DMCii, SPOT, WV2, Geoeye, aerial) Algorithms to estimate biophysical parameters Extracting ‘biophysical primaries’, i.e,. water (in this case) Baseline: no accepted methods Target: published, accepted, available, rapid, automatic method Current state: accepted, automated, un- published, involves commercial software. Driver: Emergency management
  12. 12. Density of quality-assured observations (15 years) Lewis and Thankappan, AOMSUC October 2013
  13. 13. How do you work with this??? Lewis and Thankappan, AOMSUC October 2013
  14. 14. GA Wednesday Seminar 30/10/13 - Datacube Value layer Delivery, storage and analysis layer Data acquisition and preparation layer Emergency managementWater Private sector Carbon accounting Climate and weather APS 200 and FOI reform Emergency management tools Water toolsCarbon accounting tools Climate and weather tools National framework datasets: Authoritative Base Image and Landcover of Australia Geometric correction Image generation Observation corrections Analysis of biophysical dynamics (Green/brown/water/s oil fraction and indices) Generation of landcover map Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels) -Commercial data (DMCii, SPOT, WV2, Geoeye, aerial) Scene storage Projected grid storage Un-projected grid storage Virtual compute Cloud compute Web services SecurityWMS WFS WCS WCPS Capture, analysis and application of Earth obsvns
  15. 15. Data organised for HPC - time series observations GA Wednesday Seminar 30/10/13 - Datacube
  16. 16. Data organised for HPC - time series observations Lewis and Thankappan, AOMSUC October 2013 Tile Count (Currently approx. 4M tiles) Landsat Scene Count (Currently approx. 650k scenes)
  17. 17. GA Wednesday Seminar 30/10/13 - Datacube Value layer Delivery, storage and analysis layer Data acquisition and preparation layer Emergency managementWater Private sector Carbon accounting Climate and weather APS 200 and FOI reform Emergency management tools Water toolsCarbon accounting tools Climate and weather tools National framework datasets: Authoritative Base Image and Landcover of Australia Geometric correction Image generation Observation corrections Analysis of biophysical dynamics (Green/brown/water/s oil fraction and indices) Generation of landcover map Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels) -Commercial data (DMCii, SPOT, WV2, Geoeye, aerial) Scene storage Projected grid storage Un-projected grid storage Virtual compute Cloud compute Web services SecurityWMS WFS WCS WCPS Getting useful information out of data
  18. 18. GA Wednesday Seminar 30/10/13 - Datacube Four-month non-interpolated median NDVI for entire Murray Darling Basin • Initial Datacube test area • 2,112,000,000 pixels (i.e. 2.1 Billion). • Every observation can be traced back to its source capture image through provenance information layers
  19. 19. Normalised 15-year surface water count (25m) Lewis and Thankappan, AOMSUC October 2013
  20. 20. Normalised 15-year surface water count (25m) Datacube Overview - 13/09/2013 Area NE of Lake Eyre showing channel bathymetry and porous dunes
  21. 21. Normalised 15-year surface water count (25m) Lewis and Thankappan, AOMSUC October 2013 Area NE of Lake Eyre showing channel bathymetry and porous dunes
  22. 22. Normalised 15-year surface water count (25m) Lewis and Thankappan, AOMSUC October 2013 Area NE of Lake Eyre showing channel bathymetry and porous dunes
  23. 23. Normalised 15-year surface water count (25m) Datacube Overview - 13/09/2013 Area NE of Lake Eyre showing channel bathymetry and porous dunes
  24. 24. Normalised 15-year surface water count (25m) Datacube Overview - 13/09/2013 Area NE of Lake Eyre showing channel bathymetry and porous dunes
  25. 25. Some next steps for the data cube Steering committee of key stakeholders– GA, NCI, CSIRO Data reside on the RDSI – NCI node Exploring the relationship between data and models More data • Geology - radiometrics; gravity; ASTER mineral maps • Topographics – elevation, slope, topographic elements • Climate surfaces • Additional EO datasats: Landsat-8, MODIS, Landsat-MSS, other, future satellites – Sentinel-2; himawari-8/9 • Derived measurements: Fractional cover, Surface Water, Burnt areas, etc (using nationally accepted algorithms developed through collaborative efforts) Business Systems Development NEO Team Brief
  26. 26. Discrete Global Grid Systems A common global grid architecture would allow us to: • Organise measurements over the globe • Calculate gradients faithfully • Compare time-series of globally distributed data • Make statistically meaningful regional comparisons of global data • Compare and combine data from multiple measurements taken at different resolutions • Improve operation of numerical models • Document the precision as well as location of spatial data on the globe Towards a Global Discrete Nested Grid
  27. 27. Kimerling/Goodchild Criteria for gridding systems Towards a Global Discrete Nested Grid Criterion Criteria in Kimerling et al. (1999) (Goodchild's Numbers given in parentheses) Criteria in Goodchild (1994) 1.Domain is globe Areal cells constitute a complete tiling of the globe, exhaustively covering the globe without overlapping. (3,7) 1. Each area contains one point 2. Equal area Areal cells have equal areas. This minimizes the confounding effects of area variation in analysis, and provides equal probabilities for sampling designs. (2) 2. Areas are equal in size 3. Same topology Areal cells have the same topology (same number of edges and vertices). (9, 14) 3. Areas exhaustively cover the domain 4. Equal shape Areal cells have the same shape. ideally a regular spherical polygon with edges that are great circles. (4) 4. Areas are equal in shape 5. Compactness Areal cells are compact. (10) 5. Points form a hierarchy preserving some (undefined) property for m < n points 6. Straight Edges on Projection Edges of cells are straight in a projection. (8) 6. Areas form a hierarchy preserving some (undefined) property for m < n areas 7. Perimeter Bisection The midpoint of an arc connecting two adjacent cells coincides with the midpoint of the edge between the two cells. 7. The domain is the globe (sphere, spheroid)
  28. 28. Kimerling/Goodchild Criteria for gridding systems Towards a Global Discrete Nested Grid 8. Hierarchy The points and areal cells of the various resolution grids which constitute the grid system form a hierarchy which displays a high degree of regularity. (5,6) 8. Edges of areas are straight on some projection 9. Single point A single areal cell contains only one grid reference point.(1) 9. Areas have the same number of edges 10. Maximally centred Grid reference points are maximally central within areal cells. (11) 10. Areas are compact 11. Equidistant Grid reference points are equidistant from their neighbors. (12) 11. Points are maximally central within areas 12. Addressing Grid reference points and areal cells display regularities and other properties which allow them to be addressed in an efficient manner. 12. Points are equidistant 13. Latitude Longitude The grid system has a simple relationship to latitude and longitude. 13. Edges are areas of equal length 14. Arbitrary resolution The grid system contains grids of any arbitrary defined spatial resolution. (5,6) 14. Addresses of points and areas are regular and reflect other properties
  29. 29. Progressing global gridding systems rHEALPix from Landcare Research New Zealand OGC Special Working Group to be proposed in May Towards a Global Discrete Nested Grid
  30. 30. What about the modelling layer? Business Systems Development NEO Team Brief
  31. 31. Opening up new possibilities • Establishing the ability to do things that we don’t yet know about / are not yet possible “For example, over 40% of the revenue from IBM last year came from products and services that were impossible to do just two years ago.” GA Wednesday Seminar 30/10/13 - Datacube

×