DSM training - preparing auxiliary data

Auxiliary maps for Digital Soil Mapping
FOSS software (R+SAGA+FWTools)
T. Hengl
Internationaal Bodemreferentie en Informatie Centrum - ISRIC
Wageningen UR
Digital Soil Mapping training, Sept 3, 2010, JRC Ispra

Programme
9:00–10:15 Importance of auxiliary maps for DSM;
10:30–12:00 Overview of publicly available global data sets;
12:00–14:00 lunch break
14:00–15:15 Data exploration (exporting maps to KML)
Software installation and customization.
Opening and processing worldmaps using R+SAGA/FWTools.
Exercises.
15:30–17:00 Preparing auxiliary maps for continental-scale
DSM project;
Optional: preparation of worldmaps and metadata for your
own case study;

Outline
Introduction
Literature
DSM and env maps
Worldmaps
Software
R code editors
Working with spatial data
R+SAGA
Exercises
R+FWTools
Export to Google Earth

Literature
Bivand, R., Pebesma, E., Rubio, V., 2008. Applied Spatial
Data Analysis with R. Use R Series, Springer, Heidelberg, 378
p.
Conrad, O., 2007. SAGA — Entwurf, Funktionsumfang und
Anwendung eines Systems f¨ur Automatisierte
Geowissenschaftliche Analysen. PhD thesis, University of
G¨ottingen.
Hengl, T., 2009. A Practical Guide to Geostatistical Mapping,
2nd edition. University of Amsterdam, 291 p. ISBN
978-90-9024981-0.
McBratney, A. B., Santos, M. L. M., Minasny, B., 2003. On
digital soil mapping. Geoderma, 117(1–2): 3–52.

The ASDA(R)-book.org

The ASDA(R) team

A Practical Guide to Geostatistical Mapping

DSM: the main steps (analysis)
1. Import and organize soil lab data (site/horizon).
2. Filter gross errors (outliers), harmonize data coming from
various sources.
3. Fit splines to parameterize soil-depth relationships.
4. Prepare auxiliary maps (grid deﬁnition, projection system,
data format, . . .).
5. Generate spatial predictions (predictive mapping) and assess
accuracy.
6. Prepare and distribute ﬁnal maps (with metadata) and reports.

Objectives
Where to obtain auxiliary environmental maps for DSM?

Objectives
Where to obtain and how to use the R+OSGeo software?

Objectives
How to combine GIS and DSM operations?

Objectives
How to combine GIS and DSM operations?
How to visualize maps using Google Earth/Maps?

Deﬁnitions
Digital Soil Mapping: Analytical production of soil property
and soil type maps using quantitative techniques, ground
records (soil proﬁles), auxiliary maps (covariates) and expert
knowledge.

Deﬁnitions
knowledge.
The main input to DSM is a map; the main output is a map;

Deﬁnitions
knowledge.
There are three main types of auxiliary maps:

Deﬁnitions
knowledge.
1. DEM-derivatives (morphometric, hydrological, climatic, . . .)

Deﬁnitions
knowledge.
2. Remote sensing images (thermal images, radar images, gamma
radiometrics, vegetation indices, . . .)

Deﬁnitions
knowledge.
2. Remote sensing images (thermal images, radar images, gamma
radiometrics, vegetation indices, . . .)
3. Thematic maps (geological strata, geomorphological units, . . .)

Auxiliary maps
Auxiliary maps (environmental predictors or covariates) are
equally important input to DSM as the point data.

Auxiliary maps
If we are lucky, we will be able to explain distribution of soil
properties by using a small sample of species
occurrence/attributes.

Auxiliary maps
Environmental data can come in diﬀerent resolutions (support
size), accuracy, coverage, formats etc.

Auxiliary maps
The publicly available global data sets are in general
under-used.

Auxiliary maps
The publicly available global data sets are in general
under-used.
. . . for various reasons (unpopular formats, missing metadata,
signiﬁcant processing required to put it to a usable format
etc.).

Universal model of spatial variation
In statistical terms, spatial variability of some soil variable Z
consists of three components:
Z(s) = Z∗
(s) + ε (s) + ε (1)
where Z∗(s) is the deterministic component, ε (s) is the spatially
correlated random component and ε is the pure noise — partially
micro-scale variation, partially the measurement error. This model
is, in the literature, often referred to as the universal model of
variation

SCORPAN
A well accepted formulation of the soil–environment correlation
model is the so-called SCORPAN model McBratney at al. (2003),
which is based on the general soil–vegetation formulation:
V × S[x, y,˜t] = f
s[x, y,˜t] c[x, y,˜t] o[x, y,˜t]
r[x, y,˜t] p[x, y,˜t] a[x, y,˜t]
(2)
where V stands for vegetation, S for soil, c stands for climate, o
for organisms (including humans), r is relief, p is parent material
or geology, a is age of the system, x, y are the coordinates and t is
time dimension.

Global repository of publicly available maps

worldmaps
I’ve been collecting and sorting/processing various publicly
available data sets over the years (at 1–5 km resolution, there
is a lot of free data!).
The result is a repository with cca 100 unique rasters, that
can be obtained directly from
http://spatial-analyst.net/worldmaps/.
Each gridded map consists of 7200 columns and 3600 rows;
the cell size is 0.05 arcdegrees, which corresponds to about
5.6 km; all maps fall on the same grid.
Maps are projected in the Latitude-Longitude WGS84 system
+proj=longlat +ellps=WGS84.
These maps are ideal for various mapping applications at
continental and national levels.

The happy triangle guy
GIS analysis
Storage and
browsing of
geo-data
Statistical
computing
KML
GDAL
ground
overlays,
time-series

Software we will use
R v2.11 (Windows OS) including a list of packages;
Tinn-R v2.3 (code editor);
Optional: FWTools v2.4.7 — a list of utilities to handle
spatial data; SAGA GIS v2.0.4 — a light GIS excellent for
educational purposes.

Installing the add-on packages
> install.packages("ctv")
> library(ctv)
> install.views("Spatial")
This will install all connected packages listed at R views Spatial.

Check your installation
> Sys.getenv(c("OS", "COMPUTERNAME", "R_HOME", "R_LIBS_USER",
+ "PROCESSOR_IDENTIFIER"))
OS
"Windows_NT"
COMPUTERNAME
"PC-IBED193"
R_HOME
"C:PROGRA~1RR-210~1.1"
R_LIBS_USER
"n:/R/win-library/2.10"
PROCESSOR_IDENTIFIER
"x86 Family 6 Model 15 Stepping 6, GenuineIntel"

Tinn-R

Customizing link between R and Tinn-R:
1. Open Tinn-R select“Options”→“R”→ Path and check that
the path is correct;
2. Tinn-R requires R to run either Rterm or Rgui in SDI mode:
In Rgui, select“Edit”→“GUI preferences”and set SDI and
click on Save.
3. To add the CRTL+R shortcut to Tinn-R:
First open“Options”→“Shortcuts”and replace the existing
CRTL+R shortcut to e.g.CRTL+M;
Then, open“R”menu and select“Hotkeys”; add a CRTL+R
shortcut to“Send line”;
To send blocks of text you will need to edit your Rprofile.site
file under“R/etc/”directory;
4. To learn more about how to customize Tinn-R, read the
user_guide.html file under the
“Tinn-R/doc/English/user guide/”directory.

Spatial objects
An advantage of R (as compared to e.g.Matlab) is that you can
create your own formats and structures for data. But if there are
too many formats you can easily get lots. In addition, we want to
have smooth links to external formats (R is open!).
To reduce this problem, Bivand et al.(2008) developed new-style
classes to represent spatial data.

Spatial class
The foundation object is the Spatial class, with just two basic slots
(new-style S classes have pre-deﬁned components called slots):
a bounding box — mostly used for setting up plots;
a CRS class object — deﬁning the coordinate reference
system, and may be set to CRS(as.character(NA));
Operations on Spatial* objects should update or copy these
values to the new Spatial* objects being created. The most basic
spatial data object is a point, which may have 2 or 3 dimensions.

Spatial classes
for point features: SpatialPoints;
SpatialPointsDataFrame;
for line features: SpatialLines, SpatialLinesDataFrame;
polygons: SpatialPolygons, SpatialPolygonsDataFrame;
rasters: SpatialPixels, SpatialPixelsDataFrame,
SpatialGrid, SpatialGridDataFrame;

SpatialPoints
> library(sp)
> data(meuse)
> coords <- SpatialPoints(meuse[, c("x", "y")])
> summary(coords)
Object of class SpatialPoints
Coordinates:
min max
x 178605 181390
y 329714 333611
Is projected: NA
proj4string : [NA]
Number of points: 155

SpatialPointsDataFrame
We can add the tabular data to make a
SpatialPointsDataFrame object:
> meuse1 <- SpatialPointsDataFrame(coords, meuse)
> str(meuse1, max.level = 2)
Formal class 'SpatialPointsDataFrame' [package "sp"] with 5 slots
..@ data :'data.frame': 155 obs. of 14 variables:
..@ coords.nrs : num(0)
..@ coords : num [1:155, 1:2] 181072 181025 181165 ...
.. ..- attr(*, "dimnames")=List of 2
..@ bbox : num [1:2, 1:2] 178605 329714 181390 333611
.. ..- attr(*, "dimnames")=List of 2
..@ proj4string:Formal class 'CRS' [package "sp"] with 1 slots

Basic methods
spplot — plotting of spatial objects (maps);
spsample — sample points from a set of polygons, on a set
of lines or from a gridded area;
bbox — get the bounding box;
proj4string — get or set the projection (coordinate
reference system);
coordinates — set or retrieve coordinates;
spTransform — transform coordinates from one CRS to
another (PROJ.4);
overlay — combine two diﬀerent spatial objects;

Plotting a SpatialPoints object
> plot(as(meuse1, "Spatial"), axes = TRUE)
> plot(meuse1, add = TRUE)
> plot(meuse1[meuse1$ffreq == 1, ], col = "green", add = TRUE)
178000 179000 180000 181000 182000
330000331000332000333000

Combining statistical and GIS operations
Because the Spatial*DataFrame family objects behave in
most cases like data frames, most of what we are used to
doing with standard data frames just works (but no merge,
etc., yet).
These objects are very similar to typical representations of the
same kinds of objects in geographical information systems, so
they do not suit spatial data that is not geographical (like
medical imaging) as such.
Because now sp classes for GIS data exits, this opens the door
for fusing GIS and statistical operations (this has not been
possible in e.g.2002).

Spatial packages
R now oﬀers a range of contributed packages in spatial statistics
and increasing awareness of the importance of spatial data analysis
in the broader community. Current contributed packages with
spatial applications:
point patterns: spatstat, VR:spatial, splancs;
geostatistics: gstat, geoR, geoRglm, ﬁelds, spBayes,
RandomFields, VR:spatial, sgeostat, vardiag;
lattice/area data: spdep, DCluster, spgwr, ade4;
links to GIS: rgdal, spgrass, RPy, RSAGA;

Let’s create spatial objects!
We can create spatial objects from scratch! For example a DEM:
> dem <- expand.grid(x = seq(100, 600, 100), y = seq(100,
+ 600, 100))
> dem$Z <- as.vector(c(23, 24, 34, 38, 45, 51, 24, 20,
+ 20, 28, 18, 49, 22, 20, 19, 14, 38, 45, 19, 15, 13,
+ 21, 23, 25, 14, 11, 18, 11, 18, 19, 10, 16, 23, 16,
+ 9, 6))
> gridded(dem) <- ~x + y
> dem <- as(dem, "SpatialGridDataFrame")
> str(dem)

Controlling SAGA from R
> library(RSAGA)
> rsaga.env()
$workspace
[1] "."
$cmd
[1] "saga_cmd.exe"
$path
[1] "C:/PROGRA~1/R/R-210~1.1/library/RSAGA/saga_vc"
$modules
[1] "C:/PROGRA~1/R/R-210~1.1/library/RSAGA/saga_vc/modules"

Getting list of modules
> rsaga.get.modules("ta_channels")
$ta_channels
code name interactive
1 0 Channel Network FALSE
2 1 Watershed Basins FALSE
3 2 Watershed Basins (extended) FALSE
4 3 Vertical Distance to Channel Network FALSE
5 4 Overland Flow Distance to Channel Network FALSE
6 5 D8 Flow Analysis FALSE
7 6 Strahler Order FALSE

Run stream extraction
> rsaga.geoprocessor(lib = "ta_channels", module = 5,
+ param = list(DEM = "dem6.sgrd",
+ DIRECTION = "channels.sgrd", CONNECTION = "route.sgrd",
+ NETWORK = "channels.shp"))
SAGA CMD 2.0.4
library path: C:/PROGRA~1/R/R-210~1.1/library/RSAGA/...
library name: ta_channels
module name : D8 Flow Analysis
author : (c) 2003 by O.Conrad
Load grid: dem6.sgrd...
ready
Parameters
Grid system: 100; 6x 6y; 100x 100y
DEM: dem6
Flow Direction: Flow Direction
Flow Connectivity: Flow Connectivity
Flow Network: Flow Network
Minimum Connectivity: 0
...

Read back to R
> dem$route <- readGDAL("route.sdat")$band1
route.sdat has GDAL driver SAGA
and has 6 rows and 6 columns
> channels <- readOGR("channels.shp", "channels")
OGR data source with driver: ESRI Shapefile
Source: "channels.shp", layer: "channels"
with 32 features and 2 fields
Feature type: wkbLineString with 2 dimensions

Plot the ﬁnal result
> dem.plt <- spplot(dem[1], main = "DEM", col.regions = topo.colors(25))
> channels.plt <- spplot(dem[2], col.regions = rev(gray(0:20/20)),
+ main = "Flow connectivity", sp.layout = list("sp.lines",
+ channels, col = "red"))
> print(dem.plt, split = c(1, 1, 2, 1), more = T)
> print(channels.plt, split = c(2, 1, 2, 1), more = F)
DEM
10
20
30
40
50
Flow connectivity
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0

Exercise 1
Create a working directory called NL, then download the
makeRDC.R script.

Exercise 1
makeRDC.R script.
Open new session in R and run the script from Tinn-R.

Exercise 1
makeRDC.R script.
Open new session in R and run the script from Tinn-R.
This will download a land cover map of NL and resample it to
the Dutch coordinate system.

FWTools
Created and maintained by Frank Warmerdam (director and
active contributor to OSGeo).
includes OpenEV, GDAL, MapServer, PROJ.4 and OGDI.
“it is intended to give folks a chance to use the latest and
greatest”.
FWTools contain a number of (highly eﬃcient!!) utilities
excellent for processing large datasets.

Preparing FWTools
There is still no package to control FWTools from R, but we can
simply send command lines using the system command.
Before we can use FWTools from R, we need to locate it on our
PC:
> gdalwarp <- gsub("/", "", dir(path="C:/PROGRA~2/FWTOOL~1.7",
+ pattern="gdalwarp.exe", recursive=TRUE, full.names=TRUE))
> gdalwarp
[1] "C:PROGRA~2FWTOOL~1.7bingdalwarp.exe"
> workd <- paste(gsub("/", "", getwd()), "", sep="")

MODIS data
Now we can download some GIS data from web:
> MOD12Q1 <- "ftp://anonymous:test@e4ftl01u.ecs.nasa.gov/
+ MOLT/MOD12Q1.004/2004.01.01/"
> download.file(paste(MOD12Q1,
+ "MOD12Q1.A2004001.h18v03.004.2006117173748.hdf", sep=""),
+ destfile=paste(getwd(),
+ "MOD12Q1.A2004001.h18v03.004.2006117173748.hdf", sep="/"),
+ mode='wb', method='wget')
Resolving e4ftl01u.ecs.nasa.gov... 152.61.4.83
Connecting to e4ftl01u.ecs.nasa.gov|152.61.4.83|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /MOLT/MOD12Q1.004/2004.01.01 ... done.
==> SIZE MOD12Q1.A2004001.h18v03.004.2006117173748.hdf ... 23165983
==> PASV ... done. ==> RETR MOD12Q1.A2004... done.
Length: 23165983 (22M)
0K .......... .......... 0% 64.9K 5m48s
...
22550K .......... .......... 99% 501K 0s
22600K .......... 100% 503K=65s

Reprojecting grids
We can reproject/resample the map to our local coordinate system
using the gdalwarp functionality (this combines several processing
steps in one function):
> NL.prj <- "+proj=sterea +lat_0=52.15616055555555
+ +lon_0=5.38763888888889 +k=0.999908 +x_0=155000
+ +y_0=463000 +ellps=bessel +units=m +no_defs
+ +towgs84=565.237,50.0087,465.658,
+ -0.406857,0.350733,-1.87035,4.0812"
> system(paste(gdalwarp, " HDF4_EOS:EOS_GRID:"", workd,
+ "MOD12Q1.A2004001.h18v03.004.2006117173748.hdf"
+ :MOD12Q1:Land_Cover_Type_1 -t_srs "", NL.prj, ""
+ IGBP2004NL.tif -r near -te 0 300000 280000 625000
+ -tr 500 500", sep=""))
Creating output file that is 560P x 650L.
Processing input file HDF4_EOS:EOS_GRID:MOD12Q1.A2004001...
Using internal nodata values (eg. 255) for image HDF4_EOS:EOS_...
0...10...20...30...40...50...60...70...80...90...100 - done.

Plot the ﬁnal result
In this case we have produced a MODIS-based land cover map for
the whole Netherlands in resolution of 500 m (in local coordinate
system).

Exercise 2 & 3
Create a working directory called worldmaps, then download
a selection of worldmaps from the repository (or get a copy
from the USB stick) and unzip the maps that you need to
complete the exercises.

Exercise 2 & 3
Use any GIS that you ﬁnd suitable to answer the questions.

Exercise 2 & 3
Use any GIS that you ﬁnd suitable to answer the questions.
I recommend ﬁrst testing SAGA GIS, then running the
analysis in R. These maps are Large so it could take time until
you import/open a map.

Exercise 4
We focus on extracting land surface parameters (or
geomorphometric parameters) using a DEM.
First create a new working directory on your computer, then
Download the map of world countries (countries) and the
global Digital Elevation Model at 5.6 km resolution
(globedem).
Resample and subset the DEMs to local coordinate systems
— for Germany use the European ETRS89 coordinate system
(EPSG:3035), and for Bolivia use the South America Albers
Equal Area Conic coordinate system (ESRI:102033).
Try also to derive these parameters using the RSAGA
package, i.e.by sending the commands to SAGA from R.

Writing spatial data to KML
There are two possibilities to export maps to KML: (a) using
existing packages, and (b) by writing KML files“by-hand”.
To export point or line features to KML, use the writeOGR
method that is available in R package rgdal.
More flexible way to writing KML files is by using loops.

DSM training - preparing auxiliary data

More Related Content

What's hot

Viewers also liked

Similar to DSM training - preparing auxiliary data

More from Tomislav Hengl

Recently uploaded

DSM training - preparing auxiliary data