GEOGRAPHIC SCRIPTING IN GVSIG HALFWAY BETWEEN USER AND DEVELOPER held at the University of Potsdam in 2016 - PART 3: THE SCRIPTING COMPOSER AND PYTHON

1. GEOGRAPHIC SCRIPTING IN GVSIG
HALFWAY BETWEEN USER AND DEVELOPER
Geoinformation Research Group,
Department of Geography
University of Potsdam
21-25 November 2016
Andrea Antonello
PART 3: THE SCRIPTING COMPOSER AND PYTHON

2. THE SCRIPTING COMPOSER
Scripting is one of the powerful things in GIS.
Non developers have the possibility to create great automatisms through
it.
gvSIG's scripting composer is based on the
(through jython).
python scripting language

3. THE SCRIPTING COMPOSER
To open the scripting composer access the Tools -> Scripting menu:

4. THE SCRIPTING COMPOSER
Once opened, it shows up like:
new script

5. THE SCRIPTING COMPOSER
Once opened, it shows up like:
new
script
save
script
cut
copy
paste
local
scripts
library
editor
output console
and error view
run or save
and run
syntax check*
*if Composer Tools installed

6. THE SCRIPTING COMPOSER
Running and checking errors
run
output
error
console
If you run the script by hitting the run button, the editor will not save the
script for you (CTRL-S). It does if you instead use the save+run button.

7. THE BASE LANGUAGE: PYTHON!
In order to be able exploit 100% the scripting engine, it is mandatory to
get a good insight of the base language that can be used in the editor.
Before we start with the geo part, we will investigate a bit python.
Let's create a new script to do so. Let's call it python_essentials.py and let
it print out a simple message to make sure it works.
def main(*args):
print "Python essentials"
print "*****************"

8. PYTHON: FORMATTING
In python formatting is mandatory. It is the formatting that defines the
logical blocks.
For example this will not work:
def main(*args):
print "Python essentials"
print "*****************"
and produce an error like:
IndentationError: unindent does not match any outer indentation level in
<script> at line number 4 at column number 1 python_essentials 4:1

9. PYTHON: WRITING STRINGS
We have seen writing of strings already:
# write a string
print "This is a string"
lines that begin with a hash at the begin are ignored -> comments
Comments can be placed without formatting.
Also more complex strings can be used, by surrounding text with triple
quotes:
# write a complex string
print """Inside triple quotes
you can
do almost anything.
It is kind of an advanced string!"""
And also kind of defeat the laws of formatting.

10. PYTHON: VARIABLES
Variables definition is very simple. Just write a name and assign a value:
# define variables
# a string
name = "Luis"
# a number
age = 25
# insert the variables into a string template
# through the use of %s, %f, %i
myTemplate1 = "Hi, my name is %s and I'm %i." % (name, age)
print myTemplate1
# strings and numbers can also be concatenated through the + sign
myTemplate2 = "Hi, my name is " + name + " and I'm " + str(age) + "."
print myTemplate2
Note that string concatenation with the + (plus) sign works only with
string. Hence, the age variable has first to be converted to a string
through the str(...) function.

11. PYTHON: WORKING WITH STRING
Python allows for a whole pile of operations on string. Let's focus on
those we will use later:
# strip whitespaces
string = " a line of text with spaces "
print string.lstrip()
print string.rstrip()
print string.strip()
# split a string
string = "a, comma, separated, text"
print string.split(",")
# starts/ends with
string = "a b c"
print string.startswith("a")
print string.endswith("a")
# to upper/lowercase
string = "A b C"
print string.upper()
print string.lower()
# substrings
string = "123456"
print string[2:]
print string[:-2]
print string[2:-2]
# find in strings (0 indexed)
print string.find("2")
print string.find("22")
print string.find("345")
# find after certain index
print string.find("345", index)
# replace
s = "Hola world"
print s.replace("world", "mundo")

12. PYTHON: FORMATTING NUMBERS
When printing out numbers, they can be formatted as follows
print "Formatting numbers"
oneThird = 1.0/3.0
print "%f" % oneThird
print "%i" % oneThird
print "%2.2f" % oneThird
print "%5.2f" % oneThird
print "%2.1f" % oneThird
print "%+2.2f" % oneThird
%[width][.precision]type

13. EXTRACT DATA FROM THE WEB
In python it is very simple to get data from the web, all that is necessary is
the import of the urllib and a few lines of code.
For example, let's print out the tiobe index that defines the ranking of the
programming languages:
import urllib
def main(*args):
connection = urllib.urlopen("http://www.tiobe.com/tiobe-index/")
htmlString = connection.read()
print htmlString

14. EXTRACT DATA FROM THE WEB
As an exercise let's extract the table of the ranking of the programming
languages using what we learned until now:
index = htmlString.find('TIOBE Index for November 2016')
# extract the header
startHeader = htmlString.find('<thead', index)
endHeader = htmlString.find('</thead', startHeader)
tableHeader = htmlString[startHeader:endHeader]
tableHeader = tableHeader.replace("<thead>","").replace("</thead>","")
.replace("<th>","").replace("</th>","").replace("<tr>","")
.replace("</tr>","").replace("n"," | ").strip()
print tableHeader
startData = htmlString.find('<tbody', index)
endData = htmlString.find('</tbody', startData)
tableData = htmlString[startData:endData]
tableData = tableData.replace("n"," | ").replace("<tbody>","").replace("</tbody>","")
.replace("<td>","").replace("</td>","|").replace("<tr>","")
.replace("</tr>","n|").strip()
print tableData

15. PYTHON: TYPE AND DIR
type and dir can be two valuable friends in moments of
desperation:
type: returns the type of the passed object
dir: returns the list of valid attributes and methods
of the object passed
string = "A string"
print type(string)
print dir(string)

16. PYTHON: LISTS
Lists are containers for objects and are defined through square brackets:
# working with lists
# create a list of strings
list = ["Merano", "Bolzano", "Trento"]
print list
# how to access the elements?
print "The elements start at position 0: " + list[0]
# what if you use negative indexes?
print "Element at position -1: " + list[-1]
# how to add an element to the end of the list?
list.append("Potsdam")
print list
# how to add an element as first element?
list.insert(0, "Berlin")
print list
# how to remove an element from the list?
# by object?
list.remove("Potsdam")
print list
# or by index through the del statement
del list[0]
print list
# how many elements are in the list?
print "Elements count: %s" % len(list)

17. PYTHON: LOOPING THROUGH LISTS
# looping through lists
list = ["Merano", "Bolzano", "Trento"]
# to loop over a list a for loop can be used
for value in list:
print value
# if you also need an index
for index, value in enumerate(list):
print "%s is town N.%i" % (value, index)

18. PYTHON: SORTING LISTS
# sorting lists
list = ["Merano", "Bolzano", "Trento"]
print "This is the original list: " + str(list)
# lists can be sorted
list.sort()
print "This is the sorted list: " + str(list)
# and reverse sorted
list.reverse()
print "This is reverse list: " + str(list)
To print lists concatenated with strings we use the str(...) function.

19. PYTHON: FILTERING LISTS
# lists and numbers
list = [12, 5, 7, 1, 2, 3]
print "This is the original list: " + str(list)
# calculate min, max and sum
print "max is %i, min is %i, sum is %i" % (max(list),
min(list), sum(list))
# filter based on conditions
# create the filter
myFilter = lambda x: x > 5.5
# apply it
filteredData = filter(myFilter, list)
print filteredData
The filter applies the lambda expression to each element of the list,
assigning it as x variable of the lambda expression.

20. PYTHON: CONCATENATE LISTS
# concatenate and print lists nicely
abc = ["a", "b", "c"]
cde = ["c", "d", "e"]
newabcde = abc + cde
print newabcde
# through 'join' the list can be
# concatenated to one single string
print ",".join(newabcde)
print " | ".join(newabcde)

21. PYTHON: MODIFY LISTS
# modify each element of the list with
# a lambda and map
abc = ["a", "b", "c"]
print "Original abc: " + str(abc)
# create a function to apply to the elements
lambdaFunction = lambda x: x.upper()
# and apply it through the map statmente
modifiedAbc = map(lambdaFunction, abc)
print "Modified abc: " + str(modifiedAbc)
# what if you want an index in the lambda?
lambdaFunction = lambda (i, x): "%i) %s" % (i, x.upper())
modifiedAbc = map(lambdaFunction, enumerate(abc))
print "Modified abc with index: " + str(modifiedAbc)

22. PYTHON: DICTIONARIES
# working with dictionaries
# create a dictionary from key1:value1, key2:value2, ...
townsProvinceMap = {'merano':'BZ', 'bolzano':'BZ',
'trento':'TN'}
print townsProvinceMap
# ways of accessing the values of a map
# the "array" way
print townsProvinceMap['merano']
# through the "getter"
print townsProvinceMap.get('merano')
# add an element
townsProvinceMap['St. Walburg'] = "BZ"
print townsProvinceMap
# remove an element
del townsProvinceMap['St. Walburg']
print townsProvinceMap

23. PYTHON: LOOPING THROUGH DICTIONARIES
# check if a key is present in the dict
print "has key 'merano'? %s" % townsProvinceMap.has_key("merano")
# get a list of keys, values, items
print "keys: %s" % townsProvinceMap.keys()
print "values: %s" % townsProvinceMap.values()
print "items: %s" % townsProvinceMap.items()
# how do we check the type of a variable
print type(townsProvinceMap.keys())
# loop through a dictionary
for key, value in townsProvinceMap.items():
print key + " = " + value
# or through the list of keys
for key in townsProvinceMap.keys():
print key + " = " + townsProvinceMap[key]

24. PYTHON: LOOP WITH RANGES
# working with ranges
# they are used for iterations
for i in range(0, 3):
print "Iteration: %i" % i
# using xrange
for i in xrange(0, 3):
print "Xrange iteration: %i" % i
# xrange vs. range
print type(xrange(0, 3))
print type(range(0, 3))
# xrange creates a number generator, not a list

25. PYTHON: WORKING WITH THE FILESYSTEM
import os
import getpass
def main(*args):
# get the user home folder
home = os.path.expanduser("~")
# get the user name
user = getpass.getuser()
print "home: " + home
print "user: " + user
# use the file separator
newDir = home + os.sep + "tmp_folder_" + user
print "create folder: " + newDir
# create a folder
os.mkdir(newDir)
print "folder exists: %s" % os.path.exists(newDir)
To work with the filesystem it is necessary to first understand imports,
that are statements used to make modules available. That way any
external library and script can be used from within the code.
Import statements can be places anywhere in the code, but it is best
practice to add
them at the begin
of the script.
One important
module is the
os module.

26. PYTHON: LISTING FILES
# simple folder listing
homeList = os.listdir(home)
homeList.sort()
for i in homeList:
if not i.startswith('.') and not i.startswith('_'):
path = os.path.join(home, i)
isFile = os.path.isfile(path)
typeString = "file" if isFile else "folder"
print i , " -> " , typeString
# recursive file listing
count = 0
for path,dirsInPath,filesInPath in os.walk(home):
count = count + 1
print "nFiles & folders in: " + path
print "----------------------------"
for foldername in dirsInPath:
print "FOLDER:", os.path.join(path,foldername)
for filename in filesInPath:
print "FILE:", os.path.join(path,filename)
if count > 1:
break

27. PYTHON: WRITE/READ FILESnewFile = home + os.sep + "testFileToBeRemoved.txt"
# open a new file in write mode
f = open(newFile, "w")
f.write("""A new header.
A new testline.
A second testline.
n""")
lines = ["line1n", "line2n", "line3n"]
f.writelines(lines)
f.close()
# open file in read mode
f = open(newFile, "r")
# read everything into a list
lines = f.readlines()
for line in lines:
print line
f.close()
# read line by line
f = open(newFile, "r")
line = f.readline()
while line:
print line.rstrip()
line = f.readline()
f.close()
# delete the file
os.remove(newFile)

28. PYTHON: PARSE A CSV FILES - PART 1
# first create a sample csv file
f = open(home + os.sep + "sample.csv", "w");
f.write("""@id, lat, lon, elev
1, 11.0, 46.0, 250
2, 11.1, 46.1, 251
# this is a comment
3, 11.2, 46.2, 252
""")
f.close()

29. PYTHON: PARSE A CSV FILES - PART 2
# now read and parse the csv values
header = []
dataList = []
f = open(home + os.sep + "sample.csv", "r");
for line in iter(f):
line = line.strip()
if len(line) == 0:
continue
elif line.startswith('#'):
print "Comment line hit: " + line
continue
elif line.startswith('@'):
print "Header line hit: " + line
line = line[1:]
lineSplit = line.split(",")
for i in lineSplit:
header.append(i.strip())
continue
else:
lineDict = {}
lineSplit = line.split(",")
for index,value in enumerate(lineSplit):
lineDict[header[index]] = value
dataList.append(lineDict)
f.close()
# print out the result of the parsing
for index, value in enumerate(dataList):
print "nline: " + str(index)
print "----------------------"
for k, v in value.items():
print k + "=" + v
We parse the file and check for:
empty lines
comments (starts with #)
the header (starts with @)
After the parsing we want to
have the complete information of
each line (field name + value).

30. PYTHON: PARSE A REAL CSV FILE
minPath = "/home/hydrologis/data/potsdam_data/22yr_T10MN"
dataFile = open(minPath, "r")
lines = dataFile.readlines()
dataFile.close()
data = {}
dataStarted = False
for line in lines:
if dataStarted or line.startswith("-90"):
dataStarted = True
lineSplit = line.split(" ")
lat = lineSplit[0].strip()
lon = lineSplit[1].strip()
point = (lon, lat)
temperatures = []
for i in range(2, len(lineSplit)-1): # last is avg
temperatures.append(float(lineSplit[i]))
data[point] = temperatures
# Potsdam, house 24: lat=52.40669, lon=12.97619
p1 = ("13", "52")
p2 = ("13", "53")
t1List = data[p1]
t1 = t1List[4];
t2 = data[p2][4];
t = (t2+t1)/2.0
print "Temperature in May in Potsdam: ", t
Parse one of the temperatures files from the dataset and create a
dictionary containing the tuple of the lat/lon and the list of the
temperature in each month.

31. CONDITIONS & LOOPS CHEATSHEET
x = False
y = False
if x:
print "x = true"
elif y:
print "x = true"
else:
print "x and y false"
if - elif -else
y = 5
x = "worked" if (y > 1) else "failed"
print x
the ternary operator
x = 5
while x > 0:
print x
x = x - 1
while loop
for v in "123456789":
print v
for i,v in enumerate("123456789"):
print str(i) + ") " + v
for loop

32. <license>
This work is released under Creative Commons Attribution Share Alike (CC-BY-SA).
</license>
<sources>
Much of the knowledge needed to create this training material has been produced
by the sparkling knights of the
<a href="http:www.osgeo.org">Osgeo</a>,
<a href="http://tsusiatsoftware.net/">JTS</a>,
<a href="http://www.jgrasstools.org">JGrasstools</a> and
<a href="http:www.gvsig.org">gvSIG</a> communities.
Their websites are filled up with learning material that can be use to grow
knowledge beyond the boundaries of this set of tutorials.
Another essential source has been the Wikipedia project.
</sources>
<acknowledgments>
Particular thanks go to those friends that directly or indirectly helped out in
the creation and review of this series of handbooks.
Thanks to Antonio Falciano for proofreading the course and Oscar Martinez for the
documentation about gvSIG scripting.
</acknowledgments>
<footer>
This tutorial is brought to you by <a href="http:www.hydrologis.com">HydroloGIS</a>.
<footer>