PART 3: THE SCRIPTING COMPOSER AND PYTHON

324 views

Published on

GEOGRAPHIC SCRIPTING IN GVSIG
HALFWAY BETWEEN USER AND DEVELOPER held at the University of Potsdam in 2016 - PART 3: THE SCRIPTING COMPOSER AND PYTHON

Published in: Engineering
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
324
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

PART 3: THE SCRIPTING COMPOSER AND PYTHON

  1. 1. GEOGRAPHIC SCRIPTING IN GVSIG HALFWAY BETWEEN USER AND DEVELOPER Geoinformation Research Group, Department of Geography University of Potsdam 21-25 November 2016 Andrea Antonello PART 3: THE SCRIPTING COMPOSER AND PYTHON
  2. 2. THE SCRIPTING COMPOSER Scripting is one of the powerful things in GIS. Non developers have the possibility to create great automatisms through it. gvSIG's scripting composer is based on the (through jython). python scripting language
  3. 3. THE SCRIPTING COMPOSER To open the scripting composer access the Tools -> Scripting menu:
  4. 4. THE SCRIPTING COMPOSER Once opened, it shows up like: new script
  5. 5. THE SCRIPTING COMPOSER Once opened, it shows up like: new script save script cut copy paste local scripts library editor output console and error view run or save and run syntax check* *if Composer Tools installed
  6. 6. THE SCRIPTING COMPOSER Running and checking errors run output error console If you run the script by hitting the run button, the editor will not save the script for you (CTRL-S). It does if you instead use the save+run button.
  7. 7. THE BASE LANGUAGE: PYTHON! In order to be able exploit 100% the scripting engine, it is mandatory to get a good insight of the base language that can be used in the editor. Before we start with the geo part, we will investigate a bit python. Let's create a new script to do so. Let's call it python_essentials.py and let it print out a simple message to make sure it works. def main(*args): print "Python essentials" print "*****************"
  8. 8. PYTHON: FORMATTING In python formatting is mandatory. It is the formatting that defines the logical blocks. For example this will not work: def main(*args): print "Python essentials" print "*****************" and produce an error like: IndentationError: unindent does not match any outer indentation level in <script> at line number 4 at column number 1 python_essentials 4:1
  9. 9. PYTHON: WRITING STRINGS We have seen writing of strings already: # write a string print "This is a string" lines that begin with a hash at the begin are ignored -> comments Comments can be placed without formatting. Also more complex strings can be used, by surrounding text with triple quotes: # write a complex string print """Inside triple quotes you can do almost anything. It is kind of an advanced string!""" And also kind of defeat the laws of formatting.
  10. 10. PYTHON: VARIABLES Variables definition is very simple. Just write a name and assign a value: # define variables # a string name = "Luis" # a number age = 25 # insert the variables into a string template # through the use of %s, %f, %i myTemplate1 = "Hi, my name is %s and I'm %i." % (name, age) print myTemplate1 # strings and numbers can also be concatenated through the + sign myTemplate2 = "Hi, my name is " + name + " and I'm " + str(age) + "." print myTemplate2 Note that string concatenation with the + (plus) sign works only with string. Hence, the age variable has first to be converted to a string through the str(...) function.
  11. 11. PYTHON: WORKING WITH STRING Python allows for a whole pile of operations on string. Let's focus on those we will use later: # strip whitespaces string = " a line of text with spaces " print string.lstrip() print string.rstrip() print string.strip() # split a string string = "a, comma, separated, text" print string.split(",") # starts/ends with string = "a b c" print string.startswith("a") print string.endswith("a") # to upper/lowercase string = "A b C" print string.upper() print string.lower() # substrings string = "123456" print string[2:] print string[:-2] print string[2:-2] # find in strings (0 indexed) print string.find("2") print string.find("22") print string.find("345") # find after certain index print string.find("345", index) # replace s = "Hola world" print s.replace("world", "mundo")
  12. 12. PYTHON: FORMATTING NUMBERS When printing out numbers, they can be formatted as follows print "Formatting numbers" oneThird = 1.0/3.0 print "%f" % oneThird print "%i" % oneThird print "%2.2f" % oneThird print "%5.2f" % oneThird print "%2.1f" % oneThird print "%+2.2f" % oneThird %[width][.precision]type
  13. 13. EXTRACT DATA FROM THE WEB In python it is very simple to get data from the web, all that is necessary is the import of the urllib and a few lines of code. For example, let's print out the tiobe index that defines the ranking of the programming languages: import urllib def main(*args): connection = urllib.urlopen("http://www.tiobe.com/tiobe-index/") htmlString = connection.read() print htmlString
  14. 14. EXTRACT DATA FROM THE WEB As an exercise let's extract the table of the ranking of the programming languages using what we learned until now: index = htmlString.find('TIOBE Index for November 2016') # extract the header startHeader = htmlString.find('<thead', index) endHeader = htmlString.find('</thead', startHeader) tableHeader = htmlString[startHeader:endHeader] tableHeader = tableHeader.replace("<thead>","").replace("</thead>","") .replace("<th>","").replace("</th>","").replace("<tr>","") .replace("</tr>","").replace("n"," | ").strip() print tableHeader startData = htmlString.find('<tbody', index) endData = htmlString.find('</tbody', startData) tableData = htmlString[startData:endData] tableData = tableData.replace("n"," | ").replace("<tbody>","").replace("</tbody>","") .replace("<td>","").replace("</td>","|").replace("<tr>","") .replace("</tr>","n|").strip() print tableData
  15. 15. PYTHON: TYPE AND DIR type and dir can be two valuable friends in moments of desperation: type: returns the type of the passed object dir: returns the list of valid attributes and methods of the object passed string = "A string" print type(string) print dir(string)
  16. 16. PYTHON: LISTS Lists are containers for objects and are defined through square brackets: # working with lists # create a list of strings list = ["Merano", "Bolzano", "Trento"] print list # how to access the elements? print "The elements start at position 0: " + list[0] # what if you use negative indexes? print "Element at position -1: " + list[-1] # how to add an element to the end of the list? list.append("Potsdam") print list # how to add an element as first element? list.insert(0, "Berlin") print list # how to remove an element from the list? # by object? list.remove("Potsdam") print list # or by index through the del statement del list[0] print list # how many elements are in the list? print "Elements count: %s" % len(list)
  17. 17. PYTHON: LOOPING THROUGH LISTS # looping through lists list = ["Merano", "Bolzano", "Trento"] # to loop over a list a for loop can be used for value in list: print value # if you also need an index for index, value in enumerate(list): print "%s is town N.%i" % (value, index)
  18. 18. PYTHON: SORTING LISTS # sorting lists list = ["Merano", "Bolzano", "Trento"] print "This is the original list: " + str(list) # lists can be sorted list.sort() print "This is the sorted list: " + str(list) # and reverse sorted list.reverse() print "This is reverse list: " + str(list) To print lists concatenated with strings we use the str(...) function.
  19. 19. PYTHON: FILTERING LISTS # lists and numbers list = [12, 5, 7, 1, 2, 3] print "This is the original list: " + str(list) # calculate min, max and sum print "max is %i, min is %i, sum is %i" % (max(list), min(list), sum(list)) # filter based on conditions # create the filter myFilter = lambda x: x > 5.5 # apply it filteredData = filter(myFilter, list) print filteredData The filter applies the lambda expression to each element of the list, assigning it as x variable of the lambda expression.
  20. 20. PYTHON: CONCATENATE LISTS # concatenate and print lists nicely abc = ["a", "b", "c"] cde = ["c", "d", "e"] newabcde = abc + cde print newabcde # through 'join' the list can be # concatenated to one single string print ",".join(newabcde) print " | ".join(newabcde)
  21. 21. PYTHON: MODIFY LISTS # modify each element of the list with # a lambda and map abc = ["a", "b", "c"] print "Original abc: " + str(abc) # create a function to apply to the elements lambdaFunction = lambda x: x.upper() # and apply it through the map statmente modifiedAbc = map(lambdaFunction, abc) print "Modified abc: " + str(modifiedAbc) # what if you want an index in the lambda? lambdaFunction = lambda (i, x): "%i) %s" % (i, x.upper()) modifiedAbc = map(lambdaFunction, enumerate(abc)) print "Modified abc with index: " + str(modifiedAbc)
  22. 22. PYTHON: DICTIONARIES # working with dictionaries # create a dictionary from key1:value1, key2:value2, ... townsProvinceMap = {'merano':'BZ', 'bolzano':'BZ', 'trento':'TN'} print townsProvinceMap # ways of accessing the values of a map # the "array" way print townsProvinceMap['merano'] # through the "getter" print townsProvinceMap.get('merano') # add an element townsProvinceMap['St. Walburg'] = "BZ" print townsProvinceMap # remove an element del townsProvinceMap['St. Walburg'] print townsProvinceMap
  23. 23. PYTHON: LOOPING THROUGH DICTIONARIES # check if a key is present in the dict print "has key 'merano'? %s" % townsProvinceMap.has_key("merano") # get a list of keys, values, items print "keys: %s" % townsProvinceMap.keys() print "values: %s" % townsProvinceMap.values() print "items: %s" % townsProvinceMap.items() # how do we check the type of a variable print type(townsProvinceMap.keys()) # loop through a dictionary for key, value in townsProvinceMap.items(): print key + " = " + value # or through the list of keys for key in townsProvinceMap.keys(): print key + " = " + townsProvinceMap[key]
  24. 24. PYTHON: LOOP WITH RANGES # working with ranges # they are used for iterations for i in range(0, 3): print "Iteration: %i" % i # using xrange for i in xrange(0, 3): print "Xrange iteration: %i" % i # xrange vs. range print type(xrange(0, 3)) print type(range(0, 3)) # xrange creates a number generator, not a list
  25. 25. PYTHON: WORKING WITH THE FILESYSTEM import os import getpass def main(*args): # get the user home folder home = os.path.expanduser("~") # get the user name user = getpass.getuser() print "home: " + home print "user: " + user # use the file separator newDir = home + os.sep + "tmp_folder_" + user print "create folder: " + newDir # create a folder os.mkdir(newDir) print "folder exists: %s" % os.path.exists(newDir) To work with the filesystem it is necessary to first understand imports, that are statements used to make modules available. That way any external library and script can be used from within the code. Import statements can be places anywhere in the code, but it is best practice to add them at the begin of the script. One important module is the os module.
  26. 26. PYTHON: LISTING FILES # simple folder listing homeList = os.listdir(home) homeList.sort() for i in homeList: if not i.startswith('.') and not i.startswith('_'): path = os.path.join(home, i) isFile = os.path.isfile(path) typeString = "file" if isFile else "folder" print i , " -> " , typeString # recursive file listing count = 0 for path,dirsInPath,filesInPath in os.walk(home): count = count + 1 print "nFiles & folders in: " + path print "----------------------------" for foldername in dirsInPath: print "FOLDER:", os.path.join(path,foldername) for filename in filesInPath: print "FILE:", os.path.join(path,filename) if count > 1: break
  27. 27. PYTHON: WRITE/READ FILESnewFile = home + os.sep + "testFileToBeRemoved.txt" # open a new file in write mode f = open(newFile, "w") f.write("""A new header. A new testline. A second testline. n""") lines = ["line1n", "line2n", "line3n"] f.writelines(lines) f.close() # open file in read mode f = open(newFile, "r") # read everything into a list lines = f.readlines() for line in lines: print line f.close() # read line by line f = open(newFile, "r") line = f.readline() while line: print line.rstrip() line = f.readline() f.close() # delete the file os.remove(newFile)
  28. 28. PYTHON: PARSE A CSV FILES - PART 1 # first create a sample csv file f = open(home + os.sep + "sample.csv", "w"); f.write("""@id, lat, lon, elev 1, 11.0, 46.0, 250 2, 11.1, 46.1, 251 # this is a comment 3, 11.2, 46.2, 252 """) f.close()
  29. 29. PYTHON: PARSE A CSV FILES - PART 2 # now read and parse the csv values header = [] dataList = [] f = open(home + os.sep + "sample.csv", "r"); for line in iter(f): line = line.strip() if len(line) == 0: continue elif line.startswith('#'): print "Comment line hit: " + line continue elif line.startswith('@'): print "Header line hit: " + line line = line[1:] lineSplit = line.split(",") for i in lineSplit: header.append(i.strip()) continue else: lineDict = {} lineSplit = line.split(",") for index,value in enumerate(lineSplit): lineDict[header[index]] = value dataList.append(lineDict) f.close() # print out the result of the parsing for index, value in enumerate(dataList): print "nline: " + str(index) print "----------------------" for k, v in value.items(): print k + "=" + v We parse the file and check for: empty lines comments (starts with #) the header (starts with @) After the parsing we want to have the complete information of each line (field name + value).
  30. 30. PYTHON: PARSE A REAL CSV FILE minPath = "/home/hydrologis/data/potsdam_data/22yr_T10MN" dataFile = open(minPath, "r") lines = dataFile.readlines() dataFile.close() data = {} dataStarted = False for line in lines: if dataStarted or line.startswith("-90"): dataStarted = True lineSplit = line.split(" ") lat = lineSplit[0].strip() lon = lineSplit[1].strip() point = (lon, lat) temperatures = [] for i in range(2, len(lineSplit)-1): # last is avg temperatures.append(float(lineSplit[i])) data[point] = temperatures # Potsdam, house 24: lat=52.40669, lon=12.97619 p1 = ("13", "52") p2 = ("13", "53") t1List = data[p1] t1 = t1List[4]; t2 = data[p2][4]; t = (t2+t1)/2.0 print "Temperature in May in Potsdam: ", t Parse one of the temperatures files from the dataset and create a dictionary containing the tuple of the lat/lon and the list of the temperature in each month.
  31. 31. CONDITIONS & LOOPS CHEATSHEET x = False y = False if x: print "x = true" elif y: print "x = true" else: print "x and y false" if - elif -else y = 5 x = "worked" if (y > 1) else "failed" print x the ternary operator x = 5 while x > 0: print x x = x - 1 while loop for v in "123456789": print v for i,v in enumerate("123456789"): print str(i) + ") " + v for loop
  32. 32. <license> This work is released under Creative Commons Attribution Share Alike (CC-BY-SA). </license> <sources> Much of the knowledge needed to create this training material has been produced by the sparkling knights of the <a href="http:www.osgeo.org">Osgeo</a>, <a href="http://tsusiatsoftware.net/">JTS</a>, <a href="http://www.jgrasstools.org">JGrasstools</a> and <a href="http:www.gvsig.org">gvSIG</a> communities. Their websites are filled up with learning material that can be use to grow knowledge beyond the boundaries of this set of tutorials. Another essential source has been the Wikipedia project. </sources> <acknowledgments> Particular thanks go to those friends that directly or indirectly helped out in the creation and review of this series of handbooks. Thanks to Antonio Falciano for proofreading the course and Oscar Martinez for the documentation about gvSIG scripting. </acknowledgments> <footer> This tutorial is brought to you by <a href="http:www.hydrologis.com">HydroloGIS</a>. <footer>

×