Advanced geoprocessing with…               MAGIC 2012   Chad Cooper – chad@cast.uark.edu Center for Advanced Spatial Techn...
Intros• your name• what you do/where you work• used Python much?  – any formal training?  – what do you use it for?• know ...
Objectives• informal class  – expect tangents  – code as we go• not geared totally to ArcGIS• THINK – oddball and out of t...
Outline•   data types review    • documentation•   functions            • 3rd party modules•   procedural vs. OOP   • modu...
Strings•   ordered collections of characters•   immutable – can’t change it•   raw strings: path = r”C:tempchad”•   slicin...
Strings• string formatting:    „a %s parrot‟ % „dead‟                            „a dead parrot‟• useful string formatting...
Lists• list – ordered collection of arbitrary objects  list1 = [0,1,2,3]  list2 = [zero,one,two,three]  list3 = [0,zero,1,...
Lists…• iterable – very important!     for l in list3                                        0                            ...
Dictionaries• unordered collection of arbitrary objects   d = {1:‟foo‟, 2:‟bar‟}• key/value pairs – think hash/lookup tabl...
Dictionaries• iterable  d.iteritems()  <dictionary-itemiterator object at  0x1D2D8330>  for k, v in d.iteritems():     pri...
Tuples•   ordered collection of arbitrary objects•   immutable – cannot add, remove, find•   access by offset•   basically...
List comprehensions• Map one list to another by applying a function  to each of the list elements• Original list goes unch...
Sets• unordered collections of objects• like mathematical sets – collection of distinct  objects – NO DUPLICATES• example ...
Sets• get rid of dups via set:   >>> L1=[2,2,3,4,5,5,3]   >>> set(L1)   set([2, 3, 4, 5])   >>>   L1 = list(set(L1))   >>>...
Sets   >>>   L1   [2,   2, 3, 4, 5, 5, 3]   >>>   L2   [4,   5, 6, 7]• intersection – data are the same   >>> set(L1).inte...
Programming paradigms:            big blob of code•   OK on a small scale for GP scripts•   gets out of hand quickly•   ha...
Programming paradigms:        procedural programming• basically a list of instructions• program is built from one or more ...
Functions• portion of code within a larger program that  performs a specific task• can be called anytime, anyplace• can ac...
Functionsimport arcpydef get_raster_props(in_raster):    """Get properties of a raster, return as dict"""    # Cast input ...
Programming paradigms:                Procedural exampleimport arcpydef add_field(in_fc="Magic.gdb/Fields",              i...
Programming paradigms: Object-oriented programming (OOP)• break program down into data types (classes)  that associate beh...
Programming paradigms:Object-oriented programming (OOP)      import arcpy      class Fields(object):          """Class for...
Programming paradigms: Object-oriented programming (OOP)• objects let you wrap complex processes, but  present a simple in...
Programming paradigms:           OOP - Inheritance• classes can inherit attributes and methods• allows you to reuse and cu...
import arcpy                         class Fields(object):                             """Class for working with fields"""...
import arcpy                         class Fields(object):                             """Class for working with fields"""...
Modularizing code• I’m lazy, so I want to reuse code• import statement – call functionality in  another module• Have one c...
Geometries• heirarchy:  – feature class is made of features  – feature is made of parts  – part is made of points• heirarc...
Reading geometry• accessed through the geometry object of a  feature• example: describe_geometry_arcmap.py1.open up       ...
Reading geometryimport arcpydesc =arcpy.Describe("Points")sfn = desc.ShapeFieldNamerows =arcpy.SearchCursor("Points")for r...
import arcpy                   infc = "Magic.gdb/Polygons"                   # Identify the geometry fieldReading geometry...
import arcpy                   infc = "Magic.gdb/Polygons"Reading geometry                   desc = arcpy.Describe(infc)  ...
Writing geometry• arcpy.Point• point features are point objects, lines and  polygons are arrays of point objects  – arcpy....
data_list = [[33.09500,-93.90389],                                [33.03194,-93.89806],                                [34...
import arcpy                   arcpy.env.overwriteOutput = 1                   # A list of features and coordinate pairs  ...
Rasters• arcpy.Raster class  – raster object: variable that references a raster    dataset  – gives access to raster props...
Rastersimport arcpydef get_raster_props(in_raster):    """Get properties of a raster, return as dict"""    # Cast input la...
Spatial references• can get properties from arcpy.Describe  >>> sr =  arcpy.Describe(fc).spatialReference  >>> sr.type  u‟...
Spatial references• arcpy.SpatialReference class  • methods to create/edit spatial refs >>>   sr_utm = arcpy.SpatialRefere...
Exception Handling•   It’s necessary, stuff fails•   Useful error reporting•   Proper application cleanup•   Combine it wi...
Exception handling – try/except• most basic form of error handling• wrap whole program or portions of code• use optional f...
Exception handlingimport arcpytry:     arcpy.Buffer_analysis("Observer")except Exception as e:     print e.message
Exception handlingimport arcpytry:    if arcpy.CheckExtension("3D") == "Available":         arcpy.CheckOutExtension("3D") ...
Exception handling - raise• allows you to force an exception to occur• can be used to alert of conditions
Exception handling - raiseimport arcpyclass LicenseError(Exception):    passtry:       if arcpy.CheckExtension("3D") == "A...
Exception handling         AddError and traceback• AddError – returns GP-specific errors• traceback – prints stack trace; ...
import arcpy                         import sysAddError and traceback   import traceback Exception handling –             ...
Logging• logging module• logging levels:  – DEBUG: detailed; for troubleshooting  – INFO: normal operation, statuses  – WA...
Super-basic loggingimport logginglogging.warning("Look out!")logging.info("Does this print?")
Super-basic logging to a log fileimport logginglogging.basicConfig(filename=log_example.log,                    level=logg...
Super-basic logging to a log fileimport logginglogging.basicConfig(filename="log_example.log",level=logging.DEBUG)logging....
Meaningful logging•   “customize” the logger•   add in info-level message(s) to get logged•   log our errors to log file• ...
import   arcpy                     import   sys                     import   traceback                     import   loggin...
Meaningful logging
Code documentation•   Pythonic standards covered in PEPs 8 and 257•   help()•   comments need to be worth it•   name items...
Creating documentation• pydoc – built-in; used by help()  – generate HTML on any module  – kinda plain• epydoc – old, rumo...
Branching out
Installing packages
Installing packages (on Windows)• Windows executables• Python eggs  – .zip file with metadata, renamed .egg  – distributes...
pipC:pip search “kml”C:pip install BeautifulSoupC:pip install –upgrade pykmlC:pip uninstall BeautifulSoup• can take care o...
virtualenv• a tool to create isolated Python environments• manage dependencies on a per-project basis,  rather than global...
virtualenv• install via pip, easy_install, or by    C:python virtualenv.py• create the env    C:dir virtualenv <env>• acti...
virtualenv• installs Python where you tell it, modifies  system path to point there  – good only while the env is activate...
virtualenv• YES, with a little work...>>>execfile(rC:<env>Scriptsactivate_this.py, {__file__:• tells ArcMap to use Python ...
The web• Infinite source of information• Right-click and “Save as” is so lame (and too  much work)• Python can help you ex...
Fetching data• Built-in libraries for ftp and http• ftplib – log in, nav to directory, retrieve files• urllib/urllib2 – pa...
Fetching dataimport urlliburllib.urlretrieve("http://www.fhwa.dot.gov/bridge/nbi/2011/RI11.txt",                   "C:/tem...
Scraping• Scrape data from a web page• Well-structured content is a HUGE help, as is  valid markup, which isn’t always the...
Scraping addresseshttp://www.phillypal.com/pal_locations.php
Scraping addressesimport BeautifulSoup as bsimport urllib2url = "http://www.phillypal.com/pal_locations.php"# Open the URL...
Scraping addresses   1845 N. 23rd Street, 19121   3301 Tasker Street, 19145   5801 Media Street, 19131   250 S. 63rd Stree...
Emailing• smtp built-in library• best if you have IP of your email server• port blocking can be an issue   import smtplib ...
Files• built in open function – slurp entire file into  memory – OK except for huge files        data = open(file).read()....
Excel• love, hate, love• many modules out there  – xlrd (read) / xlwt (write) – only .xls  – openPyXL – read/write .xlsx• ...
Reading Excelimport xlrd# Open the workbookwb = xlrd.open_workbook(Employees.xls)wb.sheet_names()# Get first sheetsh=wb.sh...
# Write an XLS file with a single worksheet, containing                # a heading row and some rows of data.             ...
Writing Excel
Databases•   You can connect to pretty much ANY database•   Is there one true solution??•   pyodbc – Access, SQL Server, M...
Resources - FREE•   Dive into Python•   Python Cookbook•   Think Python•   Python docs•   gis.stackexchange.com•   Google ...
Conferences• pyArkansas – annually in Conway  – pyar2 list on python.org• PyCon – THE national US Python conference• FOSS4...
IDEs and editors•   Wing – different license levels, good people•   PyScripter – open source, code completion•   Komodo – ...
More reading• http://www.voidspace.org.uk/python/articles/  OOP.shtml - great OOP article (which I used a  a lot)
Advanced geoprocessing with Python
Advanced geoprocessing with Python
Advanced geoprocessing with Python
Advanced geoprocessing with Python
Upcoming SlideShare
Loading in...5
×

Advanced geoprocessing with Python

3,961

Published on

4-hour short course given at the Mid-America GIS Consortium Biennial meeting, April 2012, Kansas City, MO.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,961
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
144
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • __init__  method signature. Called when method is instantiated, which is done by calling the class. Class by itself is just a structure, instance of a class contains content. A class is like a form that everyone has to fill out – same type of form. But content of the form varies from person to person. Your copy of the form with your specific info is an instance of the form.self  a reference to the __init__ method. self lets you access __init__ object attributes. in Fields class, in_fc and in_fields are object attributes that get set when __init__ is called. That’s how we do print f.infields.The “functions” that are part of an object are called methods. The values are called attributes.
  • __init__  method signature. Called when method is instantiated, which is done by calling the classself  a reference to the __init__ method. self lets you access __init__ object attributes. in Fields class, in_fc and in_fields are object attributes that get set when __init__ is called. That’s how we do print f.infields.The “functions” that are part of an object are called methods. The values are called attributes.
  • Show how Data Management\\Raster\\Raster Properties\\Build Batch Pyramids uses ConversionUtils.py in C:\\Program Files (x86)\\ArcGIS\\Desktop10.0\\ArcToolbox\\Scripts
  • Run: 1) describe_geometry_arcmap.py 2) describe_geometry_polygons.py 3) describe_geometry_polygons_2.py
  • Run: 1) describe_geometry_arcmap.py 2) describe_geometry_polygons.py 3) describe_geometry_polygons_2.py
  • Run: 1) describe_geometry_arcmap.py 2) describe_geometry_polygons.py 3) describe_geometry_polygons_2.py
  • Run: 1) describe_geometry_arcmap.py 2) describe_geometry_polygons.py 3) describe_geometry_polygons_2.py
  • Do nbi_list.txt exercise. The function call is PushNbiToFeatureclass(r”path to fc”, data_list)Do create_geometry_polygon.py. Run it, examine output, then mess around with the coordList.
  • Run get_raster_props.py within arcmap Python console.d = get_raster_props(‘NWA10mNED’)# map keys to listl = [k for k in d.iterkeys()]# map values to listo = [v for v in d.itervalues()]# put into a tuplet = tuple(v for v in d.itervalues())===================================Raster calculations at ArcMap Python prompt:import arcpy.sa (explain about from arcpy.sa import * vs. import arcpy.sa)out_raster = arcpy.sa.IsNull(“NWA10mNED”)------------------------------------------------------------Look at arcpy.env settings -- in help under ArcPy site package &gt; Classes &gt; envarcpy.ListEnvironments()------------------------------------------------------Something really stupid like:out_raster_3 = arcpy.Raster(‘NWA10mNED’) * 10out_raster_3.save(path to Magic.gdb)--------------------------------------------------------outr_slope = arcpy.sa.Viewshed(“NWA10mNED”, “Observer”) ------------------------------------------------------------where are raster calcs stored? arcpy.env.scratchworkspace
  • Example – get_raster_props.py used in ArcMap
  • &gt;&gt;&gt; import arcpy&gt;&gt;&gt; sr = arcpy.Describe(&quot;KittyHawkClip&quot;).spatialReference&gt;&gt;&gt; sr.typeu&apos;Projected&apos;&gt;&gt;&gt; =======================================sr.factoryCodesr.GCSCode,sr.PCSCode, sr.name
  • sr_utm.exportToString()sr3 = arcpy.SpatialReference()sr3.createFromFile(path to prj)sr3.namesr3.type
  • Finally used to execute tasks that should be executed whether an exception has occurred or not.For finally test, change the second argument in Slope_3D call to a integer, it will bomb.
  • Scenarios for error_handling_AddError_traceback.py:arcpy.GetCount_management(&quot;&quot;)x = &quot;a&quot; + 1float(&quot;a text string&quot;)
  • Log files can save your hide.Console output is great when testing or developing code, but no good for production environments. Print statements are worthless.Go to docs.python.org/howto/logging.html (link in logging on slide) and look over “When to use logging”.
  • Info doesn’t print because default level is warning, and info is lower level than warning.
  • All print out since we explicitly set level to debug, the lowest level.
  • arcpy.GetCount_management(&quot;&quot;)x = &quot;a&quot; + 1float(&quot;a text string&quot;)float(10) – will work and print out our info statement “DONE”logging_meaningful.py
  • At command prompt:import oshelp(os.path.join)At ArcMap Python prompt:import arcpyhelp(arcpy.whatever) -- play around here====================================The purpose of commenting is to help the reader know as much as the writer did.
  • PYDOCC:\\&gt; python –m pydoc sysat python prompt:import syshelp(sys)  same as pydoc callD:\\Projects\\Chad\\Training\\MAGIC\\Code&gt;python C:\\Python26\\ArcGIS10.0\\Lib\\pydoc.py –gD:\\Projects\\Chad\\Training\\MAGIC\\Code&gt;python C:\\Python26\\ArcGIS10.0\\Lib\\pydoc.py -w arcpyshow the Python docs for Sphinx example
  • Browse packages | Topic | Scientific/Engineering | Topic | GIS
  • Install and uninstall BeautifulSoup&gt;&gt; pip install BeautifulSoup&gt;&gt; pip uninstall BeautifulSoup
  • nbi_url.txt&gt;&gt;&gt; import urllib&gt;&gt;&gt; urllib.urlretrieve(&quot;http://www.fhwa.dot.gov/bridge/nbi/2011/RI11.txt&quot;, &quot;C:/temp/RI11.txt&quot;)(&apos;C:/temp/RI11.txt&apos;, &lt;httplib.HTTPMessage instance at 0x286F9990&gt;)&gt;&gt;&gt;
  • Students need BeautifulSoup.py module in same dir as their scripts.
  • http://www.phillypal.com/pal_locations.phpInspect the address element in Chromescraping_philly_pals.py
  • Love-hate relationship with Excel.
  • Lots of talks by the big-hitters of Python
  • Advanced geoprocessing with Python

    1. 1. Advanced geoprocessing with… MAGIC 2012 Chad Cooper – chad@cast.uark.edu Center for Advanced Spatial Technologies University of Arkansas, Fayetteville
    2. 2. Intros• your name• what you do/where you work• used Python much? – any formal training? – what do you use it for?• know any other languages?
    3. 3. Objectives• informal class – expect tangents – code as we go• not geared totally to ArcGIS• THINK – oddball and out of the ordinary applications will make you want more…
    4. 4. Outline• data types review • documentation• functions • 3rd party modules• procedural vs. OOP • module installation• geometries • the web• rasters – fetching – scraping• spatial references – email• error – FTP handling/logging • files
    5. 5. Strings• ordered collections of characters• immutable – can’t change it• raw strings: path = r”C:tempchad”• slicing fruit[0] „b‟• indexing: fruit[1:3] >> „an‟• iteration/membership: for each in fruit „f‟ in fruit
    6. 6. Strings• string formatting: „a %s parrot‟ % „dead‟ „a dead parrot‟• useful string formatting: import arcpy f = "string" arcpy.CalculateField_management(fc, “some_field", "%s" % f)
    7. 7. Lists• list – ordered collection of arbitrary objects list1 = [0,1,2,3] list2 = [zero,one,two,three] list3 = [0,zero,1,one,2,two,3,three]• ordered list2.sort() list2.sort(reverse=True) [one,three,...] [zero,two,...]• mutable – you can change it list1.append(4) list1.reverse() list2.insert(0,‟one-half‟) [0,1,2,3,4] [4,3,2,1,0] [„one-half‟,‟zero‟…] list2.extend([„four‟,‟five‟]) <- Extend concats lists
    8. 8. Lists…• iterable – very important! for l in list3 0 zero ...• membership 3 in list3 --> True• nestable – 2D array/matrix list4 = [[0,1,2], [3,4,5], [6,7,8]]• access by index – zero based list4[1] list4[1][2] [3,4,5] 5
    9. 9. Dictionaries• unordered collection of arbitrary objects d = {1:‟foo‟, 2:‟bar‟}• key/value pairs – think hash/lookup table (keys don’t have to be numbers) d.keys() d.values() [1, 2] [„foo‟,‟bar‟]• nestable, mutable d[3] = „spam‟ del d[key]• access by key, not offset d[2] >> „bar‟
    10. 10. Dictionaries• iterable d.iteritems() <dictionary-itemiterator object at 0x1D2D8330> for k, v in d.iteritems(): print k, v ... 1 foo 2 bar
    11. 11. Tuples• ordered collection of arbitrary objects• immutable – cannot add, remove, find• access by offset• basically an unchangeable list (1,2,‟three‟,4,…)• so what’s the purpose? – FAST – great for iterating over constant set of values – SAFE – you can’t change it
    12. 12. List comprehensions• Map one list to another by applying a function to each of the list elements• Original list goes unchanged L = [2,4,6,8] J = [elem * 2 for elem in L] >>> J [4, 8, 12, 16]
    13. 13. Sets• unordered collections of objects• like mathematical sets – collection of distinct objects – NO DUPLICATES• example – get rid of dups in a list via list comp L1=[2,2,3,4,5,5,3] L2=[] [L2.append(x) for x in L1 if x not in L2] >>> L2 [2, 3, 4, 5]
    14. 14. Sets• get rid of dups via set: >>> L1=[2,2,3,4,5,5,3] >>> set(L1) set([2, 3, 4, 5]) >>> L1 = list(set(L1)) >>> >>> L1 [2, 3, 4, 5]• union: >>> L2 = [4,5,6,7] >>> L1 + L2 [2, 3, 4, 5, 4, 5, 6, 7] >>> >>> list(set(L1).union(set(L2))) [2, 3, 4, 5, 6, 7]
    15. 15. Sets >>> L1 [2, 2, 3, 4, 5, 5, 3] >>> L2 [4, 5, 6, 7]• intersection – data are the same >>> set(L1).intersection(set(L2)) set([4, 5]) >>>• symmetrical difference – data are not the same >>> set(L1).symmetric_difference(set(L2)) set([2, 3, 6, 7])• difference – data in first set but not second >>> set(L1).difference(set(L2)) set([2, 3]) >>> set(L2).difference(set(L1)) set([6, 7])
    16. 16. Programming paradigms: big blob of code• OK on a small scale for GP scripts• gets out of hand quickly• hard to follow• think ModelBuilder-exported code
    17. 17. Programming paradigms: procedural programming• basically a list of instructions• program is built from one or more procedures (functions) – reusable chunks• procedures called at anytime, anywhere in program• focus is to break task into collection of variables, data structures, subroutines• natural style, easy to understand• strict separation between code and data
    18. 18. Functions• portion of code within a larger program that performs a specific task• can be called anytime, anyplace• can accept arguments >>> def foo(bar):• should return a value ... print bar ...• keeps code neat >>> foo(“yo”)• promotes smooth flow yo
    19. 19. Functionsimport arcpydef get_raster_props(in_raster): """Get properties of a raster, return as dict""" # Cast input layer to a Raster r = arcpy.Raster(in_raster) raster_props = {} # Create empty dictionary to put props in below raster_props["x_center"] = r.extent.XMin + (r.extent.XMax - r.extent. raster_props["y_center"] = r.extent.YMin + (r.extent.YMax - r.extent. raster_props["max_elev"] = r.maximum raster_props["min_elev"] = r.minimum raster_props["no_data"] = r.noDataValue raster_props["terr_width"] = r.width raster_props["terr_height"] = r.height raster_props["terr_cell_res"] = r.meanCellHeight # Return the dictionary of properties return raster_props
    20. 20. Programming paradigms: Procedural exampleimport arcpydef add_field(in_fc="Magic.gdb/Fields", in_fields=[("Distance", "Float", "0"), ("Name", "Text", 50)]): """Add fields to FC""" for in_field in in_fields: if in_field[1] == Text: arcpy.AddField_management(in_fc,in_field[0],in_field[1],"# "#",in_field[2],"#","NULLABLE","NON_REQUIRED", else: arcpy.AddField_management(in_fc,in_field[0],in_field[1],"# "#","#","#","NULLABLE","NON_REQUIRED","#")add_field()
    21. 21. Programming paradigms: Object-oriented programming (OOP)• break program down into data types (classes) that associate behavior (methods) with data (members or attributes)• code becomes more abstract• data and functions for dealing with it are bound together in one object
    22. 22. Programming paradigms:Object-oriented programming (OOP) import arcpy class Fields(object): """Class for working with fields""" # __init__ --> method signature def __init__(self, in_fc="Magic.gdb/Fields", in_fields=[("Distance", "Float", "0"), ("Name", "Text", 50)]): self.in_fc = in_fc self.in_fields = in_fields def add_field(self): """Add fields to FC""" for in_field in self.in_fields: if in_field[1] == "Text": arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", in_field[2], "#", "NULLABLE", "NON_REQUIRED", "#") else: arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", "#", "#", "NULLABLE", "NON_REQUIRED", "#") if __name__ == "__main__": # Instantiate the Fields class f = Fields() # Call the add_field method f.add_field() print f.in_fields print f.in_fc
    23. 23. Programming paradigms: Object-oriented programming (OOP)• objects let you wrap complex processes, but present a simple interface to them• methods and attributes are encapsulated inside the object• methods and attributes are exposed to users• you can then update the object without breaking the interface• you can pass objects around - carefully
    24. 24. Programming paradigms: OOP - Inheritance• classes can inherit attributes and methods• allows you to reuse and customize existing code inside a new class• you can override methods• you can add new methods to a class without modifying the existing class
    25. 25. import arcpy class Fields(object): """Class for working with fields"""Programming paradigms: def __init__(self, in_fc="Magic.gdb/Fields", in_fields=[("Distance", "Float", "0"), ("Name", "Text", 50)]): self.in_fc = in_fc OOP - Inheritance self.in_fields = in_fields def add_field(self): """Add fields to FC""" for in_field in self.in_fields: if in_field[1] == "Text": arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", in_field[2], "#", "NULLABLE", "NON_REQUIRED", "#") else: arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", "#", "#", "NULLABLE", "NON_REQUIRED", "#") class MyFields(Fields): """Customized fields class""" def add_field(self): """Add fields to FC""" for in_field in self.in_fields: # Test to see if in_field exists already in featureclass if in_field[0] in [f.name for f in arcpy.ListFields(self.in_fc)]: # If field exists, delete it arcpy.DeleteField_management(self.in_fc, in_field[0]) print in_field[0], "deleted" if in_field[1] == "Text": arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", in_field[2], "#", "NULLABLE", "NON_REQUIRED", "#") else: arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", "#", "#", "NULLABLE", "NON_REQUIRED", "#") if __name__ == "__main__": # Instantiate MyFields class, which in inherits the Fields class f = MyFields() # Call add_field method f.add_field()
    26. 26. import arcpy class Fields(object): """Class for working with fields""" def __init__(self, in_fc="Magic.gdb/Fields",Programming paradigms: in_fields=[("Distance", "Float", "0"), ("Name", "Text", 50)]): self.in_fc = in_fc self.in_fields = in_fields def add_field(self): """Add fields to FC""" OOP - Inheritance for in_field in self.in_fields: if in_field[1] == "Text": arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", in_field[2], "#", "NULLABLE", "NON_REQUIRED", "#") else: arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", "#", "#", "NULLABLE", "NON_REQUIRED", "#") def get_field_props(self): desc = arcpy.Describe(self.in_fc) for field in desc.fields: print field.name, "-->", field.type class MyFields(Fields): """Customized fields class""" def add_field(self): """Add fields to FC""" for in_field in self.in_fields: if in_field[0] in [f.name for f in arcpy.ListFields(self.in_fc)]: arcpy.DeleteField_management(self.in_fc, in_field[0]) print in_field[0], "deleted" if in_field[1] == "Text": arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", in_field[2], "#", "NULLABLE", "NON_REQUIRED", "#") else: arcpy.AddField_management(self.in_fc, in_field[0], in_field[1], "#", "#", "#", "#", "NULLABLE", "NON_REQUIRED", "#") if __name__ == "__main__": # Instantiate MyFields class f = MyFields() # Call add_field method f.add_field() print f.in_fields # See, we really do inherit everything from the Fields class f.get_field_props()
    27. 27. Modularizing code• I’m lazy, so I want to reuse code• import statement – call functionality in another module• Have one custom module (a .py file) with code you use all the time• Great way to package up helper functions• ESRI does this with ConversionUtils.py C:Program Files (x86)ArcGISServer10.0ArcToolBoxScripts
    28. 28. Geometries• heirarchy: – feature class is made of features – feature is made of parts – part is made of points• heirarchy in Pythonic terms: – part: [[pnt, pnt, pnt, ...]] – multipart polygon: [[pnt, pnt, pnt, ...], [pnt, pnt, pnt, ...]] – single part polygon with hole: [[pnt, pnt, pnt, ,pnt, pnt, pnt]]
    29. 29. Reading geometry• accessed through the geometry object of a feature• example: describe_geometry_arcmap.py1.open up import arcpy desc = SearchCursor arcpy.Describe("Points")2.loop through sfn = desc.ShapeFieldName rows rows = arcpy.SearchCursor("Points"3.get geometry )4.print out X, Y for row in rows: geom = row.getValue(sfn)
    30. 30. Reading geometryimport arcpydesc =arcpy.Describe("Points")sfn = desc.ShapeFieldNamerows =arcpy.SearchCursor("Points")for row in rows: geom =row.getValue(sfn) pnt = geom.getPart() print pnt.X, pnt.Y
    31. 31. import arcpy infc = "Magic.gdb/Polygons" # Identify the geometry fieldReading geometry desc = arcpy.Describe(infc) shapefieldname = desc.ShapeFieldName # Create search cursor rows = arcpy.SearchCursor(infc) # Enter for loop for each feature/row for row in rows: # Create the geometry object feat = row.getValue(shapefieldname) # Print the current multipoints ID print "Feature %i:" % row.getValue(desc.OIDFieldName) partnum = 0 # Step through each part of the feature for part in feat: # Print the part number print "Part %i:" % partnum # Step through each vertex in the feature for pnt in feat.getPart(partnum): if pnt: # Print x,y coordinates of current point print pnt.X, pnt.Y else: # If pnt is None, this represents an interior ring print "Interior Ring:" partnum += 1
    32. 32. import arcpy infc = "Magic.gdb/Polygons"Reading geometry desc = arcpy.Describe(infc) shapefieldname = desc.ShapeFieldName rows = arcpy.SearchCursor(infc) for row in rows: feat = row.getValue(shapefieldname) print "tFeature %i:" % row.getValue(desc.OIDFieldName) partnum = 0 for part in feat: parts = [] print "Part %i:" % partnum for pnt in feat.getPart(partnum): if pnt: parts.append([pnt.X, pnt.Y]) else: parts.append(" ") partnum += 1 print parts
    33. 33. Writing geometry• arcpy.Point• point features are point objects, lines and polygons are arrays of point objects – arcpy.PolyLine, arcpy.Polygon• Geometry objects can be created using the Geometry, Mulitpoint, PointGeometry, Polygo n, or Polyline classes
    34. 34. data_list = [[33.09500,-93.90389], [33.03194,-93.89806], [34.34111,-93.50056], [34.24917,-93.67667], [34.22500,-93.89500], [33.76833,-92.48500],Writing geometry [33.74500,-92.47667], [33.68000,-92.46667], [35.05425,-94.12711], [35.03472,-94.12233], [35.03333,-94.12236], [35.01500,-94.12108], [35.00392,-94.12033]] import arcpy import time def PushNbiToFeatureclass( inFc, inList): """ Take a list of NBI data and push it directly to a FGDB point FC """ try: cur = arcpy.InsertCursor(inFc) for line in inList: t = 0 feat = cur.newRow() feat.shape = arcpy.Point(line[1], line[0]) feat.setValue("Timestamp", time.strftime("%m/%d/%Y %H:%M:%S", time.localt cur.insertRow(feat) del cur except Exception as e: print e.message PushNbiToFeatureclass(r”path to fc”, data_list)
    35. 35. import arcpy arcpy.env.overwriteOutput = 1 # A list of features and coordinate pairs coordList = [[[1,2], [2,4], [3,7]],Writing geometry [[6,8], [5,7], [7,2], [9,18]]] # Create empty Point and Array objects point = arcpy.Point() array = arcpy.Array() # A list that will hold each of the Polygon objects featureList = [] for feature in coordList: # For each coordinate pair, set the x,y properties and add to the # Array object for coordPair in feature: point.X = coordPair[0] point.Y = coordPair[1] array.add(point) # Add the first point of the array in to close off the polygon array.add(array.getObject(0)) # Create a Polygon object based on the array of points polygon = arcpy.Polygon(array) # Clear the array for future use array.removeAll() # Append to the list of Polygon objects featureList.append(polygon) # Copy Polygon object to a featureclass arcpy.CopyFeatures_management(featureList, "d:/temp/polygons.shp")
    36. 36. Rasters• arcpy.Raster class – raster object: variable that references a raster dataset – gives access to raster props• raster calculations – Map Algebra – outras = Slope(“in_raster”) – can cast to Raster object for calculations
    37. 37. Rastersimport arcpydef get_raster_props(in_raster): """Get properties of a raster, return as dict""" # Cast input layer to a Raster r = arcpy.Raster(in_raster) raster_props = {} # Create empty dictionary to put props in below raster_props["x_center"] = r.extent.XMin + (r.extent.XMax - r.extent. raster_props["y_center"] = r.extent.YMin + (r.extent.YMax - r.extent. raster_props["max_elev"] = r.maximum raster_props["min_elev"] = r.minimum raster_props["no_data"] = r.noDataValue raster_props["terr_width"] = r.width raster_props["terr_height"] = r.height raster_props["terr_cell_res"] = r.meanCellHeight # Return the dictionary of properties return raster_props
    38. 38. Spatial references• can get properties from arcpy.Describe >>> sr = arcpy.Describe(fc).spatialReference >>> sr.type u‟Projected‟ or u‟Geographic‟• arcpy.SpatialReference class • methods to create/edit spatial refs
    39. 39. Spatial references• arcpy.SpatialReference class • methods to create/edit spatial refs >>> sr_utm = arcpy.SpatialReference() >>> sr_utm.factoryCode = 26915 >>> sr_utm.create() >>> sr_utm.name ...
    40. 40. Exception Handling• It’s necessary, stuff fails• Useful error reporting• Proper application cleanup• Combine it with logging try: do something... except: handle error... finally: clean up...
    41. 41. Exception handling – try/except• most basic form of error handling• wrap whole program or portions of code• use optional finally clause for cleanup – close open files – close database connections – check extensions back in
    42. 42. Exception handlingimport arcpytry: arcpy.Buffer_analysis("Observer")except Exception as e: print e.message
    43. 43. Exception handlingimport arcpytry: if arcpy.CheckExtension("3D") == "Available": arcpy.CheckOutExtension("3D") arcpy.Slope_3d("Magic.gdb/NWA10mNED","Magic.gdb/SlopeNWA")except: print arcpy.GetMessages(2)finally: # Check in the 3D Analyst extension arcpy.CheckInExtension("3D")
    44. 44. Exception handling - raise• allows you to force an exception to occur• can be used to alert of conditions
    45. 45. Exception handling - raiseimport arcpyclass LicenseError(Exception): passtry: if arcpy.CheckExtension("3D") == "Available": arcpy.CheckOutExtension("3D") else: raise LicenseError arcpy.Slope_3d("NWA10mNED", "SlopeNWA")except LicenseError: print "3D Analyst license unavailable"except: print arcpy.GetMessages(2)finally: # Check in the 3D Analyst extension arcpy.CheckInExtension("3D")
    46. 46. Exception handling AddError and traceback• AddError – returns GP-specific errors• traceback – prints stack trace; determines precise location of error – good for larger, more complex programs
    47. 47. import arcpy import sysAddError and traceback import traceback Exception handling – arcpy.env.workspace = r"C:StudentCodeMAGIC.gdb" try: # Your code goes here float("a string") except: # Get the traceback object tb = sys.exc_info()[2] tbinfo = traceback.format_tb(tb)[0] # Concatenate information together concerning the error into a message string # tbinfo: where error occurred # sys.exc_info: 3-tuple of type, value, traceback pymsg = "PYTHON ERRORS:nTraceback info:n" + tbinfo + "nError Info:n" + str(sys.exc_info()[1]) msgs = "ArcPy ERRORS:n" + arcpy.GetMessages(2) + "n" # Return python error messages for use in script tool or Python Window arcpy.AddError(pymsg) if arcpy.GetMessages(2): arcpy.AddError(msgs) print msgs # Print Python error messages for use in Python / Python Window print pymsg + "n"
    48. 48. Logging• logging module• logging levels: – DEBUG: detailed; for troubleshooting – INFO: normal operation, statuses – WARNING: still working, but unexpected behavior – ERROR: more serious, some function not working – CRITICAL: program cannot continue
    49. 49. Super-basic loggingimport logginglogging.warning("Look out!")logging.info("Does this print?")
    50. 50. Super-basic logging to a log fileimport logginglogging.basicConfig(filename=log_example.log, level=logging.DEBUG)logging.debug(This message should getlogged)logging.info(So should this)logging.warning(And this, too)
    51. 51. Super-basic logging to a log fileimport logginglogging.basicConfig(filename="log_example.log",level=logging.DEBUG)logging.debug("This message should go to the log file")logging.info("So should this")logging.warning("And this, too")
    52. 52. Meaningful logging• “customize” the logger• add in info-level message(s) to get logged• log our errors to log file• can get much more advanced, see the docs
    53. 53. import arcpy import sys import traceback import logging import datetime log_file = "meaningful_log_%s.log" % datetime.datetime.now().strftime("%Y_%m_%d_%H_%M_%S")Meaningful logging arcpy.env.workspace = r"C:StudentCodeMAGIC.gdb" # Setup logger logging.basicConfig(level=logging.DEBUG, format=%(asctime)s %(levelname)-8s %(message)s, datefmt=%Y-%m-%d %H:%M:%S, filename=log_file, filemode=w) logging.info(: START LOGGING) try: # Your code goes here float("lfkjdlk") logging.info(": DONE") except: # Get the traceback object tb = sys.exc_info()[2] tbinfo = traceback.format_tb(tb)[0] # Concatenate information together concerning the error into a message string # tbinfo: where error occurred # sys.exc_info: 3-tuple of type, value, traceback pymsg = "PYTHON ERRORS:nTraceback info:n" + tbinfo + "nError Info:n" + str(sys.exc_info()[1]) msgs = "ArcPy ERRORS:n" + arcpy.GetMessages(2) + "n" # Return python error messages for use in script tool or Python Window arcpy.AddError(pymsg) if arcpy.GetMessages(2): arcpy.AddError(msgs) logging.error(": %s" % msgs) # Log Python error messages for use in Python / Python Window logging.error(": %s" % pymsg + "n")
    54. 54. Meaningful logging
    55. 55. Code documentation• Pythonic standards covered in PEPs 8 and 257• help()• comments need to be worth it• name items well• be precise and compact• comments may be for you
    56. 56. Creating documentation• pydoc – built-in; used by help() – generate HTML on any module – kinda plain• epydoc – old, rumored to be dead – produces nicely formatted HTML – easy to install and use• Sphinx framework – “intelligent and beautiful documentation” – all the cool kids are using it (docs.python.org) – more involved to setup and use
    57. 57. Branching out
    58. 58. Installing packages
    59. 59. Installing packages (on Windows)• Windows executables• Python eggs – .zip file with metadata, renamed .egg – distributes code as a bundle – need easy_install• pip – tool for installing and managing Python packages – replacement for easy_install
    60. 60. pipC:pip search “kml”C:pip install BeautifulSoupC:pip install –upgrade pykmlC:pip uninstall BeautifulSoup• can take care of dependencies for you• uninstallation!• install via easy_install, ironically
    61. 61. virtualenv• a tool to create isolated Python environments• manage dependencies on a per-project basis, rather than globally installing• test modules without installing into site- packages• avoid unintentional upgrades
    62. 62. virtualenv• install via pip, easy_install, or by C:python virtualenv.py• create the env C:dir virtualenv <env>• activate the env C:dir<env>Scripts activate• use the env (<env>) C:dir<env>Scriptspython >>>
    63. 63. virtualenv• installs Python where you tell it, modifies system path to point there – good only while the env is activated• use yolk to list installed packages in env (test) C:dir> yolk -l• But can this work in ArcMap Python prompt?
    64. 64. virtualenv• YES, with a little work...>>>execfile(rC:<env>Scriptsactivate_this.py, {__file__:• tells ArcMap to use Python interpreter inrC:<env>Scriptsactivate_this.py}) our virtualenv – kill ArcMap, back to using default interpreter
    65. 65. The web• Infinite source of information• Right-click and “Save as” is so lame (and too much work)• Python can help you exploit the web – ftplib, http (urllib), mechanize, scraping (Beautiful Soup), send email (smtplib)
    66. 66. Fetching data• Built-in libraries for ftp and http• ftplib – log in, nav to directory, retrieve files• urllib/urllib2 – pass in the url you want, get it back• wget – GNU commandline tool – Can call with os.system()
    67. 67. Fetching dataimport urlliburllib.urlretrieve("http://www.fhwa.dot.gov/bridge/nbi/2011/RI11.txt", "C:/temp/RI11.txt")
    68. 68. Scraping• Scrape data from a web page• Well-structured content is a HUGE help, as is valid markup, which isn’t always there• BeautifulSoup 3rd party module – Built in methods and regex’s help out – Great for getting at tables of data
    69. 69. Scraping addresseshttp://www.phillypal.com/pal_locations.php
    70. 70. Scraping addressesimport BeautifulSoup as bsimport urllib2url = "http://www.phillypal.com/pal_locations.php"# Open the URLresponse = urllib2.urlopen(url)# Slurp all the HTML code into memoryhtml = response.read()# Feed it into BS parsersoup = bs.BeautifulSoup(html)# Find all the table cells whose width=37%addresses = soup.findAll("td", {"width":"37%"})print len(addresses)for address in addresses: # Print out just the text print address.find(text=True)
    71. 71. Scraping addresses 1845 N. 23rd Street, 19121 3301 Tasker Street, 19145 5801 Media Street, 19131 250 S. 63rd Street, 19139 732 N. 17th Street, 19130 631 Snyder Avenue, 19148 6901 Rising Sun Avenue, 19111 851 E. Tioga Street, 19134 720 W. Cumberland St., 19133 3890 N. 10th Street, 19140 4550 Haverford Avenue, 19139 1100 W. Rockland St., 19141 1500 W. Ontario Street, 19140 2423 N. 27th Street, 19132 1267 E. Cheltenham Ave., 24 5330 Germantown Ave., 19144 1599 Wharton Street, 19146 4253 Frankford Avenue, 19124 2524 E. Clearfield St., 19134 6300 Garnet Street, 19126 5900 Elmwood Street, 19143 4301 Wayne Avenue, 19140 4401 Aldine Street, 19136 4614 Woodland Avenue, 19143 4419 Comly Street, 19135 2251 N. 54th Street, 19131
    72. 72. Emailing• smtp built-in library• best if you have IP of your email server• port blocking can be an issue import smtplib server = smtplib.SMTP(email_server_ip) msg = „All TPS reports need new cover sheets‟ server.sendmail(from@me.com, to@you.com, msg) server.quit()• there’s always Gmail too…
    73. 73. Files• built in open function – slurp entire file into memory – OK except for huge files data = open(file).read().splitlines()• iterate over the lines for line in data: do something• CSV module reader = csv.reader(open(C:/file.csv,rb)) for line in reader: do something
    74. 74. Excel• love, hate, love• many modules out there – xlrd (read) / xlwt (write) – only .xls – openPyXL – read/write .xlsx• uses – Push text data to Excel file – Push featureclass data to Excel programmatically – Read someone else’s “database”
    75. 75. Reading Excelimport xlrd# Open the workbookwb = xlrd.open_workbook(Employees.xls)wb.sheet_names()# Get first sheetsh=wb.sheet_by_index(0)# Print out the rowsfor row in range(sh.nrows): print sh.row_values(row)# Get a single cellcell_b2 = sh.cell(1,1).valueprint "n", cell_b2
    76. 76. # Write an XLS file with a single worksheet, containing # a heading row and some rows of data. import xlwt import datetime import bs_scrape as bs import nbi_data_processing as nbi ezxf = xlwt.easyxf def write_xls(file_name, sheet_name, headings, data, heading_xf, data_xfs): book = xlwt.Workbook() sheet = book.add_sheet(sheet_name) rowx = 0 for colx, value in enumerate(headings):Writing Excel sheet.write(rowx, colx, value, heading_xf) sheet.set_panes_frozen(True) # frozen headings instead of split panes sheet.set_horz_split_pos(rowx+1) # in general, freeze after last heading row sheet.set_remove_splits(True) # if user does unfreeze, don"t leave a split there for row in data: rowx += 1 for colx, value in enumerate(row): sheet.write(rowx, colx, value, data_xfs[colx]) book.save(file_name) if __name__ == "__main__": import sys files = ["RI","HI"] all_data = [] stateDict = bs.FetchFipsCodes( ) for f in files: k = nbi.ParseNbiFile(C:/student/inputs/ + f + 11.txt, stateDict ) all_data.extend(k) hdngs = ["Structure","State","Facility carried","Lat","Lon","Year built"] kinds = "text text text double double yr".split() data = [] for each_row in all_data: data.extend([each_row]) # Format the headers heading_xf = ezxf("font: bold on; align: wrap on, vert centre, horiz center") # Set the data type formats kind_to_xf_map = { "date": ezxf(num_format_str="yyyy-mm-dd"), "int": ezxf(num_format_str="#,##0"), "money": ezxf("font: italic on; pattern: pattern solid, fore-colour grey25", num_format_str="$#,##0.00"), "price": ezxf(num_format_str="#0.000000"), "double":ezxf(num_format_str="00.00000"), "text": ezxf(), "yr": ezxf(num_format_str="0000") } data_xfs = [kind_to_xf_map[k] for k in kinds] write_xls("NBI_Data_To_Excel.xls", "NBI", hdngs, data, heading_xf, data_xfs)
    77. 77. Writing Excel
    78. 78. Databases• You can connect to pretty much ANY database• Is there one true solution??• pyodbc – Access, SQL Server, MySQL• Oracle – cx_Oracle• Others – pymssql, _mssql, MySQLdb• Execute SQL statements through a connection conn = library.connect(driver/user/pwd) cursor = conn.cursor() for row in cursor.execute(sql) do something
    79. 79. Resources - FREE• Dive into Python• Python Cookbook• Think Python• Python docs• gis.stackexchange.com• Google is your friend (as always)• Python community is HUGE and GIVING
    80. 80. Conferences• pyArkansas – annually in Conway – pyar2 list on python.org• PyCon – THE national US Python conference• FOSS4G – international open source for GIS• ESRI Developer Summit – major dork-fest, but great learning opportunity and Palm Springs in March
    81. 81. IDEs and editors• Wing – different license levels, good people• PyScripter – open source, code completion• Komodo – free version also available• Notepad2 – ole’ standby editor• Notepad++ - people swear by it• PythonWin – another standby, but barebones• …dozens (at least) more editors out there…
    82. 82. More reading• http://www.voidspace.org.uk/python/articles/ OOP.shtml - great OOP article (which I used a a lot)
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×