2013 Tips and Tricks Mashup, From ModelBuilder to Formal Python Code, Step-by-Step by Zeke Houk

783 views
599 views

Published on

ArcGIS ModelBuilder provides the user with a simple way to capture geoprocessing instructions as a Python script. This presentation explores the path to promote that script into formal production level code. Beginning with the geoprocessing foundation exported from ModelBuilder, we augment the code to:
• clarify the command line parameters
• organize the directories
• set the SDE workspace
• insert print commands
• check if feature classes exist
• use try / catch blocks to trap errors
• execute operating system commands
• add logging throughout the code
The end result of these additions is production level code that is easy to maintain and deploy. It is particularly well-suited to run automatically as a Windows Scheduled Task.

This session is intended for users who are getting started with Python, or even just thinking about it.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
783
On SlideShare
0
From Embeds
0
Number of Embeds
25
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

2013 Tips and Tricks Mashup, From ModelBuilder to Formal Python Code, Step-by-Step by Zeke Houk

  1. 1. ModelBuilder to Python: Step-by-Step Zeke Houk GIS Developer
  2. 2. ModelBuilder Model:
  3. 3. Delete Features from simple_conduits:
  4. 4. Pull snapshot from SDE:
  5. 5. Get Rid of Conduit 0:
  6. 6. Dissolve on Conduit Number:
  7. 7. Guido Van Rossum BDFL
  8. 8. Where did the name come from?
  9. 9. Python script:
  10. 10. Clean up names: • >>> # Local variables: • >>> simple_conduits__3_ = R:ARG_ProjectsConduitMapBookDissolvedConduits.gdbsimple_conduits" • >>> dissolved_conduits__3_ = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits" • >>> ARG_CONDUIT__3_ = "Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT" • >>> simple_conduits__4_ = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbsimple_conduits" • >>> N0_Zero_Conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbN0_Zero_Conduits" • >>> dissolved_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits" • >>> simple_conduits__6_ = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbsimple_conduits" >>> simple_conduits_3_, >>> simple_conduits_4_, >>> simple_Conduits_6_ >>> redundant…
  11. 11. Replace with simple_conduits: • >>> # Local variables: • >>> simple_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbsimple_conduits" • >>> dissolved_conduits__3_ = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits" • >>> ARG_CONDUIT__3_ = "Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT" • >>> simple_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbsimple_conduits" • >>> N0_Zero_Conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbN0_Zero_Conduits" • >>> dissolved_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits" • >>> simple_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbsimple_conduits" Global replace simple_conduits For simple_conduits_3_, simple_conduits_4_, and simple_conduits_6_.
  12. 12. Duplicates dropped: >>> # Local variables: >>> simple_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbsimple_conduits" >>> dissolved_conduits__3_ = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits" >>> ARG_CONDUIT__3_ = "Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT" >>> N0_Zero_Conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbN0_Zero_Conduits" >>> dissolved_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits" Global replace simple_conduits For simple_conduits_3_, simple_conduits_4_, and simple_Conduits_6_.
  13. 13. Now fix dissolved_conduits: # Local variables: simple_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbsimple_conduits" dissolved_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits" ARG_CONDUIT__3_ = "Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT" N0_Zero_Conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbN0_Zero_Conduits" Becomes… # Local variables: simple_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbsimple_conduits" dissolved_conduits__3_ = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits" ARG_CONDUIT__3_ = "Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUI N0_Zero_Conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbN0_Zero_Conduits" dissolved_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits"
  14. 14. Use File Geodatabase Name: • # Local variables: • file_gdb = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdb" • simple_conduits = file_gdb + "simple_conduits" • dissolved_conduits = file_gdb + "dissolved_conduits" • N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits" • ARG_CONDUIT = "Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT" Becomes… • # Local variables: • simple_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbsimple_conduits" • dissolved_conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbdissolved_conduits" • ARG_CONDUIT = "Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUI • N0_Zero_Conduits = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdbN0_Zero_Conduits"
  15. 15. Use File Geodatabase Name: file_gdb = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdb" simple_conduits = file_gdb + "simple_conduits" dissolved_conduits = file_gdb + "dissolved_conduits" N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"
  16. 16. Use SDE Connection Name: # Local variables: file_gdb = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdb" simple_conduits = file_gdb + "simple_conduits" dissolved_conduits = file_gdb + "dissolved_conduits" N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits" sde_connection = "Database ConnectionsConnection to DWGIS3.sde" ARG_CONDUIT = sde_connection + "ARG.CONDUIT" Becomes… # Local variables: file_gdb = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdb" simple_conduits = file_gdb + "simple_conduits" dissolved_conduits = file_gdb + "dissolved_conduits" N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits" ARG_CONDUIT = "Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT"
  17. 17. Use SDE Connection Name: sde_connection = "Database ConnectionsConnection to DWGIS3.sde" ARG_CONDUIT = sde_connection + "ARG.CONDUIT"
  18. 18. Cleaned up names: # Local variables: file_gdb = "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdb" simple_conduits = file_gdb + "simple_conduits" dissolved_conduits = file_gdb + "dissolved_conduits" N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits" sde_connection = "Database ConnectionsConnection to DWGIS3.sde" ARG_CONDUIT = sde_connection + "ARG.CONDUIT"
  19. 19. Congratulations!
  20. 20. PyWin: PyWin Python for Windows extensions by Mark Hammond http://sourceforge.net/projects/pywin32/ pywin32-218.win32-py2.7.exe
  21. 21. Process Steps: Delete Features from simple_conduits Append from SDE into simple_conduits, just COND_NUM and WATER_TYPE Delete NO_Zero_Conduits Select from simple_conduits into N0_Zero_Conduits Delete dissolved_conduits Dissolve N0_Zero_Conduits into dissolved_conduits
  22. 22. Fix up Process Steps: • # Process: Append • arcpy.Append_management(ARG_CONDUIT, simple_conduits, "NO_TEST", "COND_NUM "COND_NUM" true true false 2 Short 0 0 ,First,#,Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT,COND_NUM,-1,- 1;WATER_TYPE "WATER_TYPE" true true false 5 Text 0 0 ,First,#,Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT,WATER_TYPE,-1,- 1;Shape_Length "Shape_Length" false true true 8 Double 0 0 ,First,#", "") Becomes… • # Process: Append • arcpy.Append_management( • "'Database ConnectionsConnection to DWGIS3.sde • ARG.DistributionSystemARG.CONDUIT'", simple_conduits, • "NO_TEST", "COND_NUM "COND_NUM" true true false 2 Short 0 0 ,First,#,Database C
  23. 23. Fix up Process Steps: • # Process: Delete • arcpy.Delete_management(N0_Zero_Conduits, "FeatureClass") Becomes… • # Process: Delete • arcpy.Delete_management( • "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdb • NO_Zero_Conduits", • "FeatureClass")
  24. 24. Fix up Process Steps: • # Process: Delete • arcpy.Delete_management(dissolved_conduits, "FeatureClass") Becomes… • # Process: Delete • arcpy.Delete_management( • "R:ARG_ProjectsConduit MapBookDissolvedConduits.gdb • dissolved_conduits", • "FeatureClass")
  25. 25. All Scrubbed Up: # Process: arcpy.DeleteFeatures_management(simple_conduits) arcpy.Delete_management(N0_Zero_Conduits, "FeatureClass") arcpy.Delete_management(dissolved_conduits, "FeatureClass") arcpy.Append_management(ARG_CONDUIT, simple_conduits, "NO_TEST", "COND_NUM "COND_NUM" true true false 2 Short 0 0 ,First,#,Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT,COND_NUM,-1,-1;WATER_TYPE "WATER_TYPE" true true false 5 Text 0 0 ,First,#,Database ConnectionsConnection to DWGIS3.sdeARG.DistributionSystemARG.CONDUIT,WATER_TYPE,-1,-1;Shape_Length "Shape_Length" false true true 8 Double 0 0 ,First,#", "") arcpy.Select_analysis(simple_conduits, N0_Zero_Conduits, ""COND_NUM" <> 0") arcpy.Dissolve_management(N0_Zero_Conduits, dissolved_conduits, "COND_NUM;WATER_TYPE", "", "SINGLE_PART", "DISSOLVE_LINES")
  26. 26. Geek Speak - If: if arcpy.Exists(something): # do something with it if not arcpy.Exists(something): # it does not exist
  27. 27. Avoid Errors: if arcpy.Exists(dissolved_conduits): arcpy.Delete_management(dissolved_conduits, "FeatureClass") Becomes… arcpy.Delete_management(dissolved_conduits, "FeatureClass")
  28. 28. Delete Processes: if arcpy.Exists(N0_Zero_Conduits): arcpy.Delete_management(N0_Zero_Conduits, "FeatureClass") if arcpy.Exists(dissolved_conduits): arcpy.Delete_management(dissolved_conduits, "FeatureClass") if arcpy.Exists(simple_conduits): arcpy.DeleteFeatures_management(simple_conduits) else: print "Missing feature class -> " + simple_conduits
  29. 29. Alternative: >>> from arcpy import env >>> env.overwriteOutput = True >>> env.workspace = “C:/EsriPress/Python/Data/”
  30. 30. Either way, explain: >>> # There feature classes probably already exist. >>> # They are left over from the last time we ran this script. >>> # Deleting the old versions makes room for the new ones.
  31. 31. Geek Speak – Try / Catch: >>> try: >>> # .. To do something that might blow up >>> # and it “throws” an Exception >>> except Exception ex: >>> print ex >>>
  32. 32. Catch Exception: >>> try: >>> arcpy.Append_management(CONDUIT >> except Exception as ex: >>> print ex
  33. 33. Test Exception: >>> try: >>> bogus = 1.0 / 0.0 # forces exception >>> arcpy.Append_management(ARG_CONDUIT, >>> except Exception as ex: >>> print ex
  34. 34. Logging: import logging logfile = "C:/temp/demo.log" logging.basicConfig( level=logging.INFO, filename=logfile) logger = logging.getLogger() logger.info("A swallow beats its wings 43 times every second.") logger.warning("Go away or I shall taunt you a second time.") logger.error("Run away! Run away!") print "Done."
  35. 35. C:tempdemo.log: INFO:A swallow beats its wings 43 times every second. WARNING:Go away or I shall taunt you a second time. ERROR:Run away! Run away!
  36. 36. Logging: logging.info(" checking for " + HYDRANT_BRANCH ) if arcpy.Exists(HYDRANT_BRANCH ): try: arcpy.Append_management(APPEND_HYD_BR, hyd_branch, ...) logging.info(" HYDRANT_BRANCH_shp appended to SDE") except Exception as ex: logging.error(" HYDRANT_BRANCH_shp append FAILED") logging.Exeception(ex)
  37. 37. Recap – Do These 3 Things… 1. Organize variable names 2. Trap Errors 3. Write a log file "First shalt thou take out the Holy Pin. Then shalt thou count to three, no more, no less. Three shall be the number thou shalt count, and the number of the counting shall be three.”
  38. 38. Pythonic: • Explicit is better than implicit. • Simple is better than complex. • Readability counts. The Zen of Python, by Tim Peters, includes:
  39. 39. Geek Speak - Blocks: >>> if arcpy.Exists(simple_conduits): >>> arcpy.DeleteFeatures_management(simple_conduits) >>> # maybe do something else >>> # some additional processing >>> else: >>> print "Missing feature class -> " + simple_conduits
  40. 40. Duck Typing: If it looks like a duck, quacks like a duck, etc. Works with strings, integers, floating point, lists, etc.
  41. 41. Strings: Enclosed in either ‘single’ or “double” quotes. Concatenate using + sign. >>> topic = "GIS" >>> location = "Rockies“ >>> event = topic + " in the " + location >>> print event GIS in the Rockies
  42. 42. Strings: More built-in functions: >>> event = "GIS in the Rockies" >>> print event.lower() >>> print event.upper() >>> print event.title() >>> string_length = len(event) >>> print string_length >>> print event.find("Rockies",0,string_length) gis in the rockies GIS IN THE ROCKIES Gis In The Rockies 18 11
  43. 43. Triple quotes can span multiple lines: “””The Dead Collector: Bring out yer dead. Large Man: Here's one. The Dead Collector: That'll be ninepence. The Dead Body: I'm not dead. The Dead Collector: 'Ere, he says he's not dead. Large Man: Yes he is. The Dead Body: I'm not. The Dead Collector: He isn't. Large Man: Well, he will be soon, he's very ill. The Dead Body: I'm getting better. “””
  44. 44. Lists: >>> python_sketch = ['Dead Parrot', 'Confuse-a-Cat', 'Silly Walks', 'Four Yorkshiremen','Fish Slap']; >>> # notice duck typing >>> print python_sketch [0] >>> print python_sketch [1] >>> python_sketch.append('Limberjack Song') >>> print len(python_sketch) Dead Parrot Confuse-a-Cat 6
  45. 45. For loop: >>> for funny_bit sketch in python_sketch: >>> print funny_bit + ' hahaha‘ Dead Parrot hahaha Confuse-a-Cat hahaha Silly Walks hahaha Four Yorkshiremen hahaha Fish Slap hahaha Limberjack Song hahaha
  46. 46. Command Line Args: >>> userid = arcpy.GetParameterAsText(0) >>> if not userid: >>> print ‘Usage: python.exe my_script.py userid password ‘ >>> pword = arcpy.GetParameterAsText(1) >>> if not pword: >>> print ‘Usage: python.exe my_script.py userid password ‘
  47. 47. Command Line Args: C:> D:python27python.exe my_script.py zeke password Python.exe is located on D:python27 my_script.py is some python code Userid is zeke My Oracle password is at the end
  48. 48. Command Line Args: >>> userid = arcpy.GetParameterAsText(0) >>> pword = arcpy.GetParameterAsText(1)
  49. 49. Oracle connection: connection_string = userid + '/' + pword + '@dwsde' # looks like this -> zeke/password@dwsde import cx_Oracle connection = cx_Oracle.connect(connection_string) cursor = connection.cursor() cursor.execute('select distinct water_type from conduit') for result in cursor: print result
  50. 50. Oracle connection: cursor.execute('select distinct diameter from conduit') allresults = cursor.fetchall() for result in allresults: print result cursor.close() connection.close() print "Done."
  51. 51. Nee
  52. 52. Here is a shrubbery.
  53. 53. Not Habit Forming: But is considered a “Gateway” language, which frequently leads to more nerdy behavior.
  54. 54. Good Book:
  55. 55. Good Book:
  56. 56. GIS Day November 22: See denverwater.org for directions...
  57. 57. Becomes…

×