Download as PDF, PPTX
![PYTHON IN CIVIL ENGINEERING
GRID EXAMPLE - MANIPULATION
grid1, grid2, grid3 = make_test_grids()
fig = (
grid2.merge(grid3, how='vert', where='-', shift=2)
.merge(grid1, how='horiz', where='-', shift=11)
.refine(10, axis=1, n_points=4)
.refine(13, axis=0, n_points=2)
.transform(lambda x: x*5 + 2)
.update_cell_mask()
.plotCells(ax=ax, cell_kws=dict(facecolor=palette[2], alpha=0.75))
)](https://image.slidesharecdn.com/ceeinpythonpypdx-160325141451/85/Python-in-Civil-Environmental-Engineering-64-320.jpg)
More Related Content
Similar to Python in Civil/Environmental Engineering
Python in Civil/Environmental Engineering
- 1. PYTHON IN CIVILAND ENVIRONMENTAL ENGINEERING PAUL HOBSON GEOSYNTEC CONSULTANTS PDX PYTHON 2016-03-24
- 2. PYTHON IN CIVILENGINEERING HEALTH AND SAFETY ▸ Nearest exit in case of emergency ▸ Ergonomics ▸ Right angles ▸ Head level ▸ Take breaks, look and move around
- 3. PYTHON IN CIVILENGINEERING I’M AN ENGINEER (I STUDIED MUD) ▸ What is Civil Engineering? ▸ Structural ▸ Transportation ▸ Construction Management ▸ Geotechnical ▸ Environmental ▸ Water and Natural Resources
- 4. PYTHON IN CIVILENGINEERING I’M AN ENGINEER (I STUDIED MUD) ▸ What is Civil Engineering? ▸ Structural ▸ Transportation ▸ Construction Management ▸ Geotechnical ▸ Environmental ▸ Water and Natural Resources Thesis Title: Rheologic and Flume Erosion Characteristics of Georgia Sediments at Bridge Piers
- 5. PYTHON IN CIVILENGINEERING I’M AN ENGINEER (I STUDIED MUD) ▸ What is Civil Engineering? ▸ Structural ▸ Transportation ▸ Construction Management ▸ Geotechnical ▸ Environmental ▸ Water and Natural Resources Thesis Title: Rheologic and Flume Erosion Characteristics of Georgia Sediments at Bridge Piers
- 6. PYTHON IN CIVILENGINEERING I WORK AS AN ENVIRONMENTAL CONSULTANT ▸ Help industry with compliance and remediation ▸ Litigation support for allocation of environmental damages ▸ Grant-funded research with NCHRP, FHWA, WERF
- 7. PYTHON IN CIVILENGINEERING I WORK AS AN ENVIRONMENTAL CONSULTANT
- 8. PYTHON IN CIVILENGINEERING I WORK AS AN ENVIRONMENTAL CONSULTANT ▸ Build spatial databases of contaminant concentrations
- 9. PYTHON IN CIVILENGINEERING I WORK AS AN ENVIRONMENTAL CONSULTANT ▸ Build spatial databases of contaminant concentrations ▸ Scrape the web (brute force) to meteorologic and hydrologic data
- 10. PYTHON IN CIVILENGINEERING I WORK AS AN ENVIRONMENTAL CONSULTANT ▸ Build spatial databases of contaminant concentrations ▸ Scrape the web (brute force) to meteorologic and hydrologic data ▸ Statistical analysis of pollution mitigation and remediation systems
- 11. PYTHON IN CIVILENGINEERING I WORK AS AN ENVIRONMENTAL CONSULTANT ▸ Build spatial databases of contaminant concentrations ▸ Scrape the web (brute force) to meteorologic and hydrologic data ▸ Statistical analysis of pollution mitigation and remediation systems ▸ Build inputs for, run, and analyze output from vetted numerical models
- 12. PYTHON IN CIVILENGINEERING I WORK AS AN ENVIRONMENTAL CONSULTANT ▸ Build spatial databases of contaminant concentrations ▸ Scrape the web (brute force) to meteorologic and hydrologic data ▸ Statistical analysis of pollution mitigation and remediation systems ▸ Build inputs for, run, and analyze output from vetted numerical models ▸ Build tools to help my colleagues do the same
- 13. PYTHON IN CIVILENGINEERING SUMMARY FOR THE REST OF THE TALK ▸ My take on the general state of the practice in engineering consulting ▸ Characteristic challenges of environmental consulting ▸ Tools I like to use ▸ Tools I have built over the course of my career
- 14. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DATABASES
- 15. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DATABASES ▸ Overwhelming everything is in MS Access
- 16. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DATABASES ▸ Overwhelming everything is in MS Access ▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together
- 17. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DATABASES ▸ Overwhelming everything is in MS Access ▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together ▸ Last Client’s DB
- 18. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DATABASES ▸ Overwhelming everything is in MS Access ▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together ▸ Last Client’s DB ▸ 110 tables
- 19. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DATABASES ▸ Overwhelming everything is in MS Access ▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together ▸ Last Client’s DB ▸ 110 tables ▸ 200 saved insert/create/update
- 20. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DATABASES ▸ Overwhelming everything is in MS Access ▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together ▸ Last Client’s DB ▸ 110 tables ▸ 200 saved insert/create/update ▸ 100 save select and pivot queries
- 21. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DATABASES ▸ Overwhelming everything is in MS Access ▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together ▸ Last Client’s DB ▸ 110 tables ▸ 200 saved insert/create/update ▸ 100 save select and pivot queries ▸ 25 queries that deleted stuff
- 22. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DATABASES ▸ Overwhelming everything is in MS Access ▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together ▸ Last Client’s DB ▸ 110 tables ▸ 200 saved insert/create/update ▸ 100 save select and pivot queries ▸ 25 queries that deleted stuff ▸ Everything connected with 40 VBA forms, 15 utility modules (9000 LOC)
- 23. PYTHON IN CIVILENGINEERING
- 24. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - ANALYSIS ▸ Excel is the behemoth that cannot be stopped ▸ Most people manage their data with it ▸ No one follows best practices (www.datacarpentry.org/ spreadsheet-ecology-lesson) ▸ Will be pried from cold dead hands, despite will published statistical errors (resolved, IIRC)
- 25. PYTHON IN CIVILENGINEERING MANNING’S EQUATION
- 26. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
- 27. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS ▸ GUI or cmd-line based numerical models maintained by federal agencies
- 28. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS ▸ GUI or cmd-line based numerical models maintained by federal agencies ▸ SWMM ▸ 1-D urban hydrology ▸ Written in very legible C ▸ Open source ▸ Maintained by EPA, transitioning to UT Austin ▸ Compiles on Linux and Windows
- 29. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS ▸ GUI or cmd-line based numerical models maintained by federal agencies ▸ SWMM ▸ 1-D urban hydrology ▸ Written in very legible C ▸ Open source ▸ Maintained by EPA, transitioning to UT Austin ▸ Compiles on Linux and Windows ▸ HEC-RAS ▸ 2-D River hydraulics ▸ Completely closed source ▸ Guarded over by US Army Corps of Engineers
- 30. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS ▸ GUI or cmd-line based numerical models maintained by federal agencies ▸ SWMM ▸ 1-D urban hydrology ▸ Written in very legible C ▸ Open source ▸ Maintained by EPA, transitioning to UT Austin ▸ Compiles on Linux and Windows ▸ HEC-RAS ▸ 2-D River hydraulics ▸ Completely closed source ▸ Guarded over by US Army Corps of Engineers ▸ EFDC
- 31. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS ▸ GUI or cmd-line based numerical models maintained by federal agencies ▸ SWMM ▸ 1-D urban hydrology ▸ Written in very legible C ▸ Open source ▸ Maintained by EPA, transitioning to UT Austin ▸ Compiles on Linux and Windows ▸ HEC-RAS ▸ 2-D River hydraulics ▸ Completely closed source ▸ Guarded over by US Army Corps of Engineers ▸ EFDC ▸ Spaghetti FORTRAN
- 32. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS ▸ GUI or cmd-line based numerical models maintained by federal agencies ▸ SWMM ▸ 1-D urban hydrology ▸ Written in very legible C ▸ Open source ▸ Maintained by EPA, transitioning to UT Austin ▸ Compiles on Linux and Windows ▸ HEC-RAS ▸ 2-D River hydraulics ▸ Completely closed source ▸ Guarded over by US Army Corps of Engineers ▸ EFDC ▸ Spaghetti FORTRAN ▸ Compiles on Linux but results are garbage
- 33. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS ▸ GUI or cmd-line based numerical models maintained by federal agencies ▸ SWMM ▸ 1-D urban hydrology ▸ Written in very legible C ▸ Open source ▸ Maintained by EPA, transitioning to UT Austin ▸ Compiles on Linux and Windows ▸ HEC-RAS ▸ 2-D River hydraulics ▸ Completely closed source ▸ Guarded over by US Army Corps of Engineers ▸ EFDC ▸ Spaghetti FORTRAN ▸ Compiles on Linux but results are garbage ▸ 3-D river hydraulics, sediment and pollutant fate and transport
- 34. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS ▸ GUI or cmd-line based numerical models maintained by federal agencies ▸ SWMM ▸ 1-D urban hydrology ▸ Written in very legible C ▸ Open source ▸ Maintained by EPA, transitioning to UT Austin ▸ Compiles on Linux and Windows ▸ HEC-RAS ▸ 2-D River hydraulics ▸ Completely closed source ▸ Guarded over by US Army Corps of Engineers ▸ EFDC ▸ Spaghetti FORTRAN ▸ Compiles on Linux but results are garbage ▸ 3-D river hydraulics, sediment and pollutant fate and transport ▸ Very optimized: 1 yr simulation takes ~ 1 day
- 35. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
- 36. PYTHON IN CIVILENGINEERING
- 37. PYTHON IN CIVILENGINEERING STATE OF THE PRACTICE - DOMAIN SPECIFIC TOOLS ▸ All (numerically) solve the St. Venant Equations ▸ Differential equations in time and space ▸ Very difficult to even work with symbolically ▸ Large code bases ▸ Represent a significant intellectual investment from civil/ environmental community
- 38. PYTHON IN CIVILENGINEERING WHERE DOES PYTHON FIT IN ALL OF THIS? ▸ Scrape model inputs from web ▸ Hack input files for batch processing ▸ Move data around between formats ▸ General data and model results analysis ▸ (Rarely) Wrap C/Fortran libraries with ctypes/cython/ numpy
- 39. PYTHON IN CIVILENGINEERING CIVIL/ENVIRONMENTAL CHALLENGES ▸ Arcane input file formats ▸ Left-censored (non-detect) data ▸ Non-parametric statistics ▸ Regression-on-order statistics ▸ Kaplan-Meier ▸ Project management inertia ▸ Disconnect between libraries and project needs ▸ ~5 coworkers (out of ~600) are comfortable with python
- 40. PYTHON IN CIVILENGINEERING TOOLS I USE ▸ numpy/scipy ▸ Fast numeric arrays implemented in C ▸ Specialized scientific functions ▸ Matplotlib/seaborn: generic and statistical 2-D visualizations ▸ pandas/statsmodels: table-like data structures and statistical models ▸ Jupyter: interactive computing via notebooks ▸ conda: sane installation of python packages on Windows and other operating systems
- 41. PYTHON IN CIVILENGINEERING
- 42. PYTHON IN CIVILENGINEERING SCRAPING WEATHER DATA ▸ conda install —channel=phobson cloudside ▸ Python 3 fork of python-metar by Tom Pollard (github.com/tomp/python- metar.git) ▸ ~1000 lines of regex ▸ Cloudside uses python-metar to pull 5-min and hourly data from the FAA ▸ ~1000 lines of regex ▸ 24229KPDX PDX20120101000009801/01/12 00:00:31 5-MIN KPDX 010800Z 31005KT 10SM OVC085 04/02 A3026 -280 85 -1500 290/05 RMK A02 P0005
- 43. PYTHON IN CIVILENGINEERING WEATHER DATA ▸ Simple commands for bulk data download and parsing ▸ Creates a directory structure for each station, source, and stage of processing ▸ searches for existing files before attempting to download or parse ▸ High-level plotting functions
- 44. PYTHON IN CIVILENGINEERING
- 45. PYTHON IN CIVILENGINEERING
- 46. PYTHON IN CIVILENGINEERING WQIO - WATER QUALITY, INFLOW/OUTFLOW
- 47. PYTHON IN CIVILENGINEERING WQIO - WATER QUALITY, INFLOW/OUTFLOW ▸ conda install --channel=phobson wqio
- 48. PYTHON IN CIVILENGINEERING WQIO - WATER QUALITY, INFLOW/OUTFLOW ▸ conda install --channel=phobson wqio ▸ Centered around examining the efficacy of so-called BMPs
- 49. PYTHON IN CIVILENGINEERING WQIO - WATER QUALITY, INFLOW/OUTFLOW ▸ conda install --channel=phobson wqio ▸ Centered around examining the efficacy of so-called BMPs ▸ Handles left-censored data with ROS ▸ No such things as having “zero” pollution ▸ Can only say that things are “less than” the instrument’s precision
- 50. PYTHON IN CIVILENGINEERING WQIO - WATER QUALITY, INFLOW/OUTFLOW ▸ conda install --channel=phobson wqio ▸ Centered around examining the efficacy of so-called BMPs ▸ Handles left-censored data with ROS ▸ No such things as having “zero” pollution ▸ Can only say that things are “less than” the instrument’s precision ▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate confidence intervals around statistics
- 51. PYTHON IN CIVILENGINEERING WQIO - WATER QUALITY, INFLOW/OUTFLOW ▸ conda install --channel=phobson wqio ▸ Centered around examining the efficacy of so-called BMPs ▸ Handles left-censored data with ROS ▸ No such things as having “zero” pollution ▸ Can only say that things are “less than” the instrument’s precision ▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate confidence intervals around statistics ▸ High-level plotting interface built on seaborn
- 52. PYTHON IN CIVILENGINEERING WQIO - WATER QUALITY, INFLOW/OUTFLOW ▸ conda install --channel=phobson wqio ▸ Centered around examining the efficacy of so-called BMPs ▸ Handles left-censored data with ROS ▸ No such things as having “zero” pollution ▸ Can only say that things are “less than” the instrument’s precision ▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate confidence intervals around statistics ▸ High-level plotting interface built on seaborn ▸ Basis for client-specific libraries that I distribute to them through conda
- 53. PYTHON IN CIVILENGINEERING PARAMNORMAL - CONVENTIONALLY PARAMETERIZED STATISTICAL DISTRIBUTIONS ▸ http://phobson.github.io/paramnormal/index.html ▸ conda install --channel=phobson paramnormal ▸ Problem: scipy distributions are incredibly flexible, but also perhaps a bit over generalized ▸ Wanted to create an API that let statisticians of all caliber specify and fit distributions using the parameters they read about in text books. ▸ The activity module provides a simple interface for creating, fitting, and plotting statistical distributions
- 54. PYTHON IN CIVILENGINEERING PARANORMAL: BRIEF EXAMPLES ▸ (super fast demo)
- 55. PYTHON IN CIVILENGINEERING PARAMNORMAL: ENDORSEMENTS
- 56. PYTHON IN CIVILENGINEERING PARAMNORMAL: ENDORSEMENTS
- 57. PYTHON IN CIVILENGINEERING PARAMNORMAL: ENDORSEMENTS
- 58. PYTHON IN CIVILENGINEERING MPL-PROBSCALE ▸ Legit probability scales for matplotlib ▸ Similar to a quantile plot, but expressed as a probability instead of a z-score ▸ Simply import probscale and you’re set import probscale ax1.set_xscale('prob') ax1.set_xlim(left=2, right=98) ax1.set_xlabel('Normal probability scale') ax2.set_xscale('prob', dist=paramnormal.beta(α=3, β=2)) ax2.set_xlim(left=2, right=98) ax2.set_xlabel('Beta probability scale (α=3, β=2)')
- 59. PYTHON IN CIVILENGINEERING MPL-PROBSCALE ▸ Can fit linear regression in probability space ▸ Top-level functions for easy plotting
- 60. PYTHON IN CIVILENGINEERING GENERATING MODEL GRIDS ▸ Multi-corner, curvilinear-orthogonal grids are fairly difficult to compute — lots of very expensive proprietary stuff out there. ▸ Dr. Pavel Sakov (Australian Bureau of Meteorology wrote gridgen-c ▸ Built a types interface with help from: ▸ Dr. Robert Hetland (Texas A&M) ▸ Dr. Richard Signell (USGS) ▸ Mac and Linux binaries available through conda ▸ Docs at: phobson.github.io/pygridgen
- 61. PYTHON IN CIVILENGINEERING GRID EXAMPLE
- 62. PYTHON IN CIVILENGINEERING GRID EXAMPLE
- 63. PYTHON IN CIVILENGINEERING HIGHER LEVEL GRID OPERATIONS ▸ conda install --channel=conda-forge pygridtools ▸ geosyntec.github.io/pygridtools ▸ Interactivity with grid generation parameters ▸ Simple, general manipulations methods (merge, split, refine, transform) ▸ Took inspiration from pandas/xarray/seaborn ▸ File IO with general and GIS data formats via fiona
- 64. PYTHON IN CIVILENGINEERING GRID EXAMPLE - MANIPULATION grid1, grid2, grid3 = make_test_grids() fig = ( grid2.merge(grid3, how='vert', where='-', shift=2) .merge(grid1, how='horiz', where='-', shift=11) .refine(10, axis=1, n_points=4) .refine(13, axis=0, n_points=2) .transform(lambda x: x*5 + 2) .update_cell_mask() .plotCells(ax=ax, cell_kws=dict(facecolor=palette[2], alpha=0.75)) )
- 65. PYTHON IN CIVILENGINEERING GRIDS IN THE REAL WORLD: SULLIVAN CREEK, NE WASHINGTON
- 66. PYTHON IN CIVILENGINEERING GRIDS IN THE REAL WORLD: SULLIVAN CREEK, NE WASHINGTON
- 67. PYTHON IN CIVILENGINEERING PROJECTING MODEL RESULTS ON THE GRID
- 68. PYTHON IN CIVILENGINEERING PROJECTING MODEL RESULTS ON THE GRID
- 69. PYTHON IN CIVILENGINEERING RESOURCES FOR GETTING STARTED AS A MEAT-SPACE ENGINEER ▸ I was lucky enough to get started with numpy and matplotlib while: ▸ I was still very familiar with MATLAB ▸ Both projects were directly trying to take market share away from MATLAB ▸ APIs and efforts have changed since then. ▸ More recently, a new employee started with MATLAB experience from college. We gave them a copy of Python For Data Analysis by Wes McKinney (author of pandas), and were pretty productive with python in a couple of week.
- 70. PYTHON IN CIVILENGINEERING LINK DUMP ▸ Me: ▸ https://twitter.com/pmhobson ▸ https://github.com/phobson ▸ My (documented) Projects ▸ http://geosyntec.github.io/pygridtools/ ▸ http://phobson.github.io/paramnormal/ ▸ http://phobson.github.io/mpl-probscale ▸ https://github.com/phobson/cloudside (docs forthcoming)
- 71.