1. Department of Civil, Architectural & Environmental Engineering 1 June 2, 2011 Development of a Hydrologic Community Modeling System Using a Workflow Engine Committee BO LU Dr. Michael Piasecki Drexel University Dr. Jonathan Goodall Dr. Franco Montalto Dr. Mira Olson Dr. Ilya Zaslavsky 6/2/2011

2. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Objective: Develop a Hydrologic Community Modeling System(HCMS) that allows constructing seamlessly integrated hydrologic models with swappable and portable modules. Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

4. Lack of credibility of the algorithms or methods encapsulated in the codes

5. Poor documentation of source codes

6. Lack of “good coding practices”

7. Model Integration

8. Lack of modular model structure

9. Intertwining of user interfaces and computing kernels

10. Incompatible programming languages

11. Data Interoperability

12. Distinct input and output data structures of models

13. Distinct data models undertaken by disparate data sources

15. Facilities: tools that ease the development of component models or the migration of legacy models. Data analysis tools, transformation tools etc.

17. Supporting high-performance computations and provenance capture.

19. Will its run-time performance be affected when migrating a legacy model into the environment?

22. Composing, executing, monitoring and recording workflows

24. Scheduling workflow execution

25. Loading/running workflows from local/remote database

26. Loading/running workflows from local/remote database

27. Loading/running workflows from local/remote database

28. Running multiple workflows on different nodes of a server clusterTRIDENT SQL DATABASE 6/2/2011

29. Department of Civil, Architectural & Environmental Engineering 7 Why use TRIDENT in hydrologic modeling? Composing workflows with swappable activities via the drag-and-drop manner on a GUI. Flexible Model Setup Allowing automatic and holistic execution without any external intervenes, or alternatively, interactive execution with the control of users. Interactive/Non-interactive Execution High-performance Computing Allowing parallel or concurrent execution, distributed computations in the GRID environment. Recording who, how, what and which resources are used in a workflow, and the derivation flow of data products. It ensures repeatability of model executions. Provenance Capture Easy to Share Sharing workflow through publication mechanismsor repositories. 6/2/2011

30. 6/2/2011 Department of Civil, Architectural & Environmental Engineering Introduction of the libraries of HCMS Data Access Library Data Processing Library Hydrologic Model Library Post-Anaylysis & Utilities Library 8

31. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 9 1.Data Access Library Data Sources: Retrieving data from following data sources using SOAP/FTP protocols .

33. NLCD: 30m * 30m, GeoTIFF[Activity 1] — Access NED or NLCD data within a specified area via Application Services. [Activity 2] — Decompress downloaded data files.

35. Temperature, Precipitation, Long wave/Short wave radiation, Pressure, Vertical/Horizontal wind speed etc.[Activity 1] — Download hourly data files(GRIB) from NLDAS-2 data server. ftp://hydro1.sci.gsfc.nasa.gov/data/s4pa/NLDAS/NLDAS_FORA0125_H.002/ [Activity 2] — Make a choice of fields from a given field list, the activity then extracts data of selected fields from the downloaded data files via a decoder “WGRIB”. [Activity 3] — Cut gridded data set within a specified geospatial extent. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 11

38. It facilitates retrieving hydrologic and meteorological observation time series data from a central metadata catalogue (HISCentral located at the San Diego Supercomputer Center) which holds the richest metadata information in the world for water data. Variable Name (e.g. precipitation) Service ID (optional) Geographical Extent (watershed boundary or latitude/longitude ) Temporal Extent Get Web Services In Box Semantic Checking… Get Sites Web Service IDs Updated Variables Ontology Dictionary Sites Metadata Get Variables Verify Variable Catalog Get Time Series Data Variable Codes WaterML Time Series Data/Metadata Parse Output UI Processing Step Configuration Input Web Service 6/2/2011

39. Get Data via WaterOneFlow Web Services in TRIDENT [Activity 1] — Get web services within a specified geospatial extent. [Activity 2] — Get site and variable metadata based on given variable name. [Activity 3] — Get time series data of given variable within the given geospatial extent. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 14

42. Delineate watershed/sub-watershed boundary, Generate river network; Create Triangulated Irregular Network(TIN); Process Soil, Land Cover data; Create Hydrologic Response Unit (HRU).

43. Time Series Processing

44. Interpolation/Extrapolation, Unit Conversion.

45. Data processing customized for data sources

46. Aggregate NLDAS-2, MPE gridded data for sub-watersheds.6/2/2011

48. Locate the outlet6/2/2011

51. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 20 Creating Hydrologic Response Unit Step 2: Processing Land Cover Data Step 3: Create HRU

52. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 21 Processing Time Series Data

54. For NLDAS-2 gridded data

55. For MPE gridded data6/2/2011

57. A physically based, semi-distributed watershed model that simulates hydrologic fluxes.

58. The VB version converted from 9502 FORTRAN version is migrated into the following workflow. [Activity 1] — Compute Topographic Index Histogram for the whole watershed or each sub-basin. [Activity 2] — Compute Area-Distance Histogram for routing flow. [Activity 3] — Interactive activity for inputting/modifying initial condition and parameters. [Activity 4] — TOPMODEL computation kernel. 6/2/2011

64. Encoded in C# and compiled into Dynamic Link Libraries (DLL).

65. Define input/output variables explicitly via a metadata-tagging approach.

66. Define “Execute” function that is invoked by the engine at run time.

67. Programming work

68. Scripting from the ground up.

71. physical computation elements: Sub-basin, River, HRU

73. apply the SWAT workflow to simulate daily runoff hydrographs over a 4 year period ranging from 2005 to 2008.

74. apply the TOPMODEL workflow along with a loosely coupled hydrologic model for the simulation of a flood event.

75. Sub-studies

76. conduct DEM processing via three types of workflows, and subdivide watershed under two schemes.

77. analyze precipitation data accessed from different data sources.

78. estimate potential evapotranpiration using activities encapsulating different approaches. 6/2/2011

80. Workflows: 1)Step by Step workflow, 2)Terrain Processing workflow, 3)Web service based workflow

81. Delineation: 1) 7 sub-basins: 500,000 cells as threshold 2) 33 sub-basins: 100,000 cells as threshold Total Flow Path Sink Filled DEM Flow Direction Flow Accumulation Raw DEM Watershed Grid Stream order Watershed and River Network (.shp) Stream Raster 6/2/2011

83. Merged Soil Groups

84. HRUS-- 7 sub-basin watershed contains 23 HRUs -- 33 sub-basin watershed contains 114 HRUs 6/2/2011

86. The NLDAS and NWIS precipitation data exhibit a good correlation.

87. The NLDAS precipitation data are adopted in the following modeling.

88. Temperature, Solar radiation, wind speed, pressure are accessed from NLDAS-26/2/2011

91. The estimates of Penman-Monteith method are adopted in the following modeling. 6/2/2011

94. Aggregate the water balance of finer sub-basins belonging to each coarse sub-basin.

95. The water balance accumulated from that of finer sub-basins is close to the one of corresponding coarse sub-basin. 6/2/2011

98. Inputs----NLDAS-2 hourly precipitation

100. In general, it is remarkably straightforward to build up workflows in the HCMS for hydrologic modeling purposes.

101. It can save time and effort through the automated execution that the workflow sequences afford.

102. With the nationwide data coverage of incorporated data sources, the HCMS can be applied to anywhere in the US.

103. Performance of HCMS

106. TRIDENT workflow system provides a platform for designing the HCMS and for assembling hydrologic models as workflow sequences.

107. The HCMS was tested by carrying out several typical hydrologic modeling studies over Schuylkill watershed. It is proved to be used quite well as a modeling platform. While it is not computational cost free due to the middle ware layer, the additional time consumption is “affordable”, especially in the lengthy data preparation arena. 6/2/2011

108. Department of Civil, Architectural & Environmental Engineering 40 Future Work Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011