Managing Hydrodynamic Models with PHP Sydney Open Source Developers’ Conference 3 rd December 2008

Talk Outline PHP class library for spatial data manipulation Windows environment tweaked to facilitate usage of the library from the command line PHP used to manipulate flood models and present results MySQL table structure developed to store and manage model input and output Visualisations of flood model results can be used to summarise large and complex data sets

PHP class library for spatial data manipulation

Windows environment tweaked to facilitate usage of the library from the command line

PHP used to manipulate flood

models and present results

MySQL table structure developed to store and manage model input and output

Visualisations of flood model results can be used to summarise large and complex data sets

PHP Class Library Developed over the last 7 years as a problem solving tool Used to manipulate, transform and analyse spatial data Stores the data structures in a MySQL database PHP provides the “glue” between numerous small command line utilities. Used to produce visualisations of flood modelling results

Developed over the last 7 years as a problem solving tool

Used to manipulate, transform and analyse spatial data

Stores the data structures in a MySQL database

PHP provides the “glue” between numerous small command line utilities.

Used to produce visualisations of flood modelling results

PHP Environment Windows network with 75 users PHP class library must be available to approximately 6 users Scripts developed to perform common file manipulation tasks Single PHP installation on a network drive to ensure a uniform operating environment All users have access to the same class libraries

Windows network with 75 users

PHP class library must be available to approximately 6 users

Scripts developed to perform common file manipulation tasks

Single PHP installation on a network drive to ensure a uniform operating environment

All users have access to the same class libraries

Benefits/Disadvantages of using PHP/MySQL Benefits Rapid development time Readily interacts with text files and MySQL Open Source Can generate images using GD extension Disadvantages Slow run times Memory hungry Frequent minor changes to PHP architecture Lack of pre-existing geometry and meshing libraries MySQL Spatial support not mature

Benefits

Rapid development time

Readily interacts with text files and MySQL

Open Source

Can generate images using GD extension

Disadvantages

Slow run times

Memory hungry

Frequent minor changes to PHP architecture

Lack of pre-existing geometry and meshing libraries

MySQL Spatial support not mature

Using PHP From the Command Line Meshing class library has no GUI Scripts run from the command line Scripts can be run interactively (php –a) Interactive use of PHP allows greater flexibility when composing scripts Requires the user to be familiar with the classes and methods Several network “tweaks” are rolled out in the login script to make command line use of PHP easier

Meshing class library has no GUI

Scripts run from the command line

Scripts can be run interactively (php –a)

Interactive use of PHP allows greater flexibility when composing scripts

Requires the user to be familiar with the classes and methods

Several network “tweaks” are rolled out in the login script to make command line use of PHP easier

“ Command Prompt Here” 1.Navigate in your Registry to HKEY_LOCAL_MACHINE/Software/Classes/Folder/Shell and create a key called "Command Prompt" without the quotes. 2. Set the default string to whatever text you want to appear in the right-click menu. 3.Create a new key within your newly created command prompt named "command," and set the default string to cmd.exe /k pushd %1 4.The changes should take place immediately. Right click a folder and your new menu item should appear.

1.Navigate in your Registry to

HKEY_LOCAL_MACHINE/Software/Classes/Folder/Shell

and create a key called "Command Prompt" without the quotes.

2. Set the default string to whatever text you want to appear in the right-click menu.

3.Create a new key within your newly created command prompt named "command," and set the default string to

cmd.exe /k pushd %1

4.The changes should take place immediately. Right click a folder and your new menu item should appear.

Example PHP Command Line Session

Example PHP Script

Flood Modelling Process

cli_progress_bar

Using class cli_progress_bar <? include &quot;meshlab.php&quot;; $numsteps=100000; $c=new cli_progressbar(100000,”Testing the progress bar”); for($i=1;$i<=100000;$i++){ $c->update($i); } $c->finish(); ?>

<?

include &quot;meshlab.php&quot;;

$numsteps=100000;

$c=new cli_progressbar(100000,”Testing the progress bar”);

for($i=1;$i<=100000;$i++){

$c->update($i);

}

$c->finish();

?>

Commonly Used Tools in %PATH% PHP Library makes extensive use of the exec() command Commonly used tools are kept in a network folder which is included in the users PATH variable Allows small command line utilities to be accessible to the PHP script Allows a modular approach to the software Facilitates the use of custom written helper utilities

PHP Library makes extensive use of the exec() command

Commonly used tools are kept in a network folder which is included in the users PATH variable

Allows small command line utilities to be accessible to the PHP script

Allows a modular approach to the software

Facilitates the use of custom written helper utilities

Flood Modelling Challenges Numerous applications used during the modelling process Each application uses a different file format for storing meshes/points/other spatial information Large volumes of data generated during the process Numerous file formats used throughout the process

Numerous applications used during the modelling process

Each application uses a different file format for storing meshes/points/other spatial information

Large volumes of data generated during the process

Numerous file formats used throughout the process

Available Data Airborne Laser Scanning (ALS) ALS produces hundreds of millions of points that describe the terrain surface Geo-rectified Air Photo Subsidence predictions Flood model output

Airborne Laser Scanning (ALS)

ALS produces hundreds of millions of points that describe the terrain surface

Geo-rectified Air Photo

Subsidence predictions

Flood model output

Air Photo

Levels of Detail in Terrain Models

Initial 2D Finite Element Mesh

PHP Classes

PHP Library Input/Output Formats

Flood Modelling Process

2 Dimensional Hydrodynamic Models We use a 2 dimensional finite element mesh to represent the floodplain Balance between mesh accuracy and number of elements Elements are only required where flooding occurs Need to cull points that are above the flood level Need to reduce the level of detail in the remaining points

We use a 2 dimensional finite element mesh to represent the floodplain

Balance between mesh accuracy and number of elements

Elements are only required where flooding occurs

Need to cull points that are above the flood level

Need to reduce the level of detail in the remaining points

Finite Element Mesh Generation Require less than 50000 elements with current computing technology Resulting mesh must accurately represent the hydrodynamic features of the terrain Quadric Error Metrics used to reduce mesh from ~1million to ~50000 elements (QSlim Application) Results in “variable density” mesh Good technique because it preserves volume Important because this relates to the hydraulic conveyance of the topography qslim_example.bat 

Require less than 50000 elements with current computing technology

Resulting mesh must accurately represent the hydrodynamic features of the terrain

Quadric Error Metrics used to reduce mesh from ~1million to ~50000 elements (QSlim Application)

Results in “variable density” mesh

Good technique because it preserves volume

Important because this relates to the hydraulic conveyance of the topography

qslim_example.bat 

Data Flow and Storage

Example PHP Script

Vector vs Raster Vector Irregularly spaced data Eg: triangulated meshes Finite element models Triangulated laser scan data Raster Images Heightfields

Vector

Irregularly spaced data

Eg: triangulated meshes

Finite element models

Triangulated laser scan data

Raster

Images

Heightfields

2D Interpolation

3D Interpolation

Vector to Raster Conversion Depth Animation

Rasterised Data Triangulation across each triangle or element allows us to calculate the following values at each pixel ground height water surface elevation water velocity RGB colour ground slope Some of these values vary with time allowinf the visualisation to be animated

Triangulation across each triangle or element allows us to calculate the following values at each pixel

ground height

water surface elevation

water velocity

RGB colour

ground slope

Some of these values vary with time allowinf the visualisation to be animated

Raster Database Structure

Terrain Model Properties - Pixel 1207,535

Design Storm Flood Depths Depth Animation

Design Storm Velocities

2D Animation Example

Databases

Indexing Structures - Nodes

Indexing Structures - Elements

Indexing Structures

Indexing Structures - Edges

Indexing Structures - Fulledges

Cross Sections Through an Irregular Mesh

Selecting a cross section from the database Use of the “fulledges” indexing structure allows more complex queries to be performed For example the following query will pull out a cross section across the mesh along a particular “Y” value SELECT stored_fulledges.meshID, stored_fulledges.element1, stored_fulledges.element2, stored_nodes.x, stored_nodes.y, stored_nodes.z, stored_nodes_1.x, stored_nodes_1.y, stored_nodes_1.z FROM (stored_fulledges LEFT JOIN stored_nodes FORCE INDEX (PRIMARY) ON stored_fulledges.node1 = stored_nodes.nodeID ) LEFT JOIN stored_nodes AS stored_nodes_1 FORCE INDEX (PRIMARY) ON stored_fulledges.node2 = stored_nodes_1.nodeID WHERE ( (stored_fulledges.meshID = $meshID) AND ( stored_nodes.y > $y ) AND ( stored_nodes_1.y < $y) ) OR ( (stored_fulledges.meshID = $meshID) AND (stored_nodes.y < $y) AND ( stored_nodes_1.y > $y) ) ORDER BY stored_nodes.x, stored_nodes_1.x;

Use of the “fulledges” indexing structure allows more complex queries to be performed

For example the following query will pull out a cross section across the mesh along a particular “Y” value

SELECT stored_fulledges.meshID, stored_fulledges.element1, stored_fulledges.element2, stored_nodes.x, stored_nodes.y, stored_nodes.z, stored_nodes_1.x, stored_nodes_1.y, stored_nodes_1.z

FROM

(stored_fulledges LEFT JOIN stored_nodes FORCE INDEX (PRIMARY) ON stored_fulledges.node1 = stored_nodes.nodeID )

LEFT JOIN

stored_nodes AS stored_nodes_1 FORCE INDEX (PRIMARY) ON stored_fulledges.node2 = stored_nodes_1.nodeID

WHERE (

(stored_fulledges.meshID = $meshID) AND ( stored_nodes.y > $y ) AND ( stored_nodes_1.y < $y)

) OR (

(stored_fulledges.meshID = $meshID) AND (stored_nodes.y < $y) AND ( stored_nodes_1.y > $y)

)

ORDER BY stored_nodes.x, stored_nodes_1.x;

Levels of Detail

Area of Interest - 20000 Element Mesh

Comparison Through Cross Sections Initial mesh approximately represents the terrain More detail required for smaller channels

Initial mesh approximately represents the terrain

More detail required for smaller channels

Comparison Through Cross Sections Smaller mesh with 20000 elements much better represents the cross section More detailed analysis of the flow is possible

Smaller mesh with 20000 elements much better represents the cross section

More detailed analysis of the flow is possible

Level of Detail in the Finite Element Mesh

Mesh Validation 3D visualisations can assist in validation of the model Allows checking to ensure mesh has enough detail Integrity of channels can be visually checked Show qvis model (ground_points_20k.bat)

3D visualisations can assist in validation of the model

Allows checking to ensure mesh has enough detail

Integrity of channels can be visually checked

Show qvis model (ground_points_20k.bat)

Mesh Generation Tradeoffs Why would we need multiple models of the same region at different levels of detail? Development time vs accuracy Mesh detail vs accuracy Model Run Time vs Number of Elements Disk Storage Consider the accuracy of input data

Why would we need multiple models of the same region at different levels of detail?

Development time vs accuracy

Mesh detail vs accuracy

Model Run Time vs Number of Elements

Disk Storage

Consider the accuracy of input data

Other Applications Level of detail terrain modelling can be used for many applications Air modelling Noise modelling Identification of Vegetation communities Visual impact studies Landform analysis (e.g. slopes, x-sections) Rainfall intensity (Animation ->) …

Level of detail terrain modelling can be used for many applications

Air modelling

Noise modelling

Identification of Vegetation communities

Visual impact studies

Landform analysis (e.g. slopes, x-sections)

Rainfall intensity (Animation ->)

…

Visualising Flood Model Output in 3D Flood model output can be rendered in 3D using Povray open source rendering program Time varying data can be animated Visualising the data in 3 dimensions can summarise complex datasets Model output can be more accessible in this format

Flood model output can be rendered in 3D using Povray open source rendering program

Time varying data can be animated

Visualising the data in 3 dimensions can summarise complex datasets

Model output can be more accessible in this format

Air Photo

RMA Mesh

Thematic Representation of Maximum Depth

Heightfield

Using Povray to Create a 3D Render

Air Photo

DTM – Ground Returns

DTM – Non-Ground Returns

100 Year ARI Depth at Time 79500 seconds

Heightfield

Flood Model Results Draped on Heightfield

Rainfall Intensity Animation

Links View some of the animations from the presentation at the following URL: http://au.youtube.com/view_play_list?p=B8514B561CF5E613 http://au.youtube.com/watch?v=Z76_Wbd2cj0

View some of the animations from the presentation at the following URL:

http://au.youtube.com/view_play_list?p=B8514B561CF5E613

http://au.youtube.com/watch?v=Z76_Wbd2cj0