Bureau of Meteorology Deltares FEWS User Days Hydrology Design Engineering

1. SeqFEWS: A data-centric workflow
manager in the era of Monte-Carlo
2018 AUS FEWS Users Conference
Lindsay Millard, Hydrologist

2. Presentation Outline
• Context – Seqwater’s Dams
• Quick tour of previous work
• Building blocks in FEWS, and
• How to apply it to design workflows
• Key take-aways

3. Acknowledgements
• Terry Malone
• Michel Raymond
• David Pokarier/Steve Wang
• Deltares & Seqwater staff

4. • 26 referable dams
• 4 regulated water supply dams
• 22 unregulated water supply
dams
• Catchment areas range from
10 to 7,000km2
Seqwater’s Dams

5. Question
What is a Dutch word to describe FEWS?
“Datafabriek”
“Data-centric Workflow Manager”
Why not use it for non-forecasting purposes?
It would:
– assist with task efficiency and maximising skillsets
– avoid buying and learning new software
– enable efficient data management of AR&R 2016
– centralise and allow auditing of tailored design
engineering workflows

6. Motivation
• Wrapping together requirements of AR&R 2016
workflows
• Keeping workflows efficient and archived
– python scripts, GIS extraction, etc
• Scenario management
• Auditing and Continual Improvement
• Data sharing/record of project work
• Feed forward into next flood event or project
…
Export
Import
Transform
ations
Gen.Adapter

7. Closing the loop:
Hydrological Cycle & FEWS
Real-Time
Hydrology
Event
Calibration
Design
Hydrology
&
Hydraulic
s
Upgrade
scenario
models
Event
preparedness
Design Engineering
Forecast Engineering

8. Design Hydrology:
21 on-stream Dams:
7 dams > 100km2 catchment
30km2 < 12 dams < 100km2
2 dams <30km2
∴ Critical Duration is short (<24h)
but not short enough (>6h)

9. Catalogue of Catchment Average Rainfall:

10. Project Summary:
• Improved understanding of temporal patterns
• Created a catalogue for different duration and catchment sizes
• Complete analysis: Need for a suite of moderate (6 – 24h) duration
temporal patterns to allow improved distinction between
• PMF and PMP-DF.
• Operationalise the data: Make the information readily available to
Seqwater staff

11. Related work
Examples of completed Standalones
• GateOPS Next Gen – RTC tools module
• AWAP/Silo dataset – Storage of daily rain grids from
1900
• Somerset Physical Model – Transducers to Animation
• Stochastic Storm event database – 60 Synthetic
storms
• AWRA-L Initial Loss modelling
• Monte Carlo Hydrology/Hydraulic Interface ‘Treasury’
Next “FEW” slides used with Permission of Authors

12. Synthetic rainfall events were:
• Re-formatted and imported
into FEWS
• Exported in NetCDF-Grid
format
Stochastic Storm Database
WaterCoach - 2017

13. Seq-FEWS Implementation
• Configure ‘best fit’ equation parameters
• Develop grid and scalar displays
• Integrate initial loss estimates into existing reports*
• Finalise report and procedures
• Operational system for Initial Loss estimate
AWRA-L Analysis - 2016

14. Brisbane River Catchment Flood Study - 2015
14
7Tb of data to harvest

15. FEWS building blocks:
• An application that manages model runs efficiently
• Management of model queue to:
– assist event and scenario runs
– maximise license/hardware utilisation
• Import/Export Timeseries:
– Point, Grid and export self-contained NC.
• Transformations of Timeseries:
– Grid-grid interpolation / 2D Lookup Tables

16. AR&R 2016 Design Event Workflow
Model Build
• Calibrate Events
• Define URBS Parameters
• Storm Hyetographs
Download
• Scrape AR&R Datahub
• Chop up BoM IFD Grids
• Preburst – Initial /Continuing Loss estimation
Design
Ensemble
• Process: Areal Reduction Factors
• Temporal patterns
• Design Scenarios
Post process
Results
• Statistical Analysis & Boxplots
• Critical Duration Best Estimate for each AEP
• Plot and Report Results (1,000s)

17. 17
SeqFEWS and Python
Link websites/models using adaptors
URBS
MIKE11
Tuflow is available in 2018.03 natively
HEC RAS 4.1 adapter
Import  Translate Manipulation  Storage  Export  Publish to Reports
Use General Adapter to execute script.py or batch
.py
[roll your own]
MIKE 1D/Flood requires Iron Python
HEC RAS 5.0.4 ??
GoldSIM / RORB
Machine Learning
Matplotlib/Pandas/Seaborn  Boxplots!

18. SeqFEWS
General
Adapter
.XML
Runtime
parameters
.XML .NC
.CSV
.bat / .py
Model Input
.bat / .py
Model
Results/Log
Module Data
set
GeometryRun files
Input TS Output TS
Grid
Model .exe
ModuleConfig
RegionConfig
WorkflowFiles
Modulesmodel
ModuleDataset model.zip
0d, 1d,
2d T.S.
transformation
export import
Plots

19. Export Rainfall from
FEWS datastore
URBS output
Q h TS
Import Q h into
FEWS datastore
Compute max Q
and h from FEWS
datastore
Max Q h
Importance sampling
TPT process AEP
Calculate for Q & h
Slice for location
AEP frequency
Store AEP Q and h
in FEWS datastore
URBS input

20. Key Takeaways
20
• Variety of different types of models available in FEWS
– All stitched together using the “adapter” concept
– Python is a toolbox to overcome non-standard issues
– Models can be mixed in a single workflow for auditing
• Increasing use of distributed & complex models in workflows
– Issues: speed, database sizes, complexity, …
Keep the design workflow organised and repeatable

21. Tenets of modelling
All Models are wrong
Models are never finished, only abandoned
If you can’t make it perfect, make it adjustable