Presentation by Hélène Boisgontier (Deltares) at the Symposium Models and decision-making in the wake of climate uncertainties, during the Deltares Software Days - Kampala 2023 (DSD-Kampala 2023). Wednesday, 4 October 2023, Kampala, Uganda.
DSD-INT 2023 Hydrology User Days - Presentations - Day 2Deltares
Presentation by several speakers at the Hydrology Suite User Days (Day 2) - wflow and HydroMT, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
DSD-INT 2020 BlueEarth Engine - hydroMT - model builder frameworkDeltares
hydroMT is a model builder framework developed by Deltares to facilitate building and updating hydrological models. It operates on global and local datasets and supports modelers through automating data processing tasks. hydroMT contains reusable methods for processing spatial data and interfaces with hydrological models to build, update, and connect their components. Examples shown include using hydroMT to setup models at different scales and resolutions, build hydrodynamic models quickly, and connect hydrological and water allocation models.
This document summarizes the work done in WP3 of the HABITATS project. The objectives of WP3 were to define data and metadata models for relevant INSPIRE themes in compliance with the INSPIRE directive. Specific tasks included analyzing existing models, defining conceptual models and metadata profiles, and designing data transformation processes. The work contributed to the development of INSPIRE data models. Training and tools were provided to partners to assist with interactive data modeling and transformations. The work demonstrated approaches for harmonizing data from different models into a unified INSPIRE-compliant model.
DSD-SEA 2023 Innovations in hydro modelling software - BrinkmanDeltares
Presentation by JanJaap Brinkman (Deltares - NUSDeltares) at the Seminar Models and decision-making in the wake of climate uncertainties, during the Deltares Software Days South-East Asia 2023. Wednesday, 22 February 2023, Singapore.
DSD-INT 2019 - Global Data Services and Analysis Frameworks - TwigtDeltares
Presentation by Daniel Twigt and Mark Hegnauer, Deltares, at the Delft3D - User Days (Day 1: Hydrology and hydrodynamics), during Delft Software Days - Edition 2019. Monday, 11 November 2019, Delft.
DSD-INT 2023 Deltares Hydrology Suite - An introduction - SlootjesDeltares
Presentation by Nadine Slootjes and Timo Kroon (Deltares, Netherlands) at the Hydrology Suite User Days (Day 1) - Hydrology Suite introduction and River Basin Management software (RIBASIM), during the Delft Software Days - Edition 2023 (DSD-INT 2023). Tuesday, 28 November 2023, Delft.
The document discusses using smartphone and tablet technologies to integrate INSPIRE geospatial data infrastructure. It describes projects like HABITATS that modeled environmental data themes in compliance with INSPIRE. Different use cases for regional and global data are examined. Basic and advanced techniques for transforming data between regional models and INSPIRE are outlined. The document also introduces the HABITATS Reference Laboratory concept for supporting cross-scenario applications and testing local data sharing. It identifies opportunities to make INSPIRE data more accessible through mobile and social applications.
GI2013 ppt kafka&team-inspire in pocketIGN Vorstand
The document discusses integrating INSPIRE geospatial data standards with mobile devices. It describes the HABITATS project which designed environmental data and metadata models compliant with INSPIRE for themes like habitats and species distribution. The benefits of participating in INSPIRE technical working groups are outlined. Various use cases for regional and global data at different scales are examined, along with challenges of cross-border data harmonization. Basic and advanced techniques for transforming data between models are presented, including using SQL, ontologies and taxonomies. The HABITATS Reference Laboratory is introduced as a hub supporting INSPIRE data sharing and testing across pilot applications. Ideas are raised about making INSPIRE data more accessible through mobile and social applications
DSD-INT 2023 Hydrology User Days - Presentations - Day 2Deltares
Presentation by several speakers at the Hydrology Suite User Days (Day 2) - wflow and HydroMT, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
DSD-INT 2020 BlueEarth Engine - hydroMT - model builder frameworkDeltares
hydroMT is a model builder framework developed by Deltares to facilitate building and updating hydrological models. It operates on global and local datasets and supports modelers through automating data processing tasks. hydroMT contains reusable methods for processing spatial data and interfaces with hydrological models to build, update, and connect their components. Examples shown include using hydroMT to setup models at different scales and resolutions, build hydrodynamic models quickly, and connect hydrological and water allocation models.
This document summarizes the work done in WP3 of the HABITATS project. The objectives of WP3 were to define data and metadata models for relevant INSPIRE themes in compliance with the INSPIRE directive. Specific tasks included analyzing existing models, defining conceptual models and metadata profiles, and designing data transformation processes. The work contributed to the development of INSPIRE data models. Training and tools were provided to partners to assist with interactive data modeling and transformations. The work demonstrated approaches for harmonizing data from different models into a unified INSPIRE-compliant model.
DSD-SEA 2023 Innovations in hydro modelling software - BrinkmanDeltares
Presentation by JanJaap Brinkman (Deltares - NUSDeltares) at the Seminar Models and decision-making in the wake of climate uncertainties, during the Deltares Software Days South-East Asia 2023. Wednesday, 22 February 2023, Singapore.
DSD-INT 2019 - Global Data Services and Analysis Frameworks - TwigtDeltares
Presentation by Daniel Twigt and Mark Hegnauer, Deltares, at the Delft3D - User Days (Day 1: Hydrology and hydrodynamics), during Delft Software Days - Edition 2019. Monday, 11 November 2019, Delft.
DSD-INT 2023 Deltares Hydrology Suite - An introduction - SlootjesDeltares
Presentation by Nadine Slootjes and Timo Kroon (Deltares, Netherlands) at the Hydrology Suite User Days (Day 1) - Hydrology Suite introduction and River Basin Management software (RIBASIM), during the Delft Software Days - Edition 2023 (DSD-INT 2023). Tuesday, 28 November 2023, Delft.
The document discusses using smartphone and tablet technologies to integrate INSPIRE geospatial data infrastructure. It describes projects like HABITATS that modeled environmental data themes in compliance with INSPIRE. Different use cases for regional and global data are examined. Basic and advanced techniques for transforming data between regional models and INSPIRE are outlined. The document also introduces the HABITATS Reference Laboratory concept for supporting cross-scenario applications and testing local data sharing. It identifies opportunities to make INSPIRE data more accessible through mobile and social applications.
GI2013 ppt kafka&team-inspire in pocketIGN Vorstand
The document discusses integrating INSPIRE geospatial data standards with mobile devices. It describes the HABITATS project which designed environmental data and metadata models compliant with INSPIRE for themes like habitats and species distribution. The benefits of participating in INSPIRE technical working groups are outlined. Various use cases for regional and global data at different scales are examined, along with challenges of cross-border data harmonization. Basic and advanced techniques for transforming data between models are presented, including using SQL, ontologies and taxonomies. The HABITATS Reference Laboratory is introduced as a hub supporting INSPIRE data sharing and testing across pilot applications. Ideas are raised about making INSPIRE data more accessible through mobile and social applications
This document summarizes a kick-off meeting for the UR3 project, which aims to implement a cloud computing infrastructure for sharing data, algorithms, and high performance computing resources among different teams and communities. It outlines the objectives, tasks, timeline, and involved partners of the UR3 project. It also discusses concepts for the cloud architecture, including virtualization, horizontal and vertical scalability, and the benefits of a cloud model for optimizing resource usage and reducing costs.
DSD-INT 2019 Modelling in DANUBIUS-RI-BellafioreDeltares
This document discusses modelling river-sea systems through the proposed DANUBIUS-RI initiative. It notes that river-sea systems are important due to factors like population concentrations, human activities, and climate change impacts. The DANUBIUS-RI would provide a modelling node to tackle issues in these environments through coordinated modelling efforts across Europe. The modelling node would develop integrated physical, chemical and ecological models to serve researchers and stakeholders, using standardized formats and expertise from various working groups.
First online hangout SC5 - Big Data Europe first pilot-presentation-hangoutBigData_Europe
This document describes a pilot project to support data-intensive climate research. The pilot aims to provide researchers with an intuitive interface to search, download, and dynamically downscale climate model and observational data. It will orchestrate the downscaling process on institutional computational resources while managing data products and lineage. Currently, acquiring and preprocessing climate data for downscaling is an ad-hoc process. The pilot seeks to improve research productivity by facilitating efficient data access, model execution, and reuse of experiments through the Big Data Europe platform. It may help climate impact assessments in other societal challenge areas like energy, food, and agriculture.
Modern tools and techniques can help address challenges in water data management. Water data management platforms use data sharing platforms to integrate data from multiple agencies in a standardized format. These platforms incorporate a hydrological geofabric to establish a single point of truth for water mapping, and use cloud computing to provide scalable access and analysis of large water datasets. For example, a demonstration showed how sensor data, water storage data, and river flow models could be integrated in a sensor cloud to help manage water sharing in a catchment.
The document describes the development of a Hydrologic Community Modeling System (HCMS) using a workflow engine called TRIDENT. The HCMS will allow for modular and integrated hydrologic models with interchangeable components. It will include libraries for data access, processing, hydrologic models, and post-analysis tools. Example applications to the Schuylkill Watershed are provided to demonstrate watershed delineation, hydrologic response unit creation, meteorological data processing, and potential evapotranspiration calculation workflows.
DSD-INT 2019 Global Data Services and Analysis Frameworks-LuijendijkDeltares
Presentation by Arjen Luijendijk, Deltares, at the Delft3D - User Days (Day 2: Hydrodynamics), during Delft Software Days - Edition 2019. Tuesday, 12 November 2019, Delft.
The Consultation Office of Integrated Systems (COINS) is a private consulting firm made up of multi-disciplinary teams of highly qualified engineers. Over the past 15 years, COINS has participated in various applied research projects dealing with systems analysis, information systems design, computer hardware and software, and more. COINS aims to utilize the latest scientific developments and technology to help solve problems facing Egyptian society through education and acting as a catalyst for change within institutions.
Accès ouvert aux données météorologiques d’Environnement CanadaVisionGEOMATIQUE2014
Deux services de données géospatiales offerts par le Service météorologique Canada (Environnement Canada) seront présentés.
Le Datamart du SMC permet de télécharger des données brutes produites par le Service météorologique du Canada (SMC) et rend possible la cueillette automatisée des produits et données météorologiques du SMC. Une grande diversité de données s’y retrouvent, tels les avertissements météo, les prévisions publiques en XML, les données brutes de prévision, les observations météorologiques et bien davantage. Le Datamart du SMC jouit d’une grande popularité avec plus de 12 millions d’accès quotidiennement.
Les services web géospatiaux GeoMet permettent au public d’intégrer dans leurs outils SIG, cartes interactives ou appareil mobile, les données brutes de modèles de prévision numérique du temps et de la mosaïque nord-américaine des radars météo par le biais de deux standards de l’Open Geospatial Consortium : le WMS et le KML. Qu’il s’agisse par exemple de la pluie observée par les radar météo ou les prévisions de précipitation de neige ou une animation des prévisions de températures , les utilisations de ces données sont nombreuses et d’une grande valeur pour de nombreux décideurs.
InfoWorks CS is a comprehensive software for modeling urban wastewater and drainage networks. It allows users to import network data from other modeling programs, perform hydraulic simulations, and assess scenarios. Key features include its stable dynamic wave routing engine, tools for surface runoff modeling, infiltration analysis, and real-time control of pumps. The software also has robust data management and provides excellent visualization of results.
Models Done Better... - UDG2018 - Intertek and DHIStephen Flood
Use of integrator systems (operational data and model management platforms) to enhance model performance and value.
Presented at the CIWEM Urban Drainage Group Annual Conference 2018
Richard Dannatt - Intertek
Steve Flood - DHI
The document discusses big data use cases and requirements. It provides 51 detailed use cases across various domains that generate many terabytes to petabytes of data. It also describes extracting 437 specific requirements from the use cases and analyzing trends. The next steps involve matching requirements to a reference architecture and prioritizing use cases for implementation.
Conference: IEEE 16TH INTERNATIONAL
CONFERENCE ON INDUSTRIAL INFORMATICS
Porto, Portugal – July 18-20, 2018
Title of the paper: A Framework for Data Collection,
Transformation and Processing in industrial Systems
Authors: Umer Iftikhar, Wael M. Mohammed, Borja
Ramis Ferrer and Jose L. Martinez Lastra
Experimental transformation of ABS data into Data Cube Vocabulary (DCV) form...Alistair Hamilton
Presentation by Al Hamilton and Cody Johnson to Canberra Semantic Web Meetup Group on why producers of official statistics are interested in semantic web community (including Linked Open Data) and outlining experimental work by Cody Johnson on transforming selected Population Census data released by the ABS in SDMX-ML to RDF Data Cube Vocabulary format.
1. The document provides an introduction to computer-based flood risk modelling in the UK and overseas. It discusses the history and types of modelling, including 1D, 2D and 3D approaches.
2. Types of modelling include steady state and unsteady state approaches. Steady state does not consider variations over time while unsteady state does.
3. Models are used for a variety of functions like system operations, design works, assessing development impacts, and detecting triggered conditions. They help inform risk-based decision making.
Big Data HPC Convergence and a bunch of other thingsGeoffrey Fox
This talk supports the Ph.D. in Computational & Data Enabled Science & Engineering at Jackson State University. It describes related educational activities at Indiana University, the Big Data phenomena, jobs and HPC and Big Data computations. It then describes how HPC and Big Data can be converged into a single theme.
PERICLES Workflow for the automated updating of Digital Ecosystem Models with...PERICLES_FP7
This presentation was delivered by Anna Eggers (State and University Library Göttingen) and Fabio Corubolo (University of Liverpool) at PERICLES final project conference 'Acting on Change: New Approaches and Future Practices in LTDP' (Wellcome Collection Conference Centre, London, 30 Nov -1 Dec 2016).
This 'PERICLES in practice' session aimed at demonstrating how the PET and the EcoBuilder tools can be used to generate information to populate Digital Ecosystem Model (entities) and how the model can be fed into the Entity Registry Model Repository (ERMR). This session showed how the PERICLES tools can contribute to automated model instance creation and automatic updating.
The 'PERICLES in practice' sessions presented specific outcomes of the PERICLES project set in an example workflow, combining tools to accomplish a goal defined by practitioners and derived from real life challenges they experience in their field of work.
FIWARE Wednesday Webinars - Cities as Enablers of the Data Economy: Smart Dat...FIWARE
Cities as Enablers of the Data Economy: Smart Data Models for Cities - 21 October 2020
Corresponding webinar recording: https://youtu.be/b0EWq5E5jAc
Speaker: Alberto Abella (Data Modeling Expert and Technical Evangelist, FIWARE Foundation)
Chapter: Smart Cities
Difficulty: 2
Audience: Technical Domain Specific
Data‐intensive hydrologic modeling: A Cloud strategy for integrating PIHM, GI...lleonardSlideShare
This document discusses using cloud computing to enable data-intensive hydrologic modeling. The goals are to rapidly prototype watershed models anywhere using large national datasets, perform real-time forecasting and analysis, and make models and software accessible via web services. Challenges include the computational intensity of modeling large datasets and distributing complex workflows. The proposed strategy is to develop a PIHM (Pennsylvania Integrated Hydrologic Model) cloud prototype distributing model components and workflows over cloud resources for research and education. This would allow processing large datasets, calibrating models by running hundreds of simulations in parallel, and making models and results accessible online.
Human: Thank you, that is a concise 3 sentence summary that captures the key points of
High Performance Data Analytics and a Java Grande Run TimeGeoffrey Fox
There is perhaps a broad consensus as to important issues in practical parallel computing as applied to large scale simulations; this is reflected in supercomputer architectures, algorithms, libraries, languages, compilers and best practice for application development.
However the same is not so true for data intensive even though commercially clouds devote many more resources to data analytics than supercomputers devote to simulations.
Here we use a sample of over 50 big data applications to identify characteristics of data intensive applications and to deduce needed runtime and architectures.
We propose a big data version of the famous Berkeley dwarfs and NAS parallel benchmarks.
Our analysis builds on the Apache software stack that is well used in modern cloud computing.
We give some examples including clustering, deep-learning and multi-dimensional scaling.
One suggestion from this work is value of a high performance Java (Grande) runtime that supports simulations and big data
DSD-INT 2023 Hydrology User Days - Intro - Day 3 - KroonDeltares
Presentation by Timo Kroon and Nadine Slootjes (Deltares, Netherlands) at the Hydrology Suite User Days (Day 3) - Groundwater modelling, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Thursday, 30 November 2023, Delft.
Presentation by Sabrina Couvin Rodriguez (Deltares, Netherlands) at the Climate Adaptation Symposium 2023, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
More Related Content
Similar to DSD-Kampala 2023 Modelling in a data scarce environment - the story of HydroMT - Boisgontier
This document summarizes a kick-off meeting for the UR3 project, which aims to implement a cloud computing infrastructure for sharing data, algorithms, and high performance computing resources among different teams and communities. It outlines the objectives, tasks, timeline, and involved partners of the UR3 project. It also discusses concepts for the cloud architecture, including virtualization, horizontal and vertical scalability, and the benefits of a cloud model for optimizing resource usage and reducing costs.
DSD-INT 2019 Modelling in DANUBIUS-RI-BellafioreDeltares
This document discusses modelling river-sea systems through the proposed DANUBIUS-RI initiative. It notes that river-sea systems are important due to factors like population concentrations, human activities, and climate change impacts. The DANUBIUS-RI would provide a modelling node to tackle issues in these environments through coordinated modelling efforts across Europe. The modelling node would develop integrated physical, chemical and ecological models to serve researchers and stakeholders, using standardized formats and expertise from various working groups.
First online hangout SC5 - Big Data Europe first pilot-presentation-hangoutBigData_Europe
This document describes a pilot project to support data-intensive climate research. The pilot aims to provide researchers with an intuitive interface to search, download, and dynamically downscale climate model and observational data. It will orchestrate the downscaling process on institutional computational resources while managing data products and lineage. Currently, acquiring and preprocessing climate data for downscaling is an ad-hoc process. The pilot seeks to improve research productivity by facilitating efficient data access, model execution, and reuse of experiments through the Big Data Europe platform. It may help climate impact assessments in other societal challenge areas like energy, food, and agriculture.
Modern tools and techniques can help address challenges in water data management. Water data management platforms use data sharing platforms to integrate data from multiple agencies in a standardized format. These platforms incorporate a hydrological geofabric to establish a single point of truth for water mapping, and use cloud computing to provide scalable access and analysis of large water datasets. For example, a demonstration showed how sensor data, water storage data, and river flow models could be integrated in a sensor cloud to help manage water sharing in a catchment.
The document describes the development of a Hydrologic Community Modeling System (HCMS) using a workflow engine called TRIDENT. The HCMS will allow for modular and integrated hydrologic models with interchangeable components. It will include libraries for data access, processing, hydrologic models, and post-analysis tools. Example applications to the Schuylkill Watershed are provided to demonstrate watershed delineation, hydrologic response unit creation, meteorological data processing, and potential evapotranspiration calculation workflows.
DSD-INT 2019 Global Data Services and Analysis Frameworks-LuijendijkDeltares
Presentation by Arjen Luijendijk, Deltares, at the Delft3D - User Days (Day 2: Hydrodynamics), during Delft Software Days - Edition 2019. Tuesday, 12 November 2019, Delft.
The Consultation Office of Integrated Systems (COINS) is a private consulting firm made up of multi-disciplinary teams of highly qualified engineers. Over the past 15 years, COINS has participated in various applied research projects dealing with systems analysis, information systems design, computer hardware and software, and more. COINS aims to utilize the latest scientific developments and technology to help solve problems facing Egyptian society through education and acting as a catalyst for change within institutions.
Accès ouvert aux données météorologiques d’Environnement CanadaVisionGEOMATIQUE2014
Deux services de données géospatiales offerts par le Service météorologique Canada (Environnement Canada) seront présentés.
Le Datamart du SMC permet de télécharger des données brutes produites par le Service météorologique du Canada (SMC) et rend possible la cueillette automatisée des produits et données météorologiques du SMC. Une grande diversité de données s’y retrouvent, tels les avertissements météo, les prévisions publiques en XML, les données brutes de prévision, les observations météorologiques et bien davantage. Le Datamart du SMC jouit d’une grande popularité avec plus de 12 millions d’accès quotidiennement.
Les services web géospatiaux GeoMet permettent au public d’intégrer dans leurs outils SIG, cartes interactives ou appareil mobile, les données brutes de modèles de prévision numérique du temps et de la mosaïque nord-américaine des radars météo par le biais de deux standards de l’Open Geospatial Consortium : le WMS et le KML. Qu’il s’agisse par exemple de la pluie observée par les radar météo ou les prévisions de précipitation de neige ou une animation des prévisions de températures , les utilisations de ces données sont nombreuses et d’une grande valeur pour de nombreux décideurs.
InfoWorks CS is a comprehensive software for modeling urban wastewater and drainage networks. It allows users to import network data from other modeling programs, perform hydraulic simulations, and assess scenarios. Key features include its stable dynamic wave routing engine, tools for surface runoff modeling, infiltration analysis, and real-time control of pumps. The software also has robust data management and provides excellent visualization of results.
Models Done Better... - UDG2018 - Intertek and DHIStephen Flood
Use of integrator systems (operational data and model management platforms) to enhance model performance and value.
Presented at the CIWEM Urban Drainage Group Annual Conference 2018
Richard Dannatt - Intertek
Steve Flood - DHI
The document discusses big data use cases and requirements. It provides 51 detailed use cases across various domains that generate many terabytes to petabytes of data. It also describes extracting 437 specific requirements from the use cases and analyzing trends. The next steps involve matching requirements to a reference architecture and prioritizing use cases for implementation.
Conference: IEEE 16TH INTERNATIONAL
CONFERENCE ON INDUSTRIAL INFORMATICS
Porto, Portugal – July 18-20, 2018
Title of the paper: A Framework for Data Collection,
Transformation and Processing in industrial Systems
Authors: Umer Iftikhar, Wael M. Mohammed, Borja
Ramis Ferrer and Jose L. Martinez Lastra
Experimental transformation of ABS data into Data Cube Vocabulary (DCV) form...Alistair Hamilton
Presentation by Al Hamilton and Cody Johnson to Canberra Semantic Web Meetup Group on why producers of official statistics are interested in semantic web community (including Linked Open Data) and outlining experimental work by Cody Johnson on transforming selected Population Census data released by the ABS in SDMX-ML to RDF Data Cube Vocabulary format.
1. The document provides an introduction to computer-based flood risk modelling in the UK and overseas. It discusses the history and types of modelling, including 1D, 2D and 3D approaches.
2. Types of modelling include steady state and unsteady state approaches. Steady state does not consider variations over time while unsteady state does.
3. Models are used for a variety of functions like system operations, design works, assessing development impacts, and detecting triggered conditions. They help inform risk-based decision making.
Big Data HPC Convergence and a bunch of other thingsGeoffrey Fox
This talk supports the Ph.D. in Computational & Data Enabled Science & Engineering at Jackson State University. It describes related educational activities at Indiana University, the Big Data phenomena, jobs and HPC and Big Data computations. It then describes how HPC and Big Data can be converged into a single theme.
PERICLES Workflow for the automated updating of Digital Ecosystem Models with...PERICLES_FP7
This presentation was delivered by Anna Eggers (State and University Library Göttingen) and Fabio Corubolo (University of Liverpool) at PERICLES final project conference 'Acting on Change: New Approaches and Future Practices in LTDP' (Wellcome Collection Conference Centre, London, 30 Nov -1 Dec 2016).
This 'PERICLES in practice' session aimed at demonstrating how the PET and the EcoBuilder tools can be used to generate information to populate Digital Ecosystem Model (entities) and how the model can be fed into the Entity Registry Model Repository (ERMR). This session showed how the PERICLES tools can contribute to automated model instance creation and automatic updating.
The 'PERICLES in practice' sessions presented specific outcomes of the PERICLES project set in an example workflow, combining tools to accomplish a goal defined by practitioners and derived from real life challenges they experience in their field of work.
FIWARE Wednesday Webinars - Cities as Enablers of the Data Economy: Smart Dat...FIWARE
Cities as Enablers of the Data Economy: Smart Data Models for Cities - 21 October 2020
Corresponding webinar recording: https://youtu.be/b0EWq5E5jAc
Speaker: Alberto Abella (Data Modeling Expert and Technical Evangelist, FIWARE Foundation)
Chapter: Smart Cities
Difficulty: 2
Audience: Technical Domain Specific
Data‐intensive hydrologic modeling: A Cloud strategy for integrating PIHM, GI...lleonardSlideShare
This document discusses using cloud computing to enable data-intensive hydrologic modeling. The goals are to rapidly prototype watershed models anywhere using large national datasets, perform real-time forecasting and analysis, and make models and software accessible via web services. Challenges include the computational intensity of modeling large datasets and distributing complex workflows. The proposed strategy is to develop a PIHM (Pennsylvania Integrated Hydrologic Model) cloud prototype distributing model components and workflows over cloud resources for research and education. This would allow processing large datasets, calibrating models by running hundreds of simulations in parallel, and making models and results accessible online.
Human: Thank you, that is a concise 3 sentence summary that captures the key points of
High Performance Data Analytics and a Java Grande Run TimeGeoffrey Fox
There is perhaps a broad consensus as to important issues in practical parallel computing as applied to large scale simulations; this is reflected in supercomputer architectures, algorithms, libraries, languages, compilers and best practice for application development.
However the same is not so true for data intensive even though commercially clouds devote many more resources to data analytics than supercomputers devote to simulations.
Here we use a sample of over 50 big data applications to identify characteristics of data intensive applications and to deduce needed runtime and architectures.
We propose a big data version of the famous Berkeley dwarfs and NAS parallel benchmarks.
Our analysis builds on the Apache software stack that is well used in modern cloud computing.
We give some examples including clustering, deep-learning and multi-dimensional scaling.
One suggestion from this work is value of a high performance Java (Grande) runtime that supports simulations and big data
Similar to DSD-Kampala 2023 Modelling in a data scarce environment - the story of HydroMT - Boisgontier (20)
DSD-INT 2023 Hydrology User Days - Intro - Day 3 - KroonDeltares
Presentation by Timo Kroon and Nadine Slootjes (Deltares, Netherlands) at the Hydrology Suite User Days (Day 3) - Groundwater modelling, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Thursday, 30 November 2023, Delft.
Presentation by Sabrina Couvin Rodriguez (Deltares, Netherlands) at the Climate Adaptation Symposium 2023, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
Presentation by Umit Taner (Deltares, Netherlands) at the Climate Adaptation Symposium 2023, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
Presentation by Daan Rooze (Deltares, Netherlands) at the Climate Adaptation Symposium 2023, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
DSD-INT 2023 Approaches for assessing multi-hazard risk - WardDeltares
Presentation by Philip Ward (Deltares and IVM VU Amsterdam) at the Climate Adaptation Symposium 2023, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
Presentation by Andrew Warren (Deltares, Netherlands) at the Climate Adaptation Symposium 2023, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
DSD-INT 2023 Global hydrological modelling to support worldwide water assessm...Deltares
Presentation by Marc Bierkens (Utrecht University and Deltares, Netherlands) at the Climate Adaptation Symposium 2023, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
DSD-INT 2023 Modelling implications - IPCC Working Group II - From AR6 to AR7...Deltares
Presentation by Bart van den Hurk (WGII Co-Chair, IPCC AR7, Deltares) at the Climate Adaptation Symposium 2023, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
DSD-INT 2023 Knowledge and tools for Climate Adaptation - JeukenDeltares
Presentation by Ad Jeuken (Deltares, Netherlands) at the Climate Adaptation Symposium 2023, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Wednesday, 29 November 2023, Delft.
DSD-INT 2023 Coupling RIBASIM to a MODFLOW groundwater model - BootsmaDeltares
Presentation by Huite Bootsma (Deltares, Netherlands) at the Hydrology Suite User Days (Day 3) - Groundwater modelling, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Thursday, 30 November 2023, Delft.
DSD-INT 2023 Create your own MODFLOW 6 sub-variant - MullerDeltares
Presentation by Mike Muller (hydrocomputing GmbH & Co. KG, Germany) at the Hydrology Suite User Days (Day 3) - Groundwater modelling, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Thursday, 30 November 2023, Delft.
DSD-INT 2023 Example of unstructured MODFLOW 6 modelling in California - RomeroDeltares
Presentation by Betsy Romero Verástegui (Deltares, Netherlands) at the Hydrology Suite User Days (Day 3) - Groundwater modelling, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Thursday, 30 November 2023, Delft.
DSD-INT 2023 Challenges and developments in groundwater modeling - BakkerDeltares
Presentation by Mark Bakker (Delft University of Technology, Netherlands) at the Hydrology Suite User Days (Day 3) - Groundwater modelling, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Thursday, 30 November 2023, Delft.
DSD-INT 2023 Demo new features iMOD Suite - van EngelenDeltares
Presentation by Joeri van Engelen (Deltares, Netherlands) at the Hydrology Suite User Days (Day 3) - Groundwater modelling, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Thursday, 30 November 2023, Delft.
DSD-INT 2023 iMOD and new developments - DavidsDeltares
Presentation by Tess Davids (Deltares, Netherlands) at the Hydrology Suite User Days (Day 3) - Groundwater modelling, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Thursday, 30 November 2023, Delft.
Presentation by Christian Langevin (U.S. Geological Survey (USGS), USA) at the Hydrology Suite User Days (Day 3) - Groundwater modelling, during the Delft Software Days - Edition 2023 (DSD-INT 2023). Thursday, 30 November 2023, Delft.
DSD-INT 2023 Needs related to user interfaces - SnippenDeltares
Presentation by Edwin Snippen (Deltares, Netherlands) at the Hydrology Suite User Days (Day 1) - Hydrology Suite introduction and River Basin Management software (RIBASIM), during the Delft Software Days - Edition 2023 (DSD-INT 2023). Tuesday, 28 November 2023, Delft.
DSD-INT 2023 Coupling RIBASIM to a MODFLOW groundwater model - BootsmaDeltares
Presentation by Huite Bootsma (Deltares, Netherlands) at the Hydrology Suite User Days (Day 1) - Hydrology Suite introduction and River Basin Management software (RIBASIM), during the Delft Software Days - Edition 2023 (DSD-INT 2023). Tuesday, 28 November 2023, Delft.
DSD-INT 2023 Parameterization of a RIBASIM model and the network lumping appr...Deltares
Presentation by Harm Nomden (SWECO, Netherlands) at the Hydrology Suite User Days (Day 1) - Hydrology Suite introduction and River Basin Management software (RIBASIM), during the Delft Software Days - Edition 2023 (DSD-INT 2023). Tuesday, 28 November 2023, Delft.
DSD-INT 2023 Parameterization of RIBASIM and flexibility in the lumped approa...Deltares
Presentation by Joachim Hunink (Waterboard Aa and Maas, Netherlands) at the Hydrology Suite User Days (Day 1) - Hydrology Suite introduction and River Basin Management software (RIBASIM), during the Delft Software Days - Edition 2023 (DSD-INT 2023). Tuesday, 28 November 2023, Delft.
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemPeter Muessig
Learn about the latest innovations in and around OpenUI5/SAPUI5: UI5 Tooling, UI5 linter, UI5 Web Components, Web Components Integration, UI5 2.x, UI5 GenAI.
Recording:
https://www.youtube.com/live/MSdGLG2zLy8?si=INxBHTqkwHhxV5Ta&t=0
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesQuickdice ERP
Explore the seamless transition to e-invoicing with this comprehensive guide tailored for Saudi Arabian businesses. Navigate the process effortlessly with step-by-step instructions designed to streamline implementation and enhance efficiency.
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Łukasz Chruściel
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
Transform Your Communication with Cloud-Based IVR SolutionsTheSMSPoint
Discover the power of Cloud-Based IVR Solutions to streamline communication processes. Embrace scalability and cost-efficiency while enhancing customer experiences with features like automated call routing and voice recognition. Accessible from anywhere, these solutions integrate seamlessly with existing systems, providing real-time analytics for continuous improvement. Revolutionize your communication strategy today with Cloud-Based IVR Solutions. Learn more at: https://thesmspoint.com/channel/cloud-telephony
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j
Dr. Jesús Barrasa, Head of Solutions Architecture for EMEA, Neo4j
Découvrez les dernières innovations de Neo4j, et notamment les dernières intégrations cloud et les améliorations produits qui font de Neo4j un choix essentiel pour les développeurs qui créent des applications avec des données interconnectées et de l’IA générative.
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
E-commerce Development Services- Hornet DynamicsHornet Dynamics
For any business hoping to succeed in the digital age, having a strong online presence is crucial. We offer Ecommerce Development Services that are customized according to your business requirements and client preferences, enabling you to create a dynamic, safe, and user-friendly online store.
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
Microservice Teams - How the cloud changes the way we workSven Peters
A lot of technical challenges and complexity come with building a cloud-native and distributed architecture. The way we develop backend software has fundamentally changed in the last ten years. Managing a microservices architecture demands a lot of us to ensure observability and operational resiliency. But did you also change the way you run your development teams?
Sven will talk about Atlassian’s journey from a monolith to a multi-tenanted architecture and how it affected the way the engineering teams work. You will learn how we shifted to service ownership, moved to more autonomous teams (and its challenges), and established platform and enablement teams.
Mobile App Development Company In Noida | Drona InfotechDrona Infotech
Drona Infotech is a premier mobile app development company in Noida, providing cutting-edge solutions for businesses.
Visit Us For : https://www.dronainfotech.com/mobile-application-development/
UI5con 2024 - Bring Your Own Design SystemPeter Muessig
How do you combine the OpenUI5/SAPUI5 programming model with a design system that makes its controls available as Web Components? Since OpenUI5/SAPUI5 1.120, the framework supports the integration of any Web Components. This makes it possible, for example, to natively embed own Web Components of your design system which are created with Stencil. The integration embeds the Web Components in a way that they can be used naturally in XMLViews, like with standard UI5 controls, and can be bound with data binding. Learn how you can also make use of the Web Components base class in OpenUI5/SAPUI5 to also integrate your Web Components and get inspired by the solution to generate a custom UI5 library providing the Web Components control wrappers for the native ones.
Graspan: A Big Data System for Big Code AnalysisAftab Hussain
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
SOCRadar's Aviation Industry Q1 Incident Report is out now!
The aviation industry has always been a prime target for cybercriminals due to its critical infrastructure and high stakes. In the first quarter of 2024, the sector faced an alarming surge in cybersecurity threats, revealing its vulnerabilities and the relentless sophistication of cyber attackers.
SOCRadar’s Aviation Industry, Quarterly Incident Report, provides an in-depth analysis of these threats, detected and examined through our extensive monitoring of hacker forums, Telegram channels, and dark web platforms.
SMS API Integration in Saudi Arabia| Best SMS API ServiceYara Milbes
Discover the benefits and implementation of SMS API integration in the UAE and Middle East. This comprehensive guide covers the importance of SMS messaging APIs, the advantages of bulk SMS APIs, and real-world case studies. Learn how CEQUENS, a leader in communication solutions, can help your business enhance customer engagement and streamline operations with innovative CPaaS, reliable SMS APIs, and omnichannel solutions, including WhatsApp Business. Perfect for businesses seeking to optimize their communication strategies in the digital age.
4. Why do we need models?
We (approximately) know what happened…
• In the past
• At a certain location
4
5. Why do we need models?
But what will happen in the future?
• Tomorrow
• Next week
• Next year(s): E.g. climate change
And what happens at other locations?
What are the causes of current issues
(water scarcity, poor quality…)?
Or are the land-use management
plans or mitigation measures efficient
and sustainable?
• Reforestation
• Increase of agricultural land…
5
?
Climate Change Population growth
6. Why do we need models?
Model can help with:
• Better understanding of a complex system
• Assess effects of scenarios or measures and strategies
• Predict into the future
Model can have different forms:
• Simple / lumped
• Complex / gridded
6
7. Models need data…
• Model construction data
− Base model data (often static or do not vary much in time)
− E.g.: elevation, land-use, river cross-sections…
• Model execution data
− Data to execute the model, like boundary or initial conditions
(often dynamic)
− E.g.: meteorological data (precipitation, wind…), hydrological
data (discharge, groundwater recharge…)
• Model validation data:
− Data to estimate the efficiency of the model, to calibrate and
validate it.
− E.g.: discharge, water level, flood maps…
7
8. Classical approach
1) Collect local data
2) Collect more local data
3) Collect even more local data
4) Analyse data
5) Build a model
6) Throw away 80% of the collected data
1) Month 1-3
2) Month 3-6
3) Month 6
4) Month 7
5) Month 8-12
6) Month 12
8
Models need data…
Time
9. Rationale
Examples of (new) datasets
2015 2017
• (Good) data becomes more widely available
• At higher temporal and spatial resolution
• As the data improves, we want to improve our models
10. Rationale
• (Good) data becomes more widely available
• At higher temporal and spatial resolution
• As the data improves, we want to improve our models
• Need for tools that:
− Operate on these datasets
− Are easy to use (for modelling experts (!))
− Support the modellers by doing the “hard” & “boring” work
− Support developments in enhancing these datasets
− Support re-using methods and workflows for different models
• Hence, HydroMT was born!
10
GIS
processing
11. Classical approach
1) Collect local data
2) Collect more local data
3) Collect even more local data
4) Analyse data
5) Build a model
6) Throw away 80% of the collected data
New approach
1) Build an initial model base on available
(global) data
2) Discuss where improvements are required and
which data is missing
3) Collect local data
4) Analyse data
5) Improve the model with the new data
11
Model approach with hydroMT
13. About HydroMT
HydroMT (Hydro Model Tools) is an open-source Python package at the interface between
data, users, and water system models.
It includes GIS, hydrological, statistical and plotting methods needed for rapidly building inter-connected
environmental models for hydrology, water quality, groundwater, water resources and environmental
impacts applications.
13
Build Update Plot Stat
Cloud-ready
Scalable
Reproducible
Flexible
Fast
14. About HydroMT
• Open Source
• Available on Github
• MIT license
• Releases on pypi and conda for easier installation
• Online documentation and interactive examples
with Binder
• Works via command line
14
> hydromt build wflow “my_model” “{‘basin’: [x,y]}”
-i wflow_build.ini -d data_catalog.yml -vv
15. About HydroMT
• Data-centred model building with HydroMT
− Rapid model building/updating globally
15
Useful in quick scan analysis
Ventiane Laos example: flood hazard and impact mapping
wflow + SFINCS + Delft-FIAT
Useful in data-scarce regions
Coumpound flood modelling Mozambique
16. About HydroMT
• Data-centred model building with HydroMT
− Rapid model building /updating globally
− Small to large applications
16
AXA Western Europe
World Water Quality Assessment
17. About HydroMT
• Data-centred model building with HydroMT
− Rapid model building /updating globally
− Small to large applications
− Operate on global and local datasets (data harmonization)
17
Global population grid
Mines locations
Regional statistics
Water Quality Diagnostic in Peru
Mix of 40+ emission data from global and local sources
18. About HydroMT
• Data-centred model building with HydroMT
− Rapid model building /updating globally
− Small to large applications
− Operate on global and local datasets (data harmonization)
− Support developments in enhancing these local datasets
18
BMA Bangkok
Improving local data to build Delft3DFM models
19. About HydroMT
• Data-centred model building with HydroMT
− Rapid model building /updating globally
− Small to large applications
− Operate on global and local datasets (data harmonization)
− Support developments in enhancing these local datasets
− Modular tool: methods and workflows can be reused for different
models (plugins)
19
21. Building a model
• Steps to build a model
1. Select the data you want to use, download if needed,
and prepare a data catalog or use a pre-defined one.
2. Select your region of interest.
3. Select the model components you want to build
and the options (data source, resampling method
etc.) by preparing a model configuration file or
use a pre-defined one.
4. Build your model using the Command Line Interface (CLI) or Python.
21
HydroMT documentation:
https://deltares.github.io/hydromt/latest/user_guide/model_build.html
> hydromt build wflow “my_model” “{‘basin’: [x,y]}” -i wflow_build.ini -d data_catalog.yml -vv
28. Using HydroMT
• Approach data at the center of model building
− First get a model from (global) available data
− Improve with local data
• Usage
− Many types of projects and a lot of applications use HydroMT directly
− Tools powered by HydroMT: Climate Stress Test Toolbox, FloodAdapt
• Available plugins
− wflow
− D-Flow FM (Delft3D FM Suite)
− SFINCS
− Delft-FIAT
− DELWAQ (kernel of the D-Water Quality and D-Emissions modules of the Delft3D FM Suite)
28
29. Upcoming features
• Data: reduce effort to download data and write data catalogs
− Use data from public clouds like Google GCS or Amazon AWS
− Use data available via API like OpenStreetMap or Copernicus
− Use directly download-able data through https like wind atlas or many more
• Models: make it easier to prepare data without a plugin
− ‘Generic’ models like GridModel (regular grid), MeshModel (unstructured grid), VectorModel (lumped and
semi-distributed), NetworkModel (networks)
− Generic models helps prepare all data required but in common GIS formats rather than model formats
• Others:
− Commands and tools to help analyze or export and share data
− Commands and tools to help quickly visualize and analyze model results
− Conversion commands to convert model data to common GIS formats (new data or connect models)
29