Advertisement
Check these out next
HydrOffice: past, present, and future.
Feb. 10, 2018•0 likes•21,418 views
2 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Report
Engineering
Presentation on past, present, and future of the HydrOffice project given at the NOAA OCS Field Procedure Workshop 2018.
Giuseppe MasettiFollow
Research Faculty at CCOM/JHC, UNHAdvertisement
Advertisement
Advertisement
Recommended
Leveraging Predictions from NOAA’s Oceanographic Forecast Models to Increase ...Giuseppe Masetti
INMARTECH 2018 - G.Masetti & P.Johnson - Sound Speed Management and Environme...Giuseppe Masetti
Pydro & HydrOffice: Open Tools for Ocean MappersGiuseppe Masetti
Hydrographic Survey Validation and Chart Adequacy Assessment Using Automated ...Giuseppe Masetti
ePOM - Fundamentals of Research Software Development - IntroductionGiuseppe Masetti
DSD-INT 2017 Urban flood forecasting in central Shanghai (Pudong Area) - ZhangDeltares
HydrOffice: past, present, and future.
- HydrOffice NOAA Office of Coast Survey – Field Procedures Workshop 2018 – Portland, OR
- IN BRIEF Past Present Future • Motivations • Lessons learned • Available tools • Distribution • Next challenges • Utopias 2
- 3
- 4 specs manuals tools
- 5 specs manuals tools
- 6 CCOM/JHC Italy
- A framework of libraries and tools for Ocean Mapping 7 Quickly prototype and test innovative ideas Ease the transition from research to operation Ref.: G. Masetti, Wilson, M. J., Calder, B. R., Gallagher, B., and Zhang, C., “Research-driven Tools for Ocean Mappers”, Hydro Int., vol. 21, 5. GeoMares, 2017.
- Support open formats 8 Listen the field feedback Maintenance is a time sink Support from hydro community
- 9 OCS-UNH CO-DEVELOPMENT
- PRESENT Available tools and their distribution 10
- QC Tools ▪ Automate QC for Survey Review and Chart Compilation ▪ POCs: ▫ Clinton Marcus ▫ John Doroba 11Ref.: M. J. Wilson, Masetti, G., and Calder, B. R., “Automated Tools to Improve the Ping-to-Chart Workflow”, Int. Hydr. Review, vol. 17. IHB, pp. 21-30, 2017.
- QC Tools & VR Grids ▪ Recode to support SR and VR: a) Per tile b) Over-sampling ▪ HTD 2017-2 ▪ POCs: ▫ Janice Eisenberg ▫ Jack Riley ▫ Chen Zhang 12
- QC Tools & Bottom Sampling ▪ Coming HTD 2018-X ▪ S57 to shp + images ▪ S57-CMECS crosswalk ▪ EXIF geotagging ▪ POCs: ▫ Sarah Wolfskehl ▫ Starla Robinson ▫ Jason Baillio 13
- Sound Speed Manager ▪ Manage sound speed casts ▪ Adopted by UNOLS vessels (MAC) and many others ▪ Modified to fit OCS needs ▪ POCs: ▫ Matthew Sharr ▫ Barry Gallagher ▫ Chen Zhang 14Ref.: G. Masetti, Gallagher, B., Calder, B. R., Zhang, C., and Wilson, M. J., “Sound Speed Manager”, Int. Hydr. Review, vol. 17. IHB, pp. 31-40, 2017.
- Survey Data Monitor ▪ Merge ideas from: ▫ Manda’s svplot ▫ Wilson’s CastTime ▪ Leverage: ▫ SSM database ▫ SSM-SIS interaction 15
- SmartMap ▪ Effects of oceanographic variability on mapping surveys ▪ Two components: ▫ C++ & Python ▫ GeoServer and OGC services ▪ WebGIS: ▫ hydroffice.org/smartmap/ 16Ref.: G. Masetti, Kelley, J., Johnson, P., and Beaudoin, J., “A Ray-Tracing Uncertainty Estimation Tool for Ocean Mapping”, IEEE Access. IEEE, pp. 1-9, 2017.
- SmartMap WebGIS ▪ RTOFS + WOA13 ▪ Animation ▪ Past data ▪ Survey Planner 17
- Bress ▪ Preliminary segmentation from co-located DEMs and backscatter mosaics ▪ Based on principles of: ▫ Topographic openness ▫ Pattern recognition ▫ Texture classification 18Ref.: G. Masetti, Mayer, L. A., and Ward, L. G., “A Bathymetry- and Reflectivity-Based Approach for Seafloor Segmentation”, Geosciences, vol. 8(1). MDPI, 2018.
- 19 Bress Local Ternary Patterns Bedform Classification
- 20 Bress Area Kernels Output Segments
- 21 Bress
- Other Apps ▪ BAG Explorer 22 ▪ ENCx ▪ SIS Emulator ▪ Huddl ▪ StormFix
- HYDROFFICE APPS 23 PYTHON SCIENTIFIC STACK OCEAN MAPPING LIBS & SCRIPTS
- Distribution Pydro Universe Stand-alone Apps Python Packages See Jack’s slides www.hydroffice.org GitHub/PyPi/Conda 24
- HydrOffice.org ▪ Mobile-first, dynamic website ▪ Based on Django & JavaScript ▪ Per-tool Home Page ▫ Info ▫ Stand-alone downloads ▫ Manuals ▫ Embedded tutorials 25
- HydrOffice.org 26(*) Google Analytics, Number of Sessions, January 2018, location filtered: Durham, Silver Spring, unset.
- FUTURE Next challenges … 27
- Next Challenges … ▪ Real-time QC Tools ▪ Expand DtoN Scanner ▪ Improve Cast Time Prediction ▪ Regional Models for SmartMap ▪ Open format for backscatter and WC data ▪ Integration with existing GIS apps and/or an Ocean Mapping GIS 28 … and your ideas !!!
- THANKS! Any questions? You can contact me at: gmasetti@ccom.unh.edu
Editor's Notes
- Good morning. My name is Giuseppe Masetti. I am research faculty at the University of New Hampshire. My presentation is about the HydrOffice project.
- In my talk, I want to answer three questions: 1. What are the motivations behind HydrOffice? 2. What can you do with HydrOffice? 3. What is coming next?
- I will start with my personal motivation. I discovered the hydrographic world in 2005.
- Over time, I started to develop the feeling of being stuck in a sort of incomplete triangle. On one side, a large amount of specs and good practices. On the other side, a bunch of software tools to try to follow the specs. Now, in order to understand what the software is doing, you usually have not other choice that reading .. the user manuals. Now a couple of simple considerations. 1. A manual rarely refers to given specs simply because it describes the tool functionalities and they are not written for the rules of your specific agency. 2. With time and practice, you can master a tool.. But this alone does not mean that you are following the specs. (As hydrographer in charge, your high-level goal is not to master a specific software tools, but to follow the specs.) Is the software really doing what we want that it does? Should the tools drive specs modifications? Of course, no. But what if you cannot implement a task with the available tools?
- At the very end, you can reduce the whole situation to a single choice: Do you want to trust? Or do you want to go deeper and understand more?
- I took the red pill because I wanted to understanding. This decision motivated me in.. attending a Master program at UNH. There, I met a lot of hydro friends, people with motivations similar to mine. Among them, Matt Wilson and Brian Calder. And after a few months, we collected some ideas in a white paper.
- The White Paper was about developing a framework of libraries and tools for Ocean Mapping. A framework designed with two main tasks in mind: - To quickly prototype and test innovative ideas. - To ease the transition of the good ideas from research to operation.
- The most relevant of the lessons learned is that coding is a key component. Learning how to code is really empowering and it is not really so difficult if you start gradually. Other lessons that I have learned is that is safer to support open formats (like BAG and S57) since it is easily debug their content when things go wrong and you cannot be locked out of your data. The value in listening the field feedback is quite obvious, but I wanted to put here this item as a reminder to thank all the dozens of users that wrote us feedbacks in the past 3 years. Another lesson learned is that maintenance is a big time sink. If you make available a tool to public users, then you cannot never called it done! There will be always a new coming format or a change in the specs that you have to manage. I don’t think that we have currently a working solution for this issue. A way to partial manage the maintenance issue is to create a community and a team around each tool. This also helps to make it more structural and to survive when one of the member move forward with life and career.
- This is why I believe that the OCS-UNH co-development agreement provided a big boost to the HydrOffice project. I could have filled the next slides with long lists of names with all the contributors of this tools, but to simplify I have just added this word cloud with some of them.
- What is available in HydrOffice? In the next few slides I will briefly present 4 selected apps.
- QC Tools is a collection of tools that automate QA for survey review and chart compilation. - Convert specs into code - Familiarize new personnel to specs During the past couple of years, QC Tools functionalities have been presented in details by Matt Wilson. So I will skip all the details. The new NOAA point of contacts are Clinton Marcus and John Doroba.
- In the past year, all the algorithms have been recoded to support both SR and VR grids. Two adopted approaches: per tile and over-sampling QC Tools has been also useful during the development of the technical directive. There are the POCS if you want to know more about the VR implementation.
- An existing tool has been modified to support a coming HTD for bottom sampling QC Tools automates the creation of shapefile from S57 objects, Add CMECS attributes derived from S57 attributes, and Geo-tag the image with information coming from the S57 file so that they can be easily visualized in GIS apps. These are the POCs if you want to know more about this implementation.
- Sound Speed Manager is an application built to facilitate the management of sound speed casts. It is quite popular since it has been adopted my UNOLS vessels and many others. Matt will extensively talk later this morning about this application and the many recent improvements. Here I only want to cite the recent introduction of a new module built on top of Sound Speed Manager called Survey Data Monitor.
- This module is still in its infancy. It merges ideas from Damian Manda’s SV Plot and Matt Wilson’s Cast Time. It also leverages the SSM database and its interaction with SIS. Since we are talking about effects of sound speed on bathymetry, I want to also briefly talk about ..
- SmartMap estimates the effects of oceanographic variability on mapping surveys. It is composed of two components: A back-end that is C++ and Python, and A front-end provided by GeoServer as OGC services To make it more user-friendly we have added a Web GIS interface
- SmartMap WebGIS Support both forecast model as the Global RTOFS and climatology as the WOA 2013 Animate depth-bias estimations up to 7 days in the future It is possible to access past data. Recently added Survey Planner.
- Bress is a new algorithm that provides preliminary segmentation from co-located bathymetry and backscatter. It is based on principles of topographic openness, pattern recognition, and texture classification.
- Each node is classified by a local ternary pattern, then the patterns are classified in bedform types using a customizable look up table.
- From connected bedform types to output segments by: Identify multimodality within the same area kernel, and Comparing histograms of pairs of area kernels.
- Some promising results in discriminate among areas with different types of sediment, but more research is required to test the method.
- Several other apps have been developed in the past years. For instance: - BAG Explorer and ENCx to browse the content of BAG and S57 files. - SIS Emulator to emulate the interaction with Kongsberg SIS - Huddl to create reader and writer for raw hydrographic formats - StormFix to detect and fix issues with collected backscatter data.
- But I believe that the most relevant contribution of HydrOffice are not the apps. The apps are the tip of the iceberg. The most important part are the Ocean Mapping libs that have been developed on top of the existing Python scientific stack. They make possible to call the algorithms using scripts with tools like Charlene [and you will hear about it from Eric later this morning]. They also ease the creation of new tools..
- But you can create the perfect tool, but nobody will use it if you don’t make it accessible to users. There are currently three ways for the user to access the HydrOffice tools: 1. Pydro Universe. Jack has already widely talked about Pydro Universe. From the HydrOffice perspective, Pydro is the ideal Python distribution for easily reach a large number of users, but it is Windows only. 2. For users using different OS, the tools are structured as Python packages that can be installed on a number of different systems. 3. Finally, the third option is download the stand-alone app from the official HydrOffice website.
- Hydroffice.org is a mobile-first, dynamic website. That is based on Python Django. It provides a section for each tool with info, download, manuals, and tutorial.
- Based on last-month results from Google Analytics, there is a world-wide community interested in these tools.
- Up to this point in my presentation, I have talked about tools that are already implemented. You can download and install these tools right now. Now the future..
- I don’t usually like the section “Future works” in a presentation. I find very sad to review past presentations to discover fantastic future works that actually never happened. So I have listed just a few of them, in order of likelihood, from very likely to almost utopias. Some work is ongoing to improve the quality of the collected data. For instance, the monitoring of the multibeam runtime, machine learning to improve the prediction of the next cast and the adoption of higher resolution models in SmartMap. Moving down in the likelihood scale, the creation of an open format for BS and WC data. Another improvement might be the integration with existing GIS apps (but this would require vendors cooperation) or, in alternative, the creation of an ocean mapping GIS. And finally we would love to hear your ideas on new tools.
Advertisement