1. PYDRO & HYDROFFICE
OPEN TOOLS FOR OCEAN MAPPERS
D. MANDA AND G. MASETTI
V1

2. WHAT IS PYDRO?

3. A NOAA Python Distribution
Pydro Universe
A Collection of Open-Source
Hydrographic Tools
A Growing Community of
People with Interests in
Ocean Mapping

4. Pydro GIS
1994 2015Closed Source

5. 1994 2015Closed Source 2016

6. 1994 2015Closed Source 2016
2016: TRANSITIONING TO OPEN
▪ COLLABORATION WITH THE UNIVERSITY
OF NEW HAMPSHIRE
▪ RELEASE OF NOAA BUILT APPS
▪ OPEN UNDERLYING CODE FOR USE

7. INSTALLATION
▪ https://svn.pydro.noaa.gov/

9. WHAT IS HYDROFFICE?

10. HydrOffice
10
A research framework for
ocean mapping to facilitate
research-to-operation (R2O)

11. 11
CCOM/JHC Italy
2013
Initial thoughts

12. A framework of
libraries and tools
for Ocean Mapping
12
Quickly prototype
and test
innovative ideas
Ease the transition
from research to
operation
Ref.: G. Masetti, Wilson, M. J., Calder, B. R., Gallagher, B., and Zhang, C., “Research-driven Tools for Ocean Mappers”, Hydro Int., vol. 21, 5. GeoMares, 2017.
2015
White Paper

13. July 2016
Site Review

14. March 2017
SSM release

15. 15
CCOM/JHC Italy
January 2018
Team Change

16. Several Apps
▪ BAG Explorer
16
▪ ENCx
▪ QC Tools
▪ Huddl
▪ StormFix
▪ CA Tools
▪ Bress
▪ …

17. HYDROFFICE APPS
17
PYTHON SCIENTIFIC STACK
OCEAN MAPPING LIBS
& SCRIPTS

18. 18
OCS-UNH CO-DEVELOPMENT

19. Pydro Universe Stand-alone Apps Python Packages
www.nauticalcharts.noaa.gov www.hydroffice.org GitHub/PyPi/Conda
19
Distribution

20. HydrOffice.org
▪ Mobile-first, dynamic website
▪ Based on Django & JavaScript
▪ Per-tool Home Page
▫ Info
▫ Stand-alone downloads
▫ Manuals
▫ Embedded tutorials
20

21. HydrOffice.org
21(*) Google Analytics, Number of Sessions, January 2018, location filtered: Durham, Silver Spring, unset.

22. SOUND SPEED MANAGER
AND SMARTMAP

23. Sound Speed Manager
A ready-to-go and free
solution to ease
the management of
sound speed profiles
for ocean mapping
23Ref.: Masetti, G. et al., Sound SpeedManager: An open-source application to manage sound speed profiles, International Hydrographic Review, vol. 17, pp. 31-40, 2017.

24. Collaborative Effort
24

25. Collaborative Effort
25

26. Open Source
26

27. Liberal License
27

28. 28
✓Support
✓Open
✓Free

29. Format Converter
29

30. 30
MVP SIS

31. 31

32. 32

33. 33

34. 34

35. 35

36. 36

37. 37

38. 38

39. SmartMap
A tool to evaluate
the effects of
oceanographic variability
on mapping surveys
39Ref.: Masetti,, G., Kelley, J., Johnson, P., and Beaudoin, J., A Ray-Tracing Uncertainty Estimation Tool for Ocean Mapping, IEEE Access. IEEE, pp. 1-9, 2017.

40. 40

41. 41

42. SmartMap Algorithm
▪ Constant Gradient:
▪ Snell Law:
42

43. SmartMap WebGIS
▪ RTOFS +
WOA13
▪ Animation
▪ Survey Planner
43

44. Gulf of Maine OFS (beta)
44

45. TCARI

46. Tidal Constituent And Residual Interpolation
▪ Created in 1995 at the Coast Survey Development Lab
▪ Overcomes weaknesses in traditional tidal models
▪ Interpolates across the project area:
▫ Tidal constituents
▫ Residual water level, and
▫ Desired vertical datum.
46

47. 47

48. Disadvantages of discrete zoning:
▪ Does not capture subtle changes in tides
▪ Does not capture meteorological, hydrological and
density effects
▪ Introduces steps at boundary polygons
▪ Polygons are drawn by hand
48

49. 50

50. 51

51. 52

52. QC/CA TOOLS

53. Ping-to-chart (very simplified)
Marine
Chart
Division
Hydro
Branches
Field
Survey deliverables
(grids, features, etc.)
Data archive
and public
access
Chart
Products!
Rapid Survey Assessment (RSA)
Survey Review (SAR)
National
Center for
Environmental
Information
Processing Branch deliverables
(grids, features, reports, etc.)

54. QC Tools
Performs automated
quality control checks
on surfaces and final
feature files.
55

55. Workflow
56
GDAL Open Formats
▪ BAG
▪ S-57
▪ Shapefile
▪ KML

56. Survey Review
57
Detect Fliers
Detect Holidays
Grid QA
Scan Designated
Scan Features
VALSOU Check
SBDARE Export
Submission
Checks
Uncertainty
Calculator *
Rock or Islet
Oracle *
* Standalone Tools

57. 58

58. 59
Object Detection:
>3 collinear nodes
Full Coverage:
>3x3 nodes
All Holes

59. Grid QA Outputs

60. Grid QA Outputs

61. Rock or Islet?
62
?

62. 63

63. 64

64. 65
Which tool to improve first?

65. CA Tools
Performs chart adequacy
tasks by comparing survey
data and current ENCs
66

66. 67

67. 68
z

68. 69
z

69. 70
z
10.2
4.5
4.8

70. 71
z
10.2
4.5
4.8 5.1

71. 72
z
10.2
4.8 5.1
4.5

72. 73
z
10.2
4.8 5.1
4.5

73. 74
z
10.2
4.5
4.8 5.1

74. Tilted Triangle
75

75. Flat Triangles > Point-In-Polygon Test
76

76. 77

77. 78

78. CHARLENE

79. With Charlene…
▪ Automated data management
▪ Set-and-forget processing
▪ Single interface for user
▪ 40 page SOP encompassed in a few dropdown boxes
▪ Instruction/SOP can be done from a screenshot
80

80. 81

81. 82

82. 83

83. XMLDR

84. 85

85. 86

86. 87

87. 88
XmlDR Output

88. OTHER PYDRO TOOLS

89. 90

90. POSPac AutoQC

91. 92
Sonar Acceptance Tools

92. OTHER HYDROFFICE TOOLS

93. BAG Explorer
95

94. StormFix
96
ARTIFACTS
DETECTION
ARTIFACTS
REDUCTION
BACKSCATTER
MOSAICKING
ANGULAR
RESPONSE
ANALYSIS
Ref.: G. Masetti et al., “How to Improve the Quality and the Reproducibility for Acoustic Seafloor Characterization”, GeoHab 2017. p. Nova Scotia, Canada, 2017.

95. StormFix: How it works?

96. 98

97. 99

98. 100

99. 101
Just Removal vs Randomization Schema

100. BRESS
▪ Preliminary segmentation from
co-located DEMs and
backscatter mosaics
▪ Based on principles of:
▫ Topographic openness
▫ Pattern recognition
▫ Texture classification
102Ref.: G. Masetti, Mayer, L. A., and Ward, L. G., “A Bathymetry- and Reflectivity-Based Approach for Seafloor Segmentation”, Geosciences, vol. 8(1). MDPI, 2018.

102. 104
Landform ClassificationLocal Ternary Patterns

103. 105
Output SegmentsArea Kernels

104. 106

105. 107

106. WHAT’S COMING?

107. OPENBST

108. Raw Data File
Caris SIPS
QPS FMGT
Ifremer SonarScope
Beams
Pings
Pings
Pings
Beams
Beams
Reflectivity[dB]

109. Reflectivity[dB]
Average Initial Reflectivity per Beam NoData Values for Initial Reflectivity per Beam
Beams
Nr.ofValidData
Histogram of Initial Reflectivity per Beam
Reflectivity [dB]

110. A SHIFT IS REQUIRED FROM THE EXISTING CLOSED-SOURCE APPROACH
AN OPEN-SOURCE AND METADATA-RICH
MODULAR IMPLEMENTATION
NOT COMPETING WITH COMMERCIAL SOLUTIONS, BUT
A SET OF COMMUNITY-VETTED, REFERENCE ALGORITHMS

111. EXTENDS
SONAR-NETCDF4
FOLLOWS CF METADATA CONVENTIONS

112. RAW
MOS
Raw Data
Decoding
Geo
Referencing
Radiometric
Corrections
GEO
BS
ARA
Angle Dep.
Removal
Mosaicking
• OpenBST v. 0.1.0
• RawDecoding
• perbeam_bs_from_snippets_using_median
PROCESSING METADATA

114. GITHUB
+
PYDRO 2019

115. EPOM

116. 118
E-LEARNING PYTHON
FOR OCEAN MAPPING (EPOM)

117. 119
▪ TO FAMILIARIZE WITH KEY PROGRAMMING CONCEPTS
▪ TO TEACH PROGRAMMING HABITS AND SKILLS
▪ TO INTRODUCE THE EFFECTIVE USE OF THE EXTENSIVE
HELP AND ONLINE RESOURCES AVAILABLE FOR PYTHON
▪ TO PROVIDE ENOUGH BASIC PYTHON SKILLS
EPOM GOALS
NOT A FULL COURSE ON HOW TO PROGRAM IN PYTHON!

118. 120
▪ SIMPLE TO LEARN
▪ INCREASINGLY POPULAR
▪ FREELY AVAILABLE

121. 123
TWO MAIN LINES OF ACTION
Programming Basics
with Python
26-Aug-2019
Fall Term begins
Foundations of Ocean
Mapping Data Science
Development &
Beta Testing
1-Jul-2019
Official Release

122. 124
DEVELOPMENT & BETA TESTING
▪ SERVER CURRENTLY HOSTED AT: HTTPS://EPOM.HYDROFFICE.ORG

123. 125
DEVELOPMENT & BETA TESTING
▪ AN INITIAL COLLECTION OF NOTEBOOKS:

124. 126
DEVELOPMENT & BETA TESTING
▪ AN INITIAL COLLECTION OF NOTEBOOKS:
▫ KEY CONCEPTS
▫ SUPPLEMENTARY INFO
▫ EXERCISES WITH SOLUTIONS

125. 127
CURRENT FEEDBACK
“I think these are a great place to start.”
“Very nicely done!”
“Nope this is pretty neat!”
“I really like how I can do this from home
without having to install any software.
“All the text is very readable and easy to
comprehend.”

126. 128
PUBLICLY AVAILABLE

127. 129
AVAILABLE IN PYDRO 2019

128. THANKS!
CONTACT US AT: DAMIAN.MANDA@NOAA.GOV, GMASETTI@CCOM.UNH.EDU