Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pydro & HydrOffice: Open Tools for Ocean Mappers

1,874 views

Published on

Workshop given by Damian Manda (NOAA Office of Coast Survey) and Giuseppe Masetti (UNH Center for Coastal and Ocean Mapping/NOAA-UNH Joint Hydrographic Center) on March 18, 2019 at the US Hydro Conference in Biloxi, MS, USA.

Published in: Science
  • Fantastic job, congratulations
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Thank you for sharing Giuseppe
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Pydro & HydrOffice: Open Tools for Ocean Mappers

  1. 1. PYDRO & HYDROFFICE OPEN TOOLS FOR OCEAN MAPPERS D. MANDA AND G. MASETTI V1
  2. 2. WHAT IS PYDRO?
  3. 3. A NOAA Python Distribution Pydro Universe A Collection of Open-Source Hydrographic Tools A Growing Community of People with Interests in Ocean Mapping
  4. 4. Pydro GIS 1994 2015Closed Source
  5. 5. 1994 2015Closed Source 2016
  6. 6. 1994 2015Closed Source 2016 2016: TRANSITIONING TO OPEN ▪ COLLABORATION WITH THE UNIVERSITY OF NEW HAMPSHIRE ▪ RELEASE OF NOAA BUILT APPS ▪ OPEN UNDERLYING CODE FOR USE
  7. 7. INSTALLATION ▪ https://svn.pydro.noaa.gov/
  8. 8. WHAT IS HYDROFFICE?
  9. 9. HydrOffice 10 A research framework for ocean mapping to facilitate research-to-operation (R2O)
  10. 10. 11 CCOM/JHC Italy 2013 Initial thoughts
  11. 11. A framework of libraries and tools for Ocean Mapping 12 Quickly prototype and test innovative ideas Ease the transition from research to operation Ref.: G. Masetti, Wilson, M. J., Calder, B. R., Gallagher, B., and Zhang, C., “Research-driven Tools for Ocean Mappers”, Hydro Int., vol. 21, 5. GeoMares, 2017. 2015 White Paper
  12. 12. July 2016 Site Review
  13. 13. March 2017 SSM release
  14. 14. 15 CCOM/JHC Italy January 2018 Team Change
  15. 15. Several Apps ▪ BAG Explorer 16 ▪ ENCx ▪ QC Tools ▪ Huddl ▪ StormFix ▪ CA Tools ▪ Bress ▪ …
  16. 16. HYDROFFICE APPS 17 PYTHON SCIENTIFIC STACK OCEAN MAPPING LIBS & SCRIPTS
  17. 17. 18 OCS-UNH CO-DEVELOPMENT
  18. 18. Pydro Universe Stand-alone Apps Python Packages www.nauticalcharts.noaa.gov www.hydroffice.org GitHub/PyPi/Conda 19 Distribution
  19. 19. HydrOffice.org ▪ Mobile-first, dynamic website ▪ Based on Django & JavaScript ▪ Per-tool Home Page ▫ Info ▫ Stand-alone downloads ▫ Manuals ▫ Embedded tutorials 20
  20. 20. HydrOffice.org 21(*) Google Analytics, Number of Sessions, January 2018, location filtered: Durham, Silver Spring, unset.
  21. 21. SOUND SPEED MANAGER AND SMARTMAP
  22. 22. Sound Speed Manager A ready-to-go and free solution to ease the management of sound speed profiles for ocean mapping 23Ref.: Masetti, G. et al., Sound SpeedManager: An open-source application to manage sound speed profiles, International Hydrographic Review, vol. 17, pp. 31-40, 2017.
  23. 23. Collaborative Effort 24
  24. 24. Collaborative Effort 25
  25. 25. Open Source 26
  26. 26. Liberal License 27
  27. 27. 28 ✓Support ✓Open ✓Free
  28. 28. Format Converter 29
  29. 29. 30 MVP SIS
  30. 30. 31
  31. 31. 32
  32. 32. 33
  33. 33. 34
  34. 34. 35
  35. 35. 36
  36. 36. 37
  37. 37. 38
  38. 38. SmartMap A tool to evaluate the effects of oceanographic variability on mapping surveys 39Ref.: Masetti,, G., Kelley, J., Johnson, P., and Beaudoin, J., A Ray-Tracing Uncertainty Estimation Tool for Ocean Mapping, IEEE Access. IEEE, pp. 1-9, 2017.
  39. 39. 40
  40. 40. 41
  41. 41. SmartMap Algorithm ▪ Constant Gradient: ▪ Snell Law: 42
  42. 42. SmartMap WebGIS ▪ RTOFS + WOA13 ▪ Animation ▪ Survey Planner 43
  43. 43. Gulf of Maine OFS (beta) 44
  44. 44. TCARI
  45. 45. Tidal Constituent And Residual Interpolation ▪ Created in 1995 at the Coast Survey Development Lab ▪ Overcomes weaknesses in traditional tidal models ▪ Interpolates across the project area: ▫ Tidal constituents ▫ Residual water level, and ▫ Desired vertical datum. 46
  46. 46. 47
  47. 47. Disadvantages of discrete zoning: ▪ Does not capture subtle changes in tides ▪ Does not capture meteorological, hydrological and density effects ▪ Introduces steps at boundary polygons ▪ Polygons are drawn by hand 48
  48. 48. 50
  49. 49. 51
  50. 50. 52
  51. 51. QC/CA TOOLS
  52. 52. Ping-to-chart (very simplified) Marine Chart Division Hydro Branches Field Survey deliverables (grids, features, etc.) Data archive and public access Chart Products! Rapid Survey Assessment (RSA) Survey Review (SAR) National Center for Environmental Information Processing Branch deliverables (grids, features, reports, etc.)
  53. 53. QC Tools Performs automated quality control checks on surfaces and final feature files. 55
  54. 54. Workflow 56 GDAL Open Formats ▪ BAG ▪ S-57 ▪ Shapefile ▪ KML
  55. 55. Survey Review 57 Detect Fliers Detect Holidays Grid QA Scan Designated Scan Features VALSOU Check SBDARE Export Submission Checks Uncertainty Calculator * Rock or Islet Oracle * * Standalone Tools
  56. 56. 58
  57. 57. 59 Object Detection: >3 collinear nodes Full Coverage: >3x3 nodes All Holes
  58. 58. Grid QA Outputs
  59. 59. Grid QA Outputs
  60. 60. Rock or Islet? 62 ?
  61. 61. 63
  62. 62. 64
  63. 63. 65 Which tool to improve first?
  64. 64. CA Tools Performs chart adequacy tasks by comparing survey data and current ENCs 66
  65. 65. 67
  66. 66. 68 z
  67. 67. 69 z
  68. 68. 70 z 10.2 4.5 4.8
  69. 69. 71 z 10.2 4.5 4.8 5.1
  70. 70. 72 z 10.2 4.8 5.1 4.5
  71. 71. 73 z 10.2 4.8 5.1 4.5
  72. 72. 74 z 10.2 4.5 4.8 5.1
  73. 73. Tilted Triangle 75
  74. 74. Flat Triangles > Point-In-Polygon Test 76
  75. 75. 77
  76. 76. 78
  77. 77. CHARLENE
  78. 78. With Charlene… ▪ Automated data management ▪ Set-and-forget processing ▪ Single interface for user ▪ 40 page SOP encompassed in a few dropdown boxes ▪ Instruction/SOP can be done from a screenshot 80
  79. 79. 81
  80. 80. 82
  81. 81. 83
  82. 82. XMLDR
  83. 83. 85
  84. 84. 86
  85. 85. 87
  86. 86. 88 XmlDR Output
  87. 87. OTHER PYDRO TOOLS
  88. 88. 90
  89. 89. POSPac AutoQC
  90. 90. 92 Sonar Acceptance Tools
  91. 91. OTHER HYDROFFICE TOOLS
  92. 92. BAG Explorer 95
  93. 93. StormFix 96 ARTIFACTS DETECTION ARTIFACTS REDUCTION BACKSCATTER MOSAICKING ANGULAR RESPONSE ANALYSIS Ref.: G. Masetti et al., “How to Improve the Quality and the Reproducibility for Acoustic Seafloor Characterization”, GeoHab 2017. p. Nova Scotia, Canada, 2017.
  94. 94. StormFix: How it works?
  95. 95. 98
  96. 96. 99
  97. 97. 100
  98. 98. 101 Just Removal vs Randomization Schema
  99. 99. BRESS ▪ Preliminary segmentation from co-located DEMs and backscatter mosaics ▪ Based on principles of: ▫ Topographic openness ▫ Pattern recognition ▫ Texture classification 102Ref.: G. Masetti, Mayer, L. A., and Ward, L. G., “A Bathymetry- and Reflectivity-Based Approach for Seafloor Segmentation”, Geosciences, vol. 8(1). MDPI, 2018.
  100. 100. 104 Landform ClassificationLocal Ternary Patterns
  101. 101. 105 Output SegmentsArea Kernels
  102. 102. 106
  103. 103. 107
  104. 104. WHAT’S COMING?
  105. 105. OPENBST
  106. 106. Raw Data File Caris SIPS QPS FMGT Ifremer SonarScope Beams Pings Pings Pings Beams Beams Reflectivity[dB]
  107. 107. Reflectivity[dB] Average Initial Reflectivity per Beam NoData Values for Initial Reflectivity per Beam Beams Nr.ofValidData Histogram of Initial Reflectivity per Beam Reflectivity [dB]
  108. 108. A SHIFT IS REQUIRED FROM THE EXISTING CLOSED-SOURCE APPROACH AN OPEN-SOURCE AND METADATA-RICH MODULAR IMPLEMENTATION NOT COMPETING WITH COMMERCIAL SOLUTIONS, BUT A SET OF COMMUNITY-VETTED, REFERENCE ALGORITHMS
  109. 109. EXTENDS SONAR-NETCDF4 FOLLOWS CF METADATA CONVENTIONS
  110. 110. RAW MOS Raw Data Decoding Geo Referencing Radiometric Corrections GEO BS ARA Angle Dep. Removal Mosaicking • OpenBST v. 0.1.0 • RawDecoding • perbeam_bs_from_snippets_using_median PROCESSING METADATA
  111. 111. GITHUB + PYDRO 2019
  112. 112. EPOM
  113. 113. 118 E-LEARNING PYTHON FOR OCEAN MAPPING (EPOM)
  114. 114. 119 ▪ TO FAMILIARIZE WITH KEY PROGRAMMING CONCEPTS ▪ TO TEACH PROGRAMMING HABITS AND SKILLS ▪ TO INTRODUCE THE EFFECTIVE USE OF THE EXTENSIVE HELP AND ONLINE RESOURCES AVAILABLE FOR PYTHON ▪ TO PROVIDE ENOUGH BASIC PYTHON SKILLS EPOM GOALS NOT A FULL COURSE ON HOW TO PROGRAM IN PYTHON!
  115. 115. 120 ▪ SIMPLE TO LEARN ▪ INCREASINGLY POPULAR ▪ FREELY AVAILABLE
  116. 116. 123 TWO MAIN LINES OF ACTION Programming Basics with Python 26-Aug-2019 Fall Term begins Foundations of Ocean Mapping Data Science Development & Beta Testing 1-Jul-2019 Official Release
  117. 117. 124 DEVELOPMENT & BETA TESTING ▪ SERVER CURRENTLY HOSTED AT: HTTPS://EPOM.HYDROFFICE.ORG
  118. 118. 125 DEVELOPMENT & BETA TESTING ▪ AN INITIAL COLLECTION OF NOTEBOOKS:
  119. 119. 126 DEVELOPMENT & BETA TESTING ▪ AN INITIAL COLLECTION OF NOTEBOOKS: ▫ KEY CONCEPTS ▫ SUPPLEMENTARY INFO ▫ EXERCISES WITH SOLUTIONS
  120. 120. 127 CURRENT FEEDBACK “I think these are a great place to start.” “Very nicely done!” “Nope this is pretty neat!” “I really like how I can do this from home without having to install any software. “All the text is very readable and easy to comprehend.”
  121. 121. 128 PUBLICLY AVAILABLE
  122. 122. 129 AVAILABLE IN PYDRO 2019
  123. 123. THANKS! CONTACT US AT: DAMIAN.MANDA@NOAA.GOV, GMASETTI@CCOM.UNH.EDU

×