Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Backscatter Working Group Software Inter-comparison Project Requesting and Comparing Intermediate Backscatter Processing Results

323 views

Published on

Backscatter mosaics of the seafloor are now routinely produced from multibeam sonar data, and used in a wide range of marine applications. However, significant differences (up to 5 dB) have been observed between the levels of mosaics produced by different software processing a same dataset. This is a major detriment to several possible uses of backscatter mosaics, including quantitative analysis, monitoring seafloor change over time, and combining mosaics. A recently concluded international Backscatter Working Group (BSWG) identified this issue and recommended that “to check the consistency of the processing results provided by various software suites, initiatives promoting comparative tests on common data sets should be encouraged […]”. However, backscatter data processing is a complex (and often proprietary) sequence of steps, so that simply comparing end-results between software does not provide much information as to the root cause of the differences between results.

In order to pinpoint the source(s) of inconsistency between software, it is necessary to understand at which stage(s) of the data processing chain do the differences become substantial. We have invited willing software developers to discuss this framework and collectively adopt a list of intermediate processing steps. We provided a small dataset consisting of various seafloor types surveyed with the same multibeam sonar system, using constant acquisition settings and sea conditions, and have the software developers generate these intermediate processing results, to be eventually compared. If the experiment proves fruitful, we may extend it to more datasets, software and intermediate results. Eventually, software developers may consider making the results from intermediate stages a standard output as well as adhering to a consistent terminology, as advocated by Schimel et al. (2018). To date, the developers of four software (Sonarscope, QPS FMGT, CARIS SIPS, MB Process) have expressed their interest in collaborating on this project.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Backscatter Working Group Software Inter-comparison Project Requesting and Comparing Intermediate Backscatter Processing Results

  1. 1. Backscatter Working Group Software Inter-comparison Project Requesting and Comparing Intermediate Backscatter Processing Results Project Facilitators: Mashkoor Malik, NOAA, USA Marc Roche, ECONOMIE, Belgium Julian Le Deunf, SHOM, France Giuseppe Masetti, CCOM UNH, USA Alexandre Schimel, NIWA New Zealand Margaret Dolan, NGU, Norway Shallow Survey Conference 1 October 2018 Project Collaborators: SonarScope, IFREMER FMGT, QPS HIPS & SIPS, Teledyne CARIS MB Process, Curtin University, CMST
  2. 2. Why? ● Validation of backscatter processing steps ○ Using same data set, may result in different final products ○ Need input from developers to document the processing chain better ● Backscatter working group (BSWG, 2015) recommended: “to check the consistency of the processing results provided by various software suites, initiatives promoting comparative tests on common data sets should be encouraged [...]” (Lurton and Lamarche, 2015: BSWG report ). Proprietary ProcessingRaw data Mosaic Angular response
  3. 3. Example of end user frustration ! (Mosaic to mosaic comparison) End result of backscatter mosaic offers little insight as What went wrong and where? Lucieer et al. (2017); Roche et al. (2018)
  4. 4. What are we proposing? A collaborative project between data users and software developers: ● Goal 1: Collectively come to an agreement over processing stages and reporting of intermediate results; ○ Launched project at GeoHab 2018 (May 2018) ○ A mini-workshop with software developers conducted (June 2018) ○ Provided common data set (June 2018) ○ Software vendors provided processing results (September 2018) ● Goal 2: To identify discrepancies in the processing chain and explain differences in results when processed with different processing tools; ○ Same dataset processed by software developers ○ Compare intermediate processing results ● Goal 3: To develop consensus among software developers about adopting a standard processing chain, nomenclature and metadata.
  5. 5. Backscatter data processing sequence (Schimel et al. 2018)
  6. 6. Backscatter data processing sequence (Schimel et al. 2018) Current outputs
  7. 7. Current outputs Current outputs
  8. 8. BSWG recommended processing stages Current outputs Recommended outputs Metadata Metadata
  9. 9. BSWG recommended processing stages Current outputs Recommended outputs Metadata Metadata Available Sept 2018 CARIS and Sonar Scope (Augustin talk in this session) implemented (some) intermediate results output
  10. 10. The inter-comparison dataset Kongsberg EM302 Johnston Atoll area NOAA - USA Teledyne 7125 Wembury Bay UK Shallow survey 2015 Kongsberg EM710 and EM3002 Carré Renard reference area SHOM - FRANCE Kongsberg EM2040 Kwinte reference area FPS Economy - BELGIUM
  11. 11. Phase I: Request intermediate results from software ASCII export in Ping Beam geometry with backscatter as provided by sonar and processed results prior to mosaicking (BL0 and BL3) Vessel Data acquisition Agency Location Weather Date Depth range EM 2040 RV Simon Stevin SIS FPS Economy Kwinte reference area Calm Dec 2016 23-26 m EM 3002 HSL Guillemont SIS SHOM Carre Renard area, Brest Bay, France Calm Jan 2010 18-22 m EM 710 BH2 Borda SIS SHOM Carre Renard area, Brest Bay, France Calm Feb 2013 18-22 m EM 302 Okeanos Explorer SIS NOAA Johnston Atoll near Hawaii, USA Rough Aug 2017 ~3000 m Reson 7125 s7k Shallow survey 2015 Plymouth, UK Calm 2015 < 10 m
  12. 12. Processing status (Sept 2018) Software KB 710 KB 302 KB 3002 KB 2040 Reson 7125 Sonar Scope FMGT CARIS Curtin Hypack Swath Ed MB System Available No results Awaiting confirmation
  13. 13. Backscatter results provided by different vendors Software SonarScope FMGT CARIS Curtin # columns 31 12 17 11 Time stamp Ping Time Timestamp Ping Time Ping # Ping First ping = 0 Ping Number First ping = 1 Ping Ping Number Beam # Beam First beam = 1 Beam Number Beam Beam Number Beam location (Lat / Long) Latitude/Longitude Latitude /Longitude Longitude /Latitude Longitude / Latitude Beam location (E / N) GeoX / GeoY Easting / Northing Easting / Northing Easting / Northing Beam depth BathyRT Depth Depth Incidence angle IncidenceAngles True Angle IncidentAngle Incidence Angle BS as read from data files ReflecKM Backscatter Value BLO Backscatter value BS processed angular response ReflecSSc Corrected Bacskcatter Value BL3 Corr Backscatter Value
  14. 14. RV Simon Stevin EM2040 single Rx L0005_20160412_104116.all (300kHz, 216µs) Kwinte reference gravel and sandy gravel area black dots = GIS beam plot (lat long) white line = navigation yellow = calculation area Processing of data from Sonarscope, FMGT and CARIS: 1. GIS integration using Lat Long associated with each ping/beam 2. Extraction of the data inside the yellow calculation area (homogeneous area, bs reference subarea!) 3. Per beam number, calculation of the average value of the different backscatter provided: • SonarScope: • ReflecKM • ReflecSSc • FMGT: • Bsvalue • Bscorvalue • CARIS: • BL0 • BL3
  15. 15. RV Simon Stevin EM2040 single Rx L0005_20160412_104116.all (300kHz, 216µs) Kwinte reference gravel and sandy gravel area BoxPlot : outliers 25% 50% 75% outliers 1 % 99 % (A selected area) -17 dB -15 dB - 11.5 dB
  16. 16. (First 100,000 data points)
  17. 17. Why is CARIS incidence angles different ?
  18. 18. Within 5 cm RV Simon Stevin EM2040 single Rx L0005_20160412_104116.all (300kHz, 216µs) EM 2040 Depth (One Ping)
  19. 19. Within 15 dB RV Simon Stevin EM2040 single Rx L0005_20160412_104116.all (300kHz, 216µs) EM 2040 Backscatter (One Ping)
  20. 20. RV Simon Stevin EM2040 single Rx L0005_20160412_104116.all (300kHz, 216µs) Kwinte reference gravel and sandy gravel area
  21. 21. EM302 L0213_20170717_112534_EX1706.all (main frequency =27.75kHz, pl = 40ms) Johnston Atoll area Processing of data from Sonarscope, FMGT and CARIS: 1. GIS integration using Lat Long associated with each ping/beam 2. Per beam number, calculation of the average value of the different backscatter provided using all the data: • SonarScope: • ReflecKM • ReflecSSc • FMGT: • Bsvalue • Bscorvalue • CARIS: • BL0 • BL3 Note: considering the heterogeneity of the seabed, a sub calculation area should be select in order to keep a good coherency of the statistics. SSc Reflec KM
  22. 22. EM302 L0213_20170717_112534_EX1706.all (main frequency =27.75kHz, pl = 40ms) Johnston Atoll area (One survey line) - 26.5 dB - 26 dB - 22 dB - 21.5 dB - 27.2 dB - 24 dB BL0 BL3
  23. 23. EM302 L0213_20170717_112534_EX1706.all (main frequency =27.75kHz, pl = 40ms) Johnston Atoll area
  24. 24. EM302 L0213_20170717_112534_EX1706.all (main frequency =27.75kHz, pl = 40ms) Johnston Atoll area (One survey line)
  25. 25. (First 100,000 data points) EM302 L0213_20170717_112534_EX1706.all (main frequency =27.75kHz, pl = 40ms) Johnston Atoll area
  26. 26. EM710 0002_20130214_091514_borda.all (main frequency=81kHz, pl=0.3ms) SHOM reference area Processing of data from Sonarscope, FMGT and CARIS: 1. GIS integration using Lat Long associated with each ping/beam 2. Per beam number, calculation of the average value of the different backscatter provided using all the data: • SonarScope: • ReflecKM • ReflecSSc • FMGT: • Bsvalue • Bscorvalue • CARIS: • BL0 • BL3 Note: Due to the high homogeneity of the seabed in the Carré Renard area, defining a sub area is not required. FMGT Mosaic (beam time series, default settings)
  27. 27. EM710 0002_20130214_091514_borda.all (main frequency=81kHz, pl=0.3ms) SHOM reference area -19.3 dB -14.5 dB - 10.4 dB
  28. 28. EM710 0002_20130214_091514_borda.all (main frequency=81kHz, pl=0.3ms) SHOM reference area
  29. 29. EM710 0002_20130214_091514_borda.all (main frequency=81kHz, pl=0.3ms) … area
  30. 30. EM710 0002_20130214_091514_borda.all (main frequency=81kHz, pl=0.3ms) SHOM reference area
  31. 31. RESON 7125 20140729_082527_SMB Owen.s7k Shallow Survey 2015 Processing of data from Sonarscope, FMGT and Curtin 1. GIS integration using Lat Long associated with each ping/beam 2. Per beam number, calculation of the average value of the different backscatter provided using all the data: • MB System : • BL0 • BL3 • FMGT: • Bsvalue • Bscorvalue • CARIS: • BL0 • BL3 Reson 7125 area FMGT Mosaic
  32. 32. 40 dB - 55 dB 40 dB RESON 7125 20140729_082527_SMB Owen.s7k Shallow Survey 2015
  33. 33. RESON 7125 20140729_082527_SMB Owen.s7k Shallow Survey 2015
  34. 34. Discussions needed 1. Large unexplained differences among results BL0 and BL3 2. An initial step to show that differences exist 3. More work needed to identify reasons of these differences: Sonar dependent? Depth dependent? Differences in processing methodology? 4. Involvement of sonar manufacturers to clarify data acquisition assumptions?
  35. 35. Next Steps ● Revisit calculations of BL0 ● Request other intermediate processing results ● 2nd round of processing after agreeing upon the processing approaches ● An independent assessment of processed results ● Other software ?
  36. 36. We need your support !! ● Users: To demand results processed by different tools should agree reasonably with each other ○ Accuracy standards of backscatter still distant goal ● Software developers: To work together to implement best practices for backscatter processing ● BSWG: Provide a platform to facilitate these discussions
  37. 37. Questions Mashkoor Malik (mashkoor.malik@noaa.gov) Alexandre C. G. Schimel (alexandre.schimel@niwa.co.nz) Marc Roche (Marc.Roche@economie.fgov.be) Giuseppe Masetti (gmasetti@ccom.unh.edu) Margaret Dolan (Margaret.Dolan@ngu.no) Julian Le Deunf (julian.le.deunf@shom.fr) Thanks to software developers & BSWG
  38. 38. References cited Lamarche G, Lurton X. 2017. Recommendations for improved and coherent acquisition and processing of backscatter data from seafloor-mapping sonars. Mar. Geophys. Res. 39:5-22. Lucieer V, Roche M, Degrendele K, Malik M, Dolan M, Lamarche G. 2017. User expectations for multibeam echo sounders backscatter strength data-looking back into the future. Mar. Geophys. Res. 39:23-40. Lurton X, Lamarche G. 2015. Backscatter measurements by seafloor-mapping sonars. Guidelines and recommendations. Retrieved 27th April 2018 from http://geohab.org/wp-content/uploads/2013/02/BWSG-REPORT-MAY2015.pdf. Roche M, Degrendele K, Vrignaud C, Loyer S, Le Bas T, Augustin J-M, Lurton X. 2018. Control of the repeatability of high frequency multibeam echosounder backscatter by using natural reference areas. Mar. Geophys. Res. 39: 89-104. Schimel ACG, Beaudoin J, Parnum IM, Le Bas T, Schmidt V, Gordon K, Ierodiaconou D. 2018. Multibeam sonar backscatter data processing. Mar. Geophys. Res. 39:121-137.

×