2 Prelaunch Assessment of the NG VCM.pptx

293 views
269 views

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
293
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • These define the fundamental statistics used to evaluate the VCM
  • This shows daytime only, the nighttime approach is fundamentally the same but with no visual bands available uses a different set of cloud detection tests.
  • The purpose of this slide is to emphasize the superiority of thin cirrus detection using band M9 on VIIRS over what is possible today from MODIS. This will be a key advantage for VIIRS that these results cannot reveal as band M9 is not properly emulated by MODIS today.
  • Yes this is an “eye chart” but it reveals the extent of quantitative results that can be obtained.
  • The key here is the reduction of PCPC pixels, as indicated by the reduction of light blue coloring in the figure on the lower right.
  • Another way of showing the reduction of PCPC pixels.
  • The discussion on this slide will focus on the last four rows of the first table with the four rows of the second table.
  • This and the next slide are the heart of Aerospace’s contribution to the VCM Cal/Val effort and our contribution to the presentation.
  • 2 Prelaunch Assessment of the NG VCM.pptx

    1. 1. Prelaunch Assessment of the Northrop Grumman VIIRS Cloud Mask<br />Thomas Kopp, The Aerospace Corporation<br />Keith Hutchison, Northrop Grumman<br />Andrew Heidinger, NOAA/STAR<br />Richard Frey, University of Wisconsin<br />IGARSS<br />25 July 2011<br />
    2. 2. Outline<br />Definitions of VIIRS Cloud Mask (VCM) contents and validation conditions<br />High level review of the VCM logic<br />Global results with the pre-launch VCM without any tuning<br />Quantitative Improvements Using the Northrop Grumman (NG) tuning tool<br />Methods for evaluating individual granules during Intensive Cal/Val (ICV) of the VCM<br />
    3. 3. VCM Contents<br />The VCM itself determines one of four cloud cover conditions for each pixel<br />Confidently Cloudy<br />Probably Cloudy<br />Probably Clear<br />Confidently Clear<br />All downstream EDR products, except for imagery, require the VCM as an input<br />Downstream products will use either the confidently cloudy or confidently clear condition<br />The probably clear/cloudy cases account for pixels that are not completely cloud covered but due either to the difficulty of the scene or partial clouds such as cumulus, are not sufficiently clear to reliably determine the conditions at the surface<br />
    4. 4. VCM Performance Metrics<br /><ul><li>Probability of Correct Typing (PCT)
    5. 5. PCT = (1 - Binary Cloud Mask Error) = {1 – [(VCM = conf. clear) & (Truth = conf. cloudy) OR ((VCM = conf. cloudy) & (Truth = conf. clear)]/[total #pixels in each geographic class – PCPC]
    6. 6. Cloud Leakage (CL)
    7. 7. CL = [(VCM = conf. clear) & (Truth = conf. cloudy)]/total #pixels in the geographic class
    8. 8. False Alarm Rate (FA)
    9. 9. FA = [(VCM = conf. cloudy) & (Truth = conf. clear)]/total #pixels in the geographic class
    10. 10. Fraction of Pixels Classified as Probably Clear/Cloudy (FPCPC)
    11. 11. FPCPC = [(VCM = prob. clear) or (VCM = prob. cloudy)]/total #pixels in the geographic class</li></li></ul><li>Overview of VCM Approach<br />There are five possible processing paths in the VCM algorithm for the analysis of SDR data collected in daytime conditions<br />
    12. 12. Pre-launch, Pre-tuned Global VCM Results<br /><ul><li>VCM version used 1.5.0.48 from 2009
    13. 13. The initial thresholds were used, the VCM for this testing was not tuned
    14. 14. Comparisons made with collocated MOD35 C6 cloud mask and CALIOP matchups for comparison</li></ul>Cloudy for the VCM in this case included probably cloudy pixels<br />Clear for the VCM in this case included probably clear pixels<br />Compared only 1-km CALIOP segments with either 0% or 100% cloud cover<br />Resulted in approximately 15 million collocations per month<br />
    15. 15. Caveats and Notes to Global VCM Results<br /><ul><li>Results intended to show “where we were” in late 2009
    16. 16. Neither of the two I-band tests could be simulated using the proxy data, a significant source of error that will not be quantified until the post-launch validation of the VCM
    17. 17. Thin cirrus has a major impact on the results
    18. 18. Analysis limited to near-nadir views (MODIS viewing zenith angle of +/- 20 degrees)
    19. 19. Hit rate = (# agree cloud + # agree clear) / total #</li></ul>Hanssen-Kuiper Skill Source (HKSS) = (# agree cloud * # agree clear) – (# disagree cloud * # disagree clear) /<br /> (# agree clear + # disagree clear) * (# agree cloud + # disagree cloud) <br /><ul><li>Results follow on the next few slides</li></li></ul><li>VCM versus CALIOP, Global Results<br />
    20. 20. VCM versus CALIOP, Polar and Non-Polar Results<br />
    21. 21. VCM versus CALIOP, Day Results<br />
    22. 22. VCM versus CALIOP, Night Results<br />
    23. 23. VCM versus CALIOP, Land and Desert Results<br />
    24. 24. VCM Designed to Exploit VIIRS 1.38-µm Data<br />MODIS OOB<br />Response is as large as the in-band response<br />VIIRS OOB Response is orders of magnitude <br />less<br />MODIS vs VIIRS TOA Radiances <br />MODIS vs VIIRS RSRs<br />Thin cirrus clouds will be more readily detected with VIIRS data than in MODIS <br />
    25. 25. VCM Versus Heritage Performance, COT > 1.0<br />VCM and heritage performance are comparable when thin cirrus clouds are eliminated from the results<br />
    26. 26. One Year Means of Hit Rates and Skill<br />
    27. 27. Pre-Launch Tuning Approach<br />Pre-launch tuning is based on 14 granules which employed Global Synthetic Data (GSD)<br />Of these 14, 11 contained land backgrounds<br /><ul><li>These granules covered each VCM geographic type and ranged from straightforward to difficult scenes
    28. 28. GSD provides unique data to set the mid-point thresholds
    29. 29. Typical methods of tuning, using on-orbit sensor data, rely upon 100% cloudy and 100% cloud free distributions
    30. 30. GSD alone allows cloud distributions to be evaluated at the mid-point (50% cloudy) condition
    31. 31. GSD allows setting thresholds and then minimize the distance between the confidently cloudy and confidently clear thresholds </li></li></ul><li>Advantages Using Global Synthetic Data (GSD)<br /><ul><li>GSD allows testing of the tuning process with proxy data and then apply the procedure to VIIRS-unique data
    32. 32. Tuning process is validated by :
    33. 33. (1) tuning with GSD truth data developed with MODIS Relative Spectral Responses (RSR)
    34. 34. (2) tested in the VCM using MODIS granules
    35. 35. (3) quantitatively evaluated using manually-generated cloud data of the MODIS data
    36. 36. Tuning for VIIRS data is then completed by examining changes in cloud distribution for each test in GSD truth data using the VIIRS RSR</li></li></ul><li>Initial VCM Results Showed Following Needs<br />Reduce the number of probably clear and probably cloudy (PCPC) classifications by adjusting the overall cloud confidence threshold <br />Identify tests that generated the highest percentage of false alarms for each VCM background condition and tune the mid-point thresholds (i.e. 50% cloud cover condition) accordingly – only possible with GSD<br />Further reduce the number of PCPC classifications, as necessary, by adjusting the distance between the mid-point thresholds of a given individual cloud test and the low and/or high threshold using cloud distributions in the GSD. <br />
    37. 37. Initial Untuned VCM Performance - Land<br />
    38. 38. Overview of the Pre-Launch Tuning Process<br />Identify the tests causing the largest number of errors<br />Use GSD with MODIS RSRs to generate cloud cover distributions for the cloud detection tests identified above<br />Generate distributions for 0%, 50%, and 100% cloud cover <br />Set key mid-point threshold using the 50% cloud cover, then minimize low- and high thresholds<br />Update VCM using these thresholds<br />Execute the updated algorithm on the set of MODIS granules<br />Evaluate the performance using the manually generated cloud masks<br />Assess the changes in performance<br />
    39. 39. Example for a Case With Too Many PCPC Pixels<br />Manually-Generated Mask<br />MODA.2001.196.1755<br />Qthresh = 99%<br />Qthresh = 90%<br />Land - Pre<br />Land - Post<br />
    40. 40. Specific Cloud Detection Case, GEMI Test (Land)<br />Changed from 1.95 to 1.87<br />Changed from 1.90 to 1.82<br />Changed from 1.85 to 1.78<br />
    41. 41. Quantitative Impacts – GEMI Results<br />Previous untuned results<br />
    42. 42. Tests Improved by the Pre-Launch Tuning Effort<br />Reflectance test over desert (M1)<br />Reflectance test over land (M5)<br />Reflectance test over water (M7)<br />Ratio test over land (GEMI)<br />Ratio test over water (M7/M5)<br />Mid-Wave minus long wave infrared over snow (M12 – M15)<br />Mid-Wave infrared difference over snow (M12 – M13) <br />
    43. 43. Performance After Daytime Tuning Completed<br />Tuned VCM: November 2010<br />Untuned VCM: March 2010<br />
    44. 44. Tool for Visualization of the VCM<br />The previous analyses reveal quantitative aspects of the VCM, but lack context<br />Historically the capability to visualize the output from each individual cloud detection test has been used operationally at the Air Force Weather Agency<br />Key to a useful visualization are two fundamental factors<br />It must overlay each test on applicable imagery<br />It must contain the reflectance/brightness temperatures used within the cloud mask<br />This reveals if any bands have bad or saturated values<br />The visualization should also note if any degraded conditions of note exist in the scene<br />These include aerosols, sun glint, and shadows<br />The following pair of slides show this capability<br />
    45. 45. Aerospace Visualization Tool – Example I<br />
    46. 46. Aerospace Visualization Tool – Example 2<br />
    47. 47. Conclusion<br />Pre-launch validation of the VCM uses three different approaches to verify the VCM will meet expectations<br />Large scale quantitative analysis<br />Small scale quantitative analysis via GSD<br />Visualization of individual granules with each component cloud detection test<br />Results show promise that the VCM will meet or exceed its requirements<br />Each of these methods will be employed in some form post-launch, though we will no longer need GSD as actual VIIERS data will be available<br />

    ×