1.
© Crown copyright Met Office
WAFC CAT verification
Objective verification of GRIB CAT forecasts
Dr Philip G Gill, WAFC Science Meeting, Washington, 20 April 2009

2.
© Crown copyright Met Office
Contents
This presentation covers the following areas
• Introduction
• SIGWX forecast comparison
• Aircraft data
• Verification methodology
• Verification results
• Summary
• Further improvements

3.
© Crown copyright Met Office
Introduction
• What – Objective verification of gridded binary (GRIB)
and significant weather (SIGWX) Clear Air Turbulence
(CAT) forecasts
• Where – Global verification
• When – November 2008 to January 2009
• Why – To demonstrate the quality of the new GRIB
forecasts using objective verification.
• How – Verification against aircraft observations from the
Global Aircraft Data Set (GADS)

4.
© Crown copyright Met Office
SIGWX and GRIB CAT forecasts
• SIGWX chart
• New GRIB forecast

5.
© Crown copyright Met Office
Comparison of SIGWX charts
UK US UK&US

6.
© Crown copyright Met Office
SIGWX CAT forecast comparison
• One month UK-US comparison (January 2009)
• Average coverage of globe UK ~6%, US ~3%
• Percentage overlap of all forecasts between UK
and US ~20%
Areas forecast by both UK and US
Areas forecast by UK but not US
Areas forecast by US but not UK

7.
© Crown copyright Met Office
GRIB forecast comparison
produced by HKO

8.
© Crown copyright Met Office
Global Aircraft Data Set
• Archive of aircraft data set up by Joel Tenenbaum (State University
of New York)
• British Airways fleet of Boeing 747-400 aircraft
• Global coverage, but flights mainly over northern hemisphere
• Automated aircraft observations every 4 seconds
• Indicator of turbulence derived from vertical acceleration, aircraft
mass, altitude and airspeed called the derived equivalent vertical
gust (DEVG).

9.
© Crown copyright Met Office
GADS Data coverage
10-19 January 2007

10.
© Crown copyright Met Office
Verification methodology

11.
© Crown copyright Met Office
Forecast assessment
• Turbulent/non turbulent event defined on
10min aircraft track ~120km - approx grid
size
• Forecast turbulent event – CAT potential >=
Threshold
• Observed turbulent event - DEVG>=2m/s
• Construct 2x2 contingency tables for each
threshold
• Sum entries in contingency tables over the
verification period
• Produce a Relative Operating Characteristic
(ROC) curve by plotting the Hit rate against
False alarm rate for each threshold.
Turbulence
observed
No
turbulence
observed
Turbulence
forecast
Hit False alarm
No
turbulence
forecast
Miss Correct
rejection
2x2 contingency table

12.
© Crown copyright Met Office
Results UK GRIB and SIGWX
ROC Curve UK GRIB and SIGWX BUFR CAT
November 2008-January 2009 Global GADS data
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
False alarm rate
Hit
rate
GRIB T+24
SIGWX BUFR T+24
~ 300 000 events

13.
© Crown copyright Met Office
Variation with forecast range
ROC curve January 2009 UK GRIB against Global GADS data
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rate
T+6
T+12
T+18
T+24
~100 000 events

14.
© Crown copyright Met Office
Latitudinal variation UK GRIB
ROC curve 20N to 50N GRIB and SIGWX BUFR against GADS data
T+24 CAT November 2008 - January 2009
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rate
GRIB T+24
SIGWX BUFR T+24
ROC curve 50N to 90N GRIB and SIGWX BUFR against GADS data
T+24 CAT November 2008 - January 2009
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rate
GRIB T+24
SIGWX BUFR T+24
ROC curve 50S to 20S GRIB and SIGWX BUFR against GADS data
T+24 CAT November 2008 - January 2009
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rate
GRIB T+24
SIGWX BUFR T+24
ROC curve 20N to 20N GRIB and SIGWX BUFR against GADS data
T+24 CAT November 2008 - January 2009
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rate
GRIB T+24
SIGWX BUFR T+24
50N to 90N
~120 000 events
50S to 20S
~8 000 events
20N to 50N
~130 000 events
20S to 20N
~37 000 events

15.
© Crown copyright Met Office
UK and US Nov 2008 and Jan 2009 ROC curve
ROC curve global CAT T+24 forecasts verified against GADS data
Nov 2008 and Jan 2009
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rate
EGRR GRIB T+24
KKCI GRIB T+24
EGRR SIGWX BUFR T+24
KKCI SIGWX BUFR T+24
~200 000 events

16.
© Crown copyright Met Office
UK and US latitudinal variation
ROC curve CAT T+24 forecasts 50N to 90N verified against GADS data
Nov 2008 and Jan 2009
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rate
EGRR GRIB T+24
KKCI GRIB T+24
EGRR SIGWX BUFR T+24
KKCI SIGWX BUFR T+24
ROC curve CAT T+24 forecasts 20N to 50N verified against GADS data
Nov 2008 and Jan 2009
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rate
EGRR GRIB T+24
KKCI GRIB T+24
EGRR SIGWX BUFR T+24
KKCI SIGWX BUFR T+24
~60 000 events ~60 000 events

17.
© Crown copyright Met Office
UK GRIB and SIGWX
ROC Curve UK GRIB and SIGWX BUFR CAT
November 2008-January 2009 Global GADS data
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rate
GRIB T+24
SIGWX BUFR T+24
SIGWX SWAM T+24
~300 000 events
SIGWX Automation (SWAM) -
UK automated SIGWX chart
production system based on
GRIB data

18.
© Crown copyright Met Office
Summary of results
• Both UK and US GRIB products show more skill than the
manual SIGWX products.
• Global UK and US GRIB CAT forecasts score similarly
• Slight difference in scores as forecast range increases
• Some differences in scores at individual latitude bands –
best performance between 20N and 90N.
• UK CAT coverage on SIGWX charts greater than the US

19.
© Crown copyright Met Office
Further improvements
• Automate verification process
• Produce statistics by ICAO and WMO regions
• Improve consistency of forecasts by analysing
verification data and altering production systems.
• Use verification to test future model upgrades and re-
tune algorithms

20.
© Crown copyright Met Office
Questions and answers