Zaborowski Sixsigma Nov07
- 2. Problem Statement:
CBT appointment cancellations have resulted in 56,842 partial
refunds being issued totaling $3.7 million and representing 3% of
total registrations from Jan ’05 – April ’06.
Goal:
Reduce partial refunds as percent of total registrations by 50% to
an average 1-2% per program
2
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
- 3. Y metric # Cancellations B/4 Refundable Date
# Tests Taken + # NS + # Customer Cancellations
CBT Partial Refunds as % of Total Appointments
Appointments = Test Taken + Partial Refunds
Percent
Note: Includes TOEFL, GRE and Praxis CBT
0.056
Refunds / Total Appointments
n = 3,140,655 1
1
0.044
1 1
1
1
1
0.032 UCL=0.03120
_
X=0.02188
0.020
LCL=0.01256
1 1 1
Jan-04 Apr-04 Jul-04 Oct-04 Jan-05 Apr-05 Jul-05 Oct-05 Jan-06 Apr-06
Date
1
Refunds / Total Appointments
0.012
UCL=0.01145
0.008
__
0.004
MR=0.00350
0.000 LCL=0
Jan-04 Apr-04 Jul-04 Oct-04 Jan-05 Apr-05 Jul-05 Oct-05 Jan-06 Apr-06
Date
Although refunds have averaged 2% (Jan 04-Apr 06),
3
From Sept 05-Apr 06 the average increased to 4%
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
- 5. Scenarios and Assumptions
• Scenarios evaluated
– Best Case: 100% of customer cancels in 3-X days forfeit
their registration
– Worst Case: 100% of customers take test as originally
planned
– Most Likely: Alternate candidate action taken as follows
GRE TOEFL Praxis
Probabilities
Reschedule 20% 20% 20%
Forfeit entire fee 25% 25% 25%
Cancel earlier than 7 days prior to admin 45% 45% 45%
Take test anyway as originally scheduled 5% 5% 10%
Not register for the test in the first place 0% 0% 0%
Register later and pay the late fee 5% 5% 0%
100% 100% 100%
5
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
- 6. Most Likely Analysis (TOEFL / GRE)
12-Month Impact at Current Volumes
GRE TOEFL Total TOEFL - Low Total - Low
5-DAY SCENARIO
Base Scenario (3 Day Policy)
ETS Net $ 175,560 $ 421,920 $ 597,480 $ 36,000 $ 211,560
5 Day Policy Scenario
ETS Net $ 240,065 $ 561,681 $ 801,746 $ 47,925 $ 287,990
Change in Net $ 64,505 $ 139,761 $ 204,266 $ 11,925 $ 76,430
7-DAY SCENARIO
Base Scenario (3 Day Policy)
ETS Net $ 331,980 $ 772,200 $ 1,104,180 $ 61,200 $ 393,180
7 Day Policy Scenario
ETS Net $ 452,700 $ 1,029,600 $ 1,482,300 $ 81,600 $ 534,300
Change in Net $ 120,720 $ 257,400 $ 378,120 $ 20,400 $ 141,120
10-DAY SCENARIO
Base Scenario (3 Day Policy)
Preferred ETS Net $ 486,420 $ 1,022,400 $ 1,508,820 $ 158,400 $ 644,820
10 Day Policy Scenario
Option ETS Net $ 663,300 $ 1,363,200 $ 2,026,500 $ 211,200 $ 874,500
Change in Net $ 176,880 $ 340,800 $ 517,680 $ 52,800 $ 229,680
TOEFL volume differences range from 5,000 – 12,000
6
cancellations between 5-day to 10-day scenarios
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
- 7. Approved Plan
• Change refund policy for GRE and TOEFL effective
with the new academic year (July 2007)
However, due to volume trends and competitive pressure from IELTS,
7
Global Division postponed changing the TOEFL refund policy as planned
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
- 9. Since the GRE Refund Policy changed (7/07) the percent of
registrations with a partial refunds have declined
GRE CBT Partial Refunds as % of Total Appointments
Before After
12.00% n=697,202 n=217,201
Individual V alue
10.00%
8.00% U C L=8.28%
_
6.00% X=5.63%
4.00%
LC L=2.97%
Jan-06 M ar-06 M ay -06 Jul-06 S ep-06 N ov -06 Jan-07 M ar-07 M ay -07 Jul-07 S ep-07
Date
Before After
4.00%
M oving Range
U C L=3.27%
3.00%
2.00%
__
1.00% M R=1.00%
0.00% LC L=0.00%
Jan-06 M ar-06 M ay -06 Jul-06 S ep-06 N ov -06 Jan-07 M ar-07 M ay -07 Jul-07 S ep-07
Date
9
Before: X=7.59% After:
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
X=5.63%
- 10. Correspondingly the number of cancellations with
no-refund as a percent of registrations has increased
GRE Cancel No Refunds as % of Total Appointments
Before A fter
n=1,376 U C L=1.09%
1.00% n=1,161
Individual V alue
0.75% _
X=0.65%
0.50%
0.25% LC L=0.21%
0.00%
Jan-06 M ar-06 M ay -06 Jul-06 S ep-06 N ov -06 Jan-07 M ar-07 M ay -07 Jul-07 S ep-07
Date
Before A fter
0.60%
U C L=0.54%
0.45%
M oving Range
0.30%
__
0.15% M R=0.17%
0.00% LC L=0.00%
Jan-06 M ar-06 M ay -06 Jul-06 S ep-06 N ov -06 Jan-07 M ar-07 M ay -07 Jul-07 S ep-07
Date
10
Before: X=0.21% After:
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
X=0.65%
- 11. GRE Cancellations by Status
Before/After Policy Change
Cancel Status - Before Cancel Status - After
Jul 07-Oct 07
Jan 06-Jun 07
12000 100 100
2500
# Appointment Cancellations
# Appointment Cancellations
10000
80 2000 80
8000
60 60
Percent
Percent
1500
6000
40 1000 40
4000
20 500 20
2000
0 0 0 0
Cancel Status Partial Refund No Refund Cancel Status Partial Refund No Refund
Before 10459 1311 After 1413 1160
Percent 88.9 11.1 Percent 54.9 45.1
Cum % 88.9 100.0 Cum % 54.9 100.0
The percentage of appointment that are cancelled with
11
no-refund has increased from 11% to 45%
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
- 12. GRE Refund Policy Before
and After Process Change
BEFORE AFTER CHANGE
Time Period Jan06–Jun07 Jul07+ ----
# Refunds Avoided/Mth 86 290 204
Net Savings/Month $ 4,816 $ 16,240 $ 11,424
Annual Savings $57,792 $194,880 $137,088
Since policy was changed the net change has been $34K
12
Annual net change is projected at ~$140K
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
- 14. TOEFL Program Refund
Policy Change
• Although TOEFL program did commit to
change their refund policy consistent with
GRE this decision was later reversed
– A result of the “TOEFL Go Anywhere from Here”
campaign for market share
– 3-day cancellation with no-refund policy remained
in effect 07-08 academic year
• This policy change was deferred for
implementation until Jul 08
14
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
- 15. GRE Refunds Control Plan
• Monitor appointment cancellations and partial refunds
issued through data warehouse query for next six
months
– Closely review candidate behavior by days before test day
• If new signal emerges GRE Program staff will identify
the root causes and any corrective action steps that
will be taken
15
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.
- 16. Lessons Learned
• Data on money flow processes such as registration,
after-market, and cancellations should be routinely
monitored to provide early warning of potential
margin loss and revenue erosion
• There is strong resistance to adjusting policies
among most program staff
– Data is a powerful tool to address tribal knowledge
• We need to balance our policies with competitors
16
Confidential and Proprietary. Copyright ©2006 Educational Testing Service.