LONGITUDINAL DATA AND
HIGHER EDUCATION ACCOUNTABILITY
Tom Schenk Jr., Consultant

Iowa College Student
Aid Commission
National Student
Clearinghouse
Student
Records
Student Unemployment
Awards Insurance
Student
Faculty
Courses
Positions
Employee
K-12 Records
[Project EASIER]
[Iowa Testing Services]
[CTE Plus]
AS-28
[Program Information]

PROJECT LEAD THE WAY
EVALUATION

4 11th Grade ITED: Math
20%
18%
16%
14%
12%
10%
Participants
Average = 79
8%
6%
Nonparticipants
Average = 65
4%
2%
0%
0 10 20 30 40 50 60 70 80 90

5 11th Grade ITED: Science
20%
18%
16%
14%
12%
10%
8%
6%
4%
2%
0%
0 10 20 30 40 50 60 70 80 90

6 Project Lead The Way

7 Project Lead The Way

8 Selection Bias
Socio-demographic Background
}
Cognitive ability
School-level factors

9 Propensity Score Matching
Binary outcome
Logit Function
y = φ(x1,…,xn)

10 Propensity Bins
0% - 25% 25.1% - 50% 50.1% - 75% 75.1% - 100%

11 Subsequent Analysis
Outcome
Propensity Bins
z = β1x1,…, βnxn, βyy

12 Enrollment by Gender
Nonparticipants
Female Male
16%
Participants 49%
51%
84%

13 Percentage of Whites by Cohort
100.0%
Participants
90.0%
80.0%
70.0%
Nonparticipants
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
2008 2009 2010 2011

14 8th Grade ITBS: Math
9%
8%
7%
6%
5%
Participants
Average = 80
4%
3%
Nonparticipants
Average = 58
2%
1%
0%
0 10 20 30 40 50 60 70 80 90

15 8th Grade ITBS: Science
9%
8%
7%
6%
Participants
Average = 79
5%
Nonparticipants
Average = 61
4%
3%
2%
1%
0%
0 10 20 30 40 50 60 70 80 90

16 Evidence of Selection Bias
Participants are more likely to be male, by a wide margin.
Participation by gender holds constant around 90 percent.
Students are in the 80th percentile in math and science.

17
LABOR SUPPLY

18 Unemployment Insurance
Wages for employees in most sectors.
Provided on a quarterly basis.
Includes multiple jobs.
Excludes workers in some sectors (e.g., military).
Does not include hourly wages or full-time status.
Only includes employees within Iowa.

19 Matching Wage Data
• Educational
NSC • Match
data • Remove wage data
students
MIS IWD

20 Human Capital Theory
Wages
Completers
Leavers
Direct
Time
Costs

21 Returns on Investment
Σ
T
yi - xi
t
-C
(1+r)
t=1

22 Returns on Investment
Track wages over a set period of time, T.
Find the difference between wages between completers and leavers
on the student level.
Find the cost of tuition.
If you assert an interest rate, r, then solve to find Net Present Value.
If the interest rate if left unknown, then solve to find the Internal Rate
of Return.

23 Methodology
Assemble a cohort of graduates (completers) and those who left
college without a degree (leavers).
Stagger the cohorts so completers are finishing their final year in
college as leavers are in their first year in the workforce.
Exclude students who are found in any postsecondary institution.
Estimate the tuition expenses for completers in their final year.
Cohorts from 2002 and 2006.

24 Wages by Year
Net Present Value for Completers was $3,131.
Internal Rate of Return was 6 percent.

25
NPV
Returns by Cluster

26 Returns by Select Clusters

28 In-state Retention Rate

Out-of-state In-state
School
Working
School
Working

HIGHER EDUCATION
ACCOUNTABILITY

31 Success Rate
60%
50%
40%
30%
20%
10%
0%
2005 2006
Slightly over 52 percent of first-time, full-time students entering in 2006
either transferred or graduated within three years (by 2008).

32

33 Graduation Rate
50%
40%
30%
20%
10%
0%
2001 2002 2003 2004 2005 2006
The three year graduation rate for first-time, full-time students rose slightly to
39.1 percent for the 2006 cohort. The graduation rate has fluctuated, but
stayed ahead of the national graduation rate (33 percent).

34 Time to Degree
2004
2003
0 1 2
Time-to-degree rose slightly to 2.4 years for the 2004 cohort from 2.3 years.

35 Accountability Systems
Measuring the effectiveness of institutions through student outcomes.
Desirable qualities of accountability measures:
1. Specifically measures the effectiveness of the institution, not
other factors.
2. Measures improvement.
3. Flexible to accommodate a variety of outcomes.
The current traditional measures cannot capture these elements.

36 Accountability Systems
Issues with graduation/success rate:
1. Denominator debate.
2. Selection bias.
3. Positive feedback mechanism.
Solutions: Use the same methods that assist with Project Lead The
Way evaluations.

37 Estimating Success
50% 50% 50% 50%

38 Simulated Example

OBTAINING
DATA

40 Obtaining Data
Contact your major professor and/or the Department of Education.
Education data can be obtained from the Department of Education,
other data (e.g., UI records) will need special permission.
Data is distributed de-identified and must be returned or destroyed at
the completion of the research study.
Obtaining data is relatively low-cost, nominal fee (<$100) for a secure
flash drive.

Tom Schenk Jr.
Consultant – Institutional Effectiveness & Accountability
Iowa Department of Education
Phone: 515-281-3753
E-mail: tom.schenk@iowa.gov