1. Alumni Surveys: Are Respondents Typical Students?
Casey Donnelly, Melissa Grant, Rebecca Jones, Mikala Morrow, Rebecca Redman
Advisors: Dr. Michael Stoloff, Dr. Megan Rodgers Good, Kristen Smith
Alumni surveys are typically sent to graduates to see how their undergraduate career benefitted their post-collegiate life. Typically, alumni survey response rates tend to be low. Are alumni who respond
representative of all graduates? We compared the transcripts of psychology majors who responded to an alumni survey with the transcripts of a random sample of psychology majors. Overall, the two
groups were similar but they also differed. For example, compared with randomly selected students, alumni-survey respondents had a higher GPA and received better grades in particular classes. Alumni-
survey respondents also completed more high-impact experiences. Alumni survey respondents may not be representative of all graduates, but data from alumni who respond to surveys can still be useful if
analyzed in depth. It is possible to examine the relationship between undergraduate experiences and post-graduation success. This will be the focus of the next phase of our research.
Introduction
Alumni surveys are distributed to gain vital information about
students’ post-graduate success. Generally response rates on
alumni surveys are on the decline and Porter (2004) calls for more
research in this area. With declining response rates, we’re left to
wonder: Are alumni survey respondents representative of the
typical graduate?
During the major, students have a series of requirements, but
there are also electives and opportunities for students to choose
their courses (Stoloff, 2010). Kuh (1995) found through 149
interviews that 85% of the graduates attributed their success to
outside school experiences that required them to plan, organize,
manage, and make decisions. Kuh suggested that such experiences
could be used to enhance knowledge gained in the classroom.
To date, prior researchers have not determined if alumni survey
respondents are indeed different from typical students, although
this is a testable question. Specifically, we predicted that survey
respondents received better grades than typical students and took
greater advantage of opportunities outside of the classroom. To
test this hypothesis we compared alumni survey respondents’
transcripts with a randomly selected group of alumni transcripts.
We rely on alumni surveys to measure the success of our graduates
and ideally adjust our programs to improve alumni outcomes. This study
demonstrated that alumni respondents are not typical graduates. These
students engage in more high impact courses and earn higher grades in
challenging courses. The alumni who responded to our survey were
remarkably successful in terms of graduate school completion and post-
graduation employment. We suspected that they were not typical
students and our results confirmed this hypothesis. These data suggest
that if students enroll in courses that require initiative and responsibility
they might have greater post-graduation success. We recommend that
departments offer more courses that encourage student independence,
initiative, and responsibility. Furthermore, we advise departments to use
caution when making inferences from alumni survey data; survey
respondents may not be representative of all graduates. However, useful
data can be gleaned from alumni surveys, especially when you relate the
college experiences (including information from transcripts) to post-
graduation outcomes. Our future work will investigate such connections.
Kuh, G. D. (1995). The other curriculum: Out-of-class experiences associated with student
learning and personal development. The Journal of Higher Education, 123-155.
Porter, S. R. (2004) Raising response Rates: What works? New Directions for Institutional
Research, Inc., 2004(121), 5-21.
Stoloff, M., McCarthy, M., Keller, L., Varfolomeeva, V., Lynch, J., Makara, K., Simmons, S. &
Smiley, W. (2010). The undergraduate psychology major: An examination of structure and
sequence. Teaching of Psychology, 37(1), 4-15.
19
39
5 6
21
74
9
40
54
3
7
32 32
23
0
10
20
30
40
50
60
70
80
90
100
PSYC290C* PSYC402C* PSYC 492 PSYC493 PSYC495 PSYC497* PSYC499*
%ofStudentsCompleting
Upper Level Course Enrollment
Control
Survey
Control
Mean (SD)
Survey
Mean (SD)
GPA 3.33 (.37) 3.47 (.32)
PSYC 101- Introduction
to Psychology
3.27 (.64) 3.49 (.58)
PSYC 211- Psychological
Research Methods
3.25 (.64) 3.47 (.55)
PSYC 285 – Drugs and
Behavior
3.13 (.61) 3.47 (.43)
PSYC 390 – Psychology
of Learning
3.11 (.73) 3.42 (.61)
PSYC 335- Abnormal
Psychology
3.03 (.61) 3.22 (.68)
PSYC 425 - School
Psychology
3.77(.21) 4.00 (.00)
Across the curriculum survey respondents and randomly selected
students took similar classes; however, there were a few differences
including examples displayed in the figure. Overall, the survey group had
a higher GPA than the control group. We compared grades in every
psychology course. We found statistically significant differences in grades
for six courses; in each case the survey group had a higher average grade
than the control group. Thus, the alumni survey respondents were not
representative of typical psychology major graduates.
Note. *Indicates a statistically significant difference between the survey and
control group.
METHOD
Discussion
Researchers emailed a survey to 1,186 James Madison University
Psychology alumni who graduated between 2000 and 2012. These
participants had provided their email addresses to the department.
Of these 1,186 alumni, 324 responded to the survey (27% response
rate). The survey asked participants an array of questions about
their psychology experience, including whether or not they would
release their transcripts to researchers. 216 respondents (67%)
permitted us this access.
To test whether or not this sample of alumni transcripts was
representative of the “typical” psychology major student, we
randomly sampled 102 student transcripts, stratified by graduation
year. (For example, if 15% of the alumni graduated in 2013, we
ensured 15% of our random sample was from 2013). We coded all
transcripts, using common coding procedures, noting the courses
taken and grades received for each psychology-related course. We
conducted a series of inter-rater reliability checks which suggest
that the data are reliable.
Method
References
Special thanks to: Kira McCarthy who began this research
as part of her honors thesis, 2012.
Enrollment
Grades
Results
Note. All variables were significant (p <.05). Numbers reflect GPA, such that 4
corresponds with “A”, 3 with “B,” etc.