Professional Writing Service http://HelpWriting.net/Alumni-Surveys--Three-Conceptualization 👈

1. 1
Alumni Survey:
Three conceptualizations to alumni research
Alberto F. Cabrera
Professor & WISCAPE Sr. Researcher
E-mail: cabrera@education.wisc.edu
David J. Weerts
Project Director& WISCAPE Research Associate
E-mail: dweerts@education.wisc.edu
&
Bradford J. Zulick
Doctoral Student & Graduate Research Assistant
E-mail: bzulick@wisc.edu
University of Wisconsin-
Madison
Métodos de análisis de
la inserción laboral de
los universitarios
Universidad de León
9- 11 de Junio de 2003

2. 2
Topics
1. Approaches to alumni
assessment
– Conceptualizations
– Policy questions
– Methodological
considerations
2. Common
methodological issues
3. Conclusions &
recommendations

3. 3

4. 4
Growth of Alumni Surveys in the
United States
Alumni Surveys 1979 – present
70%
15%
15%
Outcome
Engagement/Competency
Giving

5. 5
I. Outcomes Approach
1) Oldest Approach – Used since 1937 (Pace, 1979)
2) Assumption
– Institutional quality and effectiveness can be appraised
by what alumni have accomplished
3) Policy Questions
1) How satisfied are graduates with their employment?
2) How satisfied are the graduates with the degree
granting institution?
3) To what extent are graduates fully participating in
civic activities?

6. 6
Why stress job outcomes?
Most perspective students rate getting
a good job as important in selecting
an institution

7. 7
Top 6 reasons when selecting a college
Top 6 reasons when selecting a college
(281,064 students at 421 baccalaureate colleges & universities)
1. Academic Reputation (56%)
2. Graduates getting good jobs (51%)
3. Size of the institution (35%)
4. Financial Assistance (33%)
5. Admission to top graduate schools (30%)
6. School’s social activities (28%)
Source: HERI Fall 2001 Survey

8. 8
Outcomes Approach
4) Forces
– Accreditation
– State Mandates
– Social Justice Issues (Shape of the River)
5) Measures
– Job Satisfaction
– Satisfaction with the Institution
– Income
– Occupational Status
– Participation in civic activities/organizations

9. 9

10. 10
Civic Engagement

11. 11
II. Engagement & Competencies Approach
• Assumption
– Student engagement in college and attainment of
competencies while attending college are the best
predictors of alumni success.
• General advice
– Align assessment efforts with what research
shows matters in a student’s development,
learning and attainment of competencies (e.g.
Pascarella, 2002; Kuh 2001; Roberson, Carnes &
Vice, 2002).

12. 12
Policy Questions
1. What are the competencies (i.e., outcomes,
abilities & values) that college education should
foster most?
2. To what extent are alumni engaged with
faculty/staff/peers while attending college?
3. To what extent colleges and universities engage
students on those learning activities most prone
to produce critical competencies?
4. To what extent graduates apply those
competencies in the job or in graduate school?

13. 13
Measures of Engagement
• Active &
collaborative learning
• Interactions with
faculty & staff
• Educational
experiences
• Campus climate
• Enriching educational
experiences
• Classroom practices

14. 14
Measures of competencies
ACT College Outcomes Survey
• Gains in problem solving
& group skills
• Critical thinking
• Practical Competence
• Cognitive Complexity
• Knowledge Acquisition &
Application
• Preparation for career

15. 15
The Extent to Which 35 Abilities and Skills Were Enhanced by the
Undergraduate Program at Albany (N=1775 Alumni)
Average Response (Scale of 1 to 5)
Exercise personal responsibility
Maintain openness to new ideas
Think analytically & logically
Function independently
Learn how to learn
Exercise self-discipline
Acquire new skills & knowledge
Adapt to social situations
Listen effectively
Understand myself
Exercise problem solving skills
Write effectively
Possess clear goals
Speak effectively
Carry out research
Relate to people of different races
Know literature of recognized merit
70
68
66
66
64
63
59
50
50
49
47
43
43
43
41
41
Percent responding 4 or 5(high)
on a 5 point scale
100
8
8
8
9
17
11
10
17
15
17
16
17
18
23
26
27
29
Percent responding 1 or 2(low)
on a 5 point scale
3.9
3.9
3.8
3.8
3.8
3.8
53
3.7
3.5
3.5
3.4
3.4
3.4
3.3
3.3
3.2
3.2
3.2
0 20 40 60 80 100
80 60 40 20 0
Source: Volkwein (2002)

16. 16
The Extent to Which 35 Abilities and Skills Were Enhanced by the
Undergraduate Program at Albany (N=1775 Alumni)
Average Response (Scale of 1 to 5)
Evaluate and choose among alternatives
Understand cultural differences
Be effective team member
Produce scholarship
Analyze literature critically
Understand scientific findings
Cope with conflict
Appreciate artistic expression
Use statistics
Understand math concepts
Take moral & ethical action
Employ scientific methods
Lead and supervise others
Solve math problems
Analyze scientific data
Use foreign language skills
Utilize computing skills
0
20
40
60
80
100
11
34
27
33
35
36
28
36
37
35
41
37
42
38
44
45
58
61
39
39
38
37
37
37
36
34
34
32
32
31
31
30
30
29
27
19
Place problems in historical perspective
3.2
3.1
3.1
3.0
3.0
3.0
3.1
2.9
2.9
3.0
2.8
2.9
2.8
2.9
2.7
2.7
2.4
2.2
Percent responding 4 or 5(high)
on a 5 point scale
Percent responding 1 or 2(low)
on a 5 point scale
0 20 40 60 80 100
Source: Volkwein (2002)

17. 17
Approaches in identifying competencies
• Focus groups with experts
• Content Validation approach
1. Define a universe of competencies. Do a comprehensive job
analyses of those jobs graduates are most likely to hold. Single
out those skills, knowledge and abilities each task involves.
Then, identify those competencies that can be taught in college
from those that are learned on the job. Pay particular attention to
those contexts where those competencies are most likely to be
effective.
2. Develop and adopt assessment measures. Make certain that your
assessment tool reflects the competencies that enable graduates
succeed on the job under a variety of relevant contexts
3. Validate assessment measures with experts. Pilot test the
instrument with graduates, job holders, employers and other
experts.

18. 18
Competencies & Skills for Business Program
Honesty & trust, ethics & social responsibility
Integrity & character
Coaching others, developing others, motivating
others
Coaching & facilitating
Approachability, composure, interpersonal savvy,
negotiating, managing peer relationships, conflict
management
Interpersonal skills
Written & oral communication, presentation skills,
informing others
Communication
Collecting information, analyzing root causes,
developing alternatives, making effective decisions
Problem solving & decision
making
Planning, organizing, time management, processes
management, delegating, directing others
Implementation
Vision & strategic agility, action oriented, costumer
focus, building teamwork & political savvy
Leadership influence
Skill group
Competency dimension
Source: Roberson, M. T., Carners, L. W. & Vice, J. P. (2002). Defining and measuring student competencies:
A content validation approach for Business program outcome assessment. Delta Phi Epsilon Journal,
XLIV (1), p. 17.

19. 19
Forces pushing for adoption of
engagement & competency approach
1. Private funding organizations
2. Accreditation
3. State regulations

20. 20
Good teaching practices as an
alternative to rankings
(Chronicle, October 22, 1999)
“Unless we develop measures of quality where
colleges can provide evidence of their
contribution to student, then this whole
system [of ranking colleges] turns on
resources and reputation, and reinforces
the elitism of higher education
Russell Edgerton. Pew Charitable Fund

21. 21
How regional & professional accreditors are
changing
•
• Middle States Association of Colleges & Schools:
Middle States Association of Colleges & Schools:
– Revising standards to focus on student learning and other outcomes
such as the quality of student services.
– 1996 Middle States: Framework for Outcomes Assessment:
• “The ultimate goal of outcomes assessment is the improvement
of teaching and learning”
•
• New England Association of Schools & Colleges:
New England Association of Schools & Colleges:
– Developing better ways for colleges to measure student learning.
•
• North Central Association of Colleges & Schools
North Central Association of Colleges & Schools:
– Drafted revisions in eligibility requirements to require evidence of
student achievement and institutional effectiveness. Changes must still
be voted on by the group. Revisions of standards to begin this fall.
Based on The Chronicle of Higher Education (July 7, 00) A29

22. 22
How regional & professional
How regional & professional accreditors
accreditors are
are
changing (continued)
changing (continued)
• Southern Association of Colleges & Schools:
– Revising standards to make them less prescriptive and more
focused on measuring results, such as student achievement.
• Western Association of Schools & Colleges,
Commission for Community & Junior Colleges
– Revised standards in 2001 similar to North Central’s
• Commission for Senior Colleges & Universities
– Approved revised standards simplifying the compliance
process and stress evidence of student learning.
Based on The Chronicle of Higher Education (July 7, 00) A29

23. 23
ABET: Engineering Criteria 2002
Engineering programs must demonstrate that their graduates have:
• Ability to apply knowledge of
mathematics, science &
engineering
• Ability to design & conduct
experiments, as well as to analyze
and interpret data
• Ability to design a system,
component, or process to meet
desired needs
• Ability to function on multi-
disciplinary teams
• Ability to identify, formulate, and
solve engineering problems
• Ability to communicate effectively
• The broad education necessary
to understand the impact of
engineering solutions in a global
& societal context
• A recognition of the need for,
and ability to engage in life-long
learning
• Knowledge of contemporary
issues
• An ability to use the techniques,
skills, and modern engineering
tools necessary for engineering
practice

24. 24
Use of indicators in performance funding among 11
Use of indicators in performance funding among 11
states.
states.
8
44
25
3
15
42
23
16
4
20
0
5
10
15
20
25
30
35
40
45
50
Inputs &
Reputation
Process Outputs Outcomes Combined
2-year
4-year
Fuente: Burke & Serban (1998). Performance funding for public higher education: Fad or trend?

25. 25
Methodological concerns
• Can be expensive & time consuming
– Content validation approach can be prohibited
for all majors
• Gathering information from alumni
regarding collegiate experiences & current
employment experiences & attainments can
introduce ‘halo’ errors into results.

26. 26
III. Alumni Giving Approach
1) Assumption
1) Giving is a function of positive experiences
with the institution and ability to contribute
2) Policy Question
1) What is the ability and inclination of alumni
to support their alma mater through
philanthropy, service, and advocacy?

27. 27
Declining state support is driving
alumni assessment towards
alumni giving

28. 28

29. 29
Alumni Giving
Increasing reliance on alumni support in the USA
Office of the Chancellor, University of Wisconsin – Madison

30. 30
General advice:
Concentrate efforts in identifying alumni
with greatest capacity and willingness to
give

31. 31
Concentrate efforts on 10% of the people
who contribute 90% of gift dollars
Capacity
Rating
(10
is
highest)
0
5
5 10
These constitute
your primary
“prospect list”
Require long-term
work: need to
become engaged
Great volunteers
and lower level
gifts
You have better
things to do with
your time
Inclination Rating (10 is highest)
10

32. 32
Measures of Alumni Giving
•Age
•Family income
•Career and educational history
•Current job duties and responsibilities
•Board memberships
•Honors achievements, publications, or
creative works
•Leisure, activities, and hobbies
•Spouse’s career and educational
history, memberships, and activities
•Ages and education of children and
grandchildren
•Quality of Educational Experiences
•Extent to which institution prepared
alumnus for career
•Degree to which faculty members
exerted a positive influence
•Extent to which alumni maintain
contact with faculty and former
classmates
•Current impressions with the institution
•History of involvement with the
institution
•Willingness to use influence on behalf of
the alma mater
•Willingness to consider specific
assignments of gifts
Capacity
Willingness
Based on: Leslie & Ramey, (1988); Brittingham & Pezzulo, (1990); Bodigan & Dehne, (1997); Taylor & Martin, (1995).

33. 33
UW Law School Alumni Giving Survey

34. 34
Common Methodological Issues
•May yield lower
response rates
•Instrument not
accessible to entire
population
•Highly efficient
Internet Based
•Low response rate
•Self selecting responses
affect validity of results
•Low Expense
•Highest reliability of
instrument
Mail
•Higher cost
•Interviewer bias may
affect validity
•High response rate
•Most likely to be
completes in full
Telephone
Cons
Pros
Type of survey

35. 35
Future directions for alumni surveys
• Avoid capturing in a single survey data from
college experiences and post-college outcomes
– Cause & effect problems
– Measurement artifacts (‘Halo effect’ & ‘All Positive or
All Negative Answers’).
• Make surveying alumni part of your enrollment
management system.
– Incorporate all major campus units in the
conceptualization, design, analyses & application of
results.
– Approach middle school students as if they were future
contributing alumni
– Collect information at points in time for critical
developmental stages

36. 36
What to
What to
assess
assess &
&
when
when to do
to do
it?
it?

37. 37
A longitudinal approach to assessing alumni success
College Experiences
Engagement
Inside & outside
The classroom
Social
Integration
Classroom
Experiences
Climate &
Diversity
College Outcomes
College Decision
Persistence
Transfer
Stop-out
Competencies
Satisfaction &
Commitment
Degree
Completion
Employment
& Income
Job
Satisfaction
Civic
Engagement
Engagement
With alma
nater
Giving &
Advocacy for
Alma mater
Preparation
for College
Family
Encouragement
& Involvement
K-16
Communication
& Engagement
Aspirations
& Plans
Awareness of
College Characteristics,
Admission Standards,
& Costs

38. 38
What to
What to
measure
measure &
&
when
when to do
to do
it?
it?

39. 39
Graduation/
Post College
Collegiate Experiences
Preparation for College
Alumni Survey
Grad school
Degrees earned
Occupation/Income
Job/Career/
Satisfaction
Adequacy of college
Preparation
Leadership, service
Awards, recognition
Alumni giving
Student survey
Advising
Faculty & peer
interactions
Classroom & learning
experiences
Family & work
responsibilities
Faculty survey
Student survey
Orientation &
advisement
Remediation
Faculty & peer
interactions
Classroom & learning
experiences
Family & work
responsibilities
Faculty survey
Student Survey
Family education
Knowledge of careers
Family involvement
Information about
universities
Plans & intentions
Transfer plans
Intended major
Inquire process
Student Survey
Family education
Occupation
Encouragement
Family involvement
Saving for college
Aspirations
Plans & intentions
Campus information &
transcripts
Time to degree
Major & changes
Transfer
Stop-out, return,drop
Campus information
& transcripts
Enrollment information
Financial aid data
Campus profile
Academic performance
Campus information
& transcripts
Enrollment
information
Financial aid data
Campus profile
Academic
performance
School information &
transcript
Preparation for college
School district profile
School information
& transcript
Preparation for
college
School district
profile
Outcomes
Junior Year
Freshman Year
High School
Application &
readiness
(11th-12th)
Middle School
Aspirations &
readiness
(7th –10th)