This document discusses best practices and common pitfalls to avoid when conducting alumni surveys. It covers several areas to focus on, including question context and structure, question and answer order effects, answer categories, and question wording. For question wording, the document cautions against leading or loaded questions that could bias responses. Overall, the goal is to design surveys that gather meaningful feedback from alumni in a clear, unbiased manner.

1. Worst Practices in Alumni Surveys
(And How to Avoid Them)
Jerold Pearson
Principal, eAdvancement

2. • Context and Framing
• Structure and Flow
• Satisficing
• Question Order
• Answer Order
• Answer Categories
• Question Wording

3. Northwest Center for
Health and Wellness
Personal Health Survey

4. Northwest Center for
Health and Wellness
Personal Health Survey

5. What do you most value now about the experience you had as a student at Stanford?
__________
__________
How satisfied are you overall with the experience you had as a student at Stanford?
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied
How would you describe your overall feelings about the University today?
Very positive
Somewhat positive
Mixed/Neutral
Somewhat negative
Very negative
Opening Questions

6. What was best about Leading Matters?
How did you hear about Leading Matters?
How would you rate Leading Matters overall?
How easy or difficult was the registration process?
To what extent did Leading Matters enhance or reaffirm…
The pride you take in your Stanford affiliation
A feeling that you are part of the Stanford community
A feeling that Stanford is part of who you are
What was disappointing or needed improvement?
How would you rate the sessions you attended?
Nanotechnology
Stanford War Criminals
Photoshop for Fraud and Deception

7. How did you hear about Leading Matters?
How easy or difficult was the registration process?
How would you rate the sessions you attended?
Nanotechnology
Stanford War Criminals
Photoshop for Fraud and Deception
What was best about Leading Matters?
What was disappointing or needed improvement?
How would you rate Leading Matters overall?
To what extent did Leading Matters enhance or reaffirm…
The pride you take in your Stanford affiliation
A feeling that you are part of the Stanford community
A feeling that Stanford is part of who you are
Chronological order – before, during, after

8. How important do you think it is for the University to help students:
Only Not
Very Fairly Somewhat Important
Communicate well orally
Write effectively
Tell good jokes
Acquire new skills and knowledge on their own
Select body piercings
Think analytically
Formulate creative ideas and solutions
Hide their acne
Lead tasks and groups of people
Relate well to people of different races
Function effectively as a member of a team
Use quantitative tools
Place current problems in historical/cultural perspective
Identify moral and ethical issues
Waste time more efficiently
Function independently
Tweet more concisely
Develop self-esteem
Trim List Split SampleRandomize

9. Question Order: 1
Were you aware that Senator Donald J. Snort was
indicted last week on multiple charges of fraud,
embezzlement, indecent exposure, and cruelty to
animals?
How would you describe your impressions of
Senator Snort – very positive, somewhat
positive, somewhat negative, or very negative?

10. Question Order: 2
Do you think a communist country like Russia should let American
newspaper reporters come in and send back to America the news as
they see it?
Do you think the United States should let communist newspaper
reporters from other countries come in here and send back to their
papers the news as they see it?
In 1950, 36% said yes
When asked second, 73% said yes

11. Answer Order
Self-Administered
If the election for president were held today, would you vote for…
Betty Boop ___
Dr. Evil ___
Lisa Simpson ___
Eric Cartman ___
Phone
If the election for president were held today, would you vote for
Betty Boop, Dr. Evil, Lisa Simpson, or Eric Cartman?

12. Answer Order
Self-Administered - PRIMACY EFFECT
If the election for president were held today, would you vote for…
Betty Boop ___
Dr. Evil ___
Lisa Simpson ___
Eric Cartman ___
Phone
If the election for president were held today, would you vote for
Betty Boop, Dr. Evil, Lisa Simpson, or Eric Cartman?

13. Answer Order
Self-Administered - PRIMACY EFFECT
If the election for president were held today, would you vote for…
Betty Boop ___
Dr. Evil ___
Lisa Simpson ___
Eric Cartman ___
Phone – RECENCY EFFECT
If the election for president were held today, would you vote for
Betty Boop, Dr. Evil, Lisa Simpson, or Eric Cartman?

15. Answer Categories: 1
How would you rate the job Tom Bull is doing as Executive Director of the
PSU Alumni Association?
Indescribably terrific
Outstanding
Exceptional
Excellent
Very very good
Very good
So so
He’s a bum
Unbalanced scale - 6 positive, 2 negative
Do the categories clearly differ from one another?
Is there equal distance between them?

16. Answer Categories 2: Scales vs Labels
How would you rate Reunion Homecoming overall?
Numeric scales: 1-7 or 0-7 or 1-10 or 1-100
Labelled scales: Excellent, Good, Only fair, Poor
Labels are easier for respondents to make meaningful distinctions among.
How granular do you need the results to be?

17. Answer Categories 2: Scales vs Labels
Labels are also easier to interpret and report.
Mean score (on scale of 1-7)
Men: 5.1
Women: 5.7
Excellent
Men: 40%
Women: 52%

18. Answer Categories 3: “Don’t Know” Option
A. How satisfied are you overall with your undergraduate experience
at Stanford?
B. How do you feel about Stanley Slouch, the Acting Assistant Director
of Campus Facilities?
C. How would you rate Stanford at providing job and career services
for alumni?
D. How would you rate your experience at Reunion Homecoming
overall?

19. Answer Categories 3: “Don’t Know” Option
A. How satisfied are you overall with your undergraduate experience
at Stanford?
B. How do you feel about Stanley Slouch, the Acting Assistant Director
of Campus Facilities?
C. How would you rate Stanford at providing job and career services
for alumni?
D. How would you rate your experience at Reunion Homecoming
overall?

20. Answer Categories 4: Middle Option
A. How would you describe your overall feelings about Stanford today?
Very positive
Somewhat positive
Mixed/Neutral
Somewhat negative
Very negative
B. How would you rate your experience at Reunion Homecoming overall?
Excellent
Good
Only fair
Poor
C. How satisfied are you overall with your experience as a student at Stanford?
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied

21. Answer Categories 5: Ranking
Please rank the following animals in order of your
preference.
Cats Koalas
Dogs Sharks
Beetles Worms
Beatles Mosquitoes
Rabbits Pelicans
Chinchillas Iguanas
Snakes Parrots
Marmots Turtles
Leeches Scorpions
Kangaroos Buffaloes
Unicorns Tasmanian devils
Bears Otters

22. Answer Categories 5: Ranking
Max/Diff
Which of these alumni benefits are you most likely to
use…and which are you least likely to use?
Most Likely Least Likely
Credit card
Career services
Library borrowing privileges
Discounted life insurance

23. Question Wording: 1
How would you rate the job Nobel Laureate Kilgore Trout is doing as
President of the university?
How would you rate the job oxycontin addict Kilgore Trout is doing as
President of the university?
How would you rate the job conservative firebrand Milton Skink is doing at
shaking things up in Congress?
How would you rate the job our dynamic and inspirational leader, Kilgore
Trout, is doing as President of the university?

24. Question Wording: 2
How much did you enjoy the donor recognition event with Dean Horace
Rumpole?
To what extent did the donor recognition event with Dean Rumpole…
A. Demonstrate the impact your gifts have
B. Give you a better understanding of how the research at the Medical School is
making a difference in the world today
C. Make you feel appreciated for your support
D. Give you a better understanding of the needs and challenges facing the Medical
School in today’s changing health care environment

25. Question Wording: 3
How would you rate the class party and the
class panel?
How would you rate the class-based events at
Reunion Homecoming?
How would you rate…
Your class party
Your class panel
Your class skydive

26. Question Wording: 4
1. How many lawyers are in your firm?
2. Please estimate the percentage of your work
last year that was…
• Insurance Defense Litigation ___%
• Direct Representation of Defendant ___%
• Non-Litigation ___%

27. Question Wording: 5
Do you favor or oppose The Riches for All Act?
Do you favor or oppose the Riches For All Act that will give
every American a million dollars, a mansion, tax cuts, and a
chicken in every pot?
Do you favor or oppose the Riches For All Act that will give
every American a million dollars, a mansion, tax cuts, and a
chicken in every pot…but will bankrupt the nation, exhaust our
natural resources, destroy the atmosphere, and end life on
Earth?

28. Question Wording: 6
Do you approve or disapprove of the Affordable Care Act?
Do you approve or disapprove of Obamacare?
46% approve of Affordable Care Act
38% approve of Obamacare
(Gallup, November 2013)
Do you approve or disapprove of the following health care reforms?
No annual or lifetime limits on the healthcare you receive
Young adults can stay on their parents’ plan until they are 26
You can't be denied coverage for pre-existing conditions
New Health Insurance Marketplaces allow you to compare plans and prices
Tax credits for small businesses who offer their employees insurance

29. Question Wording: 7
Do you favor or oppose the University’s new alcohol policy?
Do you favor or oppose the University’s new alcohol policy proposed
by President Goodfellow?
Do you favor or oppose the University’s new alcohol policy proposed
by the American Liquor Stores Association?

30. Col. North complained of Congress' failure to give
consistent aid to the anti-communist freedom
fighters in Nicaragua. In September, Congress will be
asked to approve such aid. Should Congress
continue aid to the Nicaraguan freedom fighters?
a) Yes, they're battling for our freedom too.
b) No, abandon Central America to the communists.
Question Wording: 7

31. Col. North complained of Congress' failure to give
consistent aid to the anti-communist freedom
fighters in Nicaragua. In September, Congress will be
asked to approve such aid. Should Congress
continue aid to the Nicaraguan freedom fighters?
a) Yes, they're battling for our freedom too.
b) No, abandon Central America to the communists.
Question Wording: 7

32. Col. North complained of Congress' failure to give
consistent aid to the anti-communist freedom
fighters in Nicaragua. In September, Congress will be
asked to approve such aid. Should Congress
continue aid to the Nicaraguan freedom fighters?
a) Yes, they're battling for our freedom too.
b) No, abandon Central America to the communists.
Question Wording: 7

33. Okay, now relax.
Jerold Pearson
jpearson@alumni.stanford.edu
(650) 390-3972
www.jeroldpearson.org