1. 1
BEYOND
STANDARDIZED
TESTS:
Assessing Creativity, Common
Sense, Wisdom, and More in Admissions

2. Contact Information
2
Robert J. Sternberg
Provost and Senior Vice President
George Kaiser Family Foundation Chair in Ethical
Leadership
Regents Professor of Psychology and Education
Oklahoma State University;
Honorary Professor, Heidelberg University
President, Federation of Associations in the Behavioral
and
Brain Sciences;
robert.sternberg@okstate.edu

3. 3

4. Goals of the Presentation
4
 To show that, in admissions, in addition to
memory and analytical skills, it is important
rigorously to assess creative, practical, and
wisdom-based/ethical skills of students
 To show how to assess these additional
skills in a rigorous way
 To show data regarding what happens
when an assessment process rigorously
assesses these additional skills

5. Organization of Talk
5
 Introduction
 Five Projects with Non-Traditional Measures
 The Rainbow Project
 The Kaleidoscope Project
 The Panorama Project
 Advanced Placement Project
 The Choate Rosemary Hall Project
 Implementing Non-Traditional Measures in
Your Institution
 Conclusions

6. The Problem
6
 Traditional standardized-test measures in
assessment, and even school
grades, give us good information about
some valued skills of students, but
practically no information about other
valued skills

7. In Particular…
7
 If we wish to develop students who will be
the active citizens and future leaders of
tomorrow, we need to measure a broader
range of the skills important to future
success—not just the memory and
analytical skills measured by standardized
tests (such as IQ tests, the
SSAT, SAT, and the ACT), but also
creative, practical, and wisdom-based
skills

8. What Skills do Standardized Tests
8
Measure Well?
 Memory skills
 Who?
 What?
 Where?
 When?
 Why?
 How?

9. What Else Do Standardized Tests
9
Measure Well?
 Analytical skills
 Analyze
 Critique
 Judge
 Compare and contrast
 Evaluate

10. What Skills Don’t Standardized
10
Tests Measure Well?
 Creative skills
 Invention
 Exploration
 Discovery
 Imagination
 Supposition

11. What Skills Don’t Standardized
11
Tests Measure Well?
 Practical Skills
 Application
 Utilization
 Implementation
 Putting ideas into practice
 Persuasion

12. What Skills Don’t Standardized
12
Tests Measure Well?
 Wisdom-Based Skills
 Ethical thinking
 Seeing multiple points of view
 Understanding how what is true can
change over time
 Understanding long-term as well as
short-term implications
 Thinking for the common good

13. Why These Particular Skills?
13
 Motivation for the “Theory of Successful
Intelligence”
 Alice
 Barbara
 Celia
 Diane
 Zachary

14. Assessments on which Talk is
14
Based
 Rainbow (admissions)
 National project involving roughly 1000
diverse seniors in high school and
college freshmen
 Kaleidoscope (admissions)
 Project at Tufts University involving over
30,000 applicants to the freshman class
over a period of 5 years

15. Assessments on which Talk is
15
Based
 Panorama (admissions)
 New project at Oklahoma State
University—no data yet
 Advanced Placement (achievement)
 Project at diverse high schools in the
United States measuring (so far)
achievement among several hundred
students at high levels in
psychology, statistics, and physics

16. Assessments on which Talk is
16
Based
 Choate Rosemary Hall Assessment
 An assessment to help select students
who most can profit from, and in turn
profit, the environment of the
preparatory school

17. Augmented Theory of
17
Successful Intelligence
 People are successfully intelligent to the
extent that they make the most of their
lives
 They do so by figuring out their strengths
and weaknesses, and (a) capitalizing on
the strengths while (b) compensating for
or correcting weaknesses
 They do so through a mix of
analytical, creative, practical, and wisdom-
based/ethical skills applied to a knowledge

18. In particular…
18
 Individuals are successfully intelligent to the
extent they display
 Creative skills to generate novel ideas
 Analytical skills to ascertain whether the ideas
are good ones
 Practical skills in order to implement their ideas
and persuade others of their value
 Wisdom-based/ethical skills in order to ensure
the ideas help to achieve a common good based
upon positive ethical principles

19. Source for Details
19
Sternberg, R. J. (2010).
College Admissions for the
21st Century.
Cambridge, MA: Harvard
University Press.

20. The Rainbow Project
20
 This project was conducted at Yale
University in the early 2000s on roughly
1000 students with freshmen from 13
colleges and seniors from 2 high schools.
The institutions were geographically very
dispersed and were of greatly varying
levels of selectivity, from community
colleges to highly prestigious ones. There
was a high level of ethnic diversity.
Assessments were proctored in the

21. Analytical Assessments
21
 Learn meanings of words from context
 (The blen arose early in the morning on
the horizon.)
 Complete a number series
 (3, 7, 13, 21, ….)
 Complete a figural matrix

22. Practical Assessment Sample Item
College Life Tacit-Knowledge Inventory
22
You are enrolled in a large introductory lecture
course. Requirements consist of three term-
time exams and a final. Please indicate how
characteristic it is of your behavior to spend
time doing the following, if your goal is getting
an A in the course.

23. Practical Assessment Sample Item
College Life Tacit-Knowledge Inventory
23
Rate on a 1 (low) to 9 (high) scale:
___Attending class regularly.
___Attending optional weekly review
sessions, if there are any, with the T.A.
___Reading assigned text chapters
thoroughly.
___Taking comprehensive class notes.
___Speaking with the Professor after class
and during office hours.
…

24. Practical Assessment Sample Item
General Workplace/Common Sense Tacit-Knowledge Inventory
24
You’ve been assigned to work on a project
for a day with a fellow employee whom
you really dislike. He is rude, lazy, and
rarely does a proper job. What would be
the best thing for you to do?

25. Practical Assessment Sample Item
General Workplace/Common Sense Tacit-Knowledge Inventory
25
Rate on a 1 (low) to 9 (high) scale:
___Tell the worker that you think he is worthless.
___ Warn the worker that, if he is not “on his toes”
today, you will complain to the supervisor.
___ Avoid all conversation and eye contact with the other
worker.
___ Be polite to the other worker and try to maintain as
business-like a manner as possible so that hopefully he
will follow your example for the day.
…

26. Practical Everyday Situational Judgment - Movies
26
Examinees see seven digitized movies
depicting various real-life situations
that college students confront or may
confront:
The Party: Entering a party where
one does not know anyone
A Fair Portion: Discussing shares of
rental payments for a flat
Professor’s Dilemma: Asking for a
letter of recommendation from a
professor who does not know you

27. Creative Written Stories
27
SHORT STORY TASK: TITLES
“A Fifth Chance”
“2983”
“Beyond the Edge”
“The Octopus’s Sneakers”
“It’s Moving Backwards”
“Not Enough Time”

28. Creative Oral Stories
28
SHORT STORY TASK
Students see several pictorial collages.
They have to tell a story about one of them.

29. 29

30. Creative Cartoon Titles
30
Examinees see five cartoons and need
to provide titles for three of the five.

31. ________________________________________________________________________
________________________________________________________________________
31

32. Exploratory Factor Analysis: Rainbow Tasks
Factor 1 Factor 2 Factor 3
Oral Stories 0.57 -0.06 -0.06
Written Stories 0.79 0.01 -0.02
Cartoons 0.20 0.28 -0.08
STAT-creative 0.00 0.73 0.09
STAT-analytic -0.06 0.80 -0.04
STAT-practical 0.03 0.81 -0.02
Movies 0.12 0.05 0.52
College Life -0.13 0.01 1.00
Common Sense 0.12 -0.01 0.92
Promax rotation: 62.8% variance explained

33. Predicting College GPA: SAT +
Analytical
Step 1: 20
SAT-Verbal, SAT-Math
15
R squared (%)
Step 2: 9.8 9.9
Analytic (STAT) 10
5
0
Step 1 Step 2
33

34. Predicting College GPA: SAT
+Practical
20
Step 1:
SAT-Verbal, SAT-Math
15
R squared (%)
Step 2: 9.8
10.7
10
Practical (STAT + performance)
5
0
Step 1 Step 2
34

35. Predicting College GPA: SAT
+Creative
20
Step 1:
SAT-Verbal, SAT-Math
15
R squared (%)
12.8
Step 2:
Creative (STAT + performance) 9.8
10
5
0
Step 1 Step 2
35

36. Predicting GPA: SAT +
Analytic, Creative, Practical
19.9
Step 1: 20
SAT-Verbal, SAT-Math
15
R squared (%)
Step 2:
All Rainbow Project Items 9.8
(STAT Analytic, Practical, 10
Creative,
Practical performance,
5
Creative performance)
0
Step 1 Step 2
36

37. Predicting GPA: All measures (practical before
creative)*
30
Step 1: SAT-M
24.8
SAT-V 25
HSGPA 20
R squared
15.6 15.2 15.9
15
Step 2: + Analytic
10
Step 3: + Practical
5
Step 4: + Creative
0
*Controlling for school quality in Step Step Step Step
1 2 3 4
dependent variable
37

38. In sum
38
 In the Rainbow sample,
 Adding Rainbow measures over SAT
roughly doubles prediction of college
success
 Adding Rainbow measures over SAT +
High School GPA increases prediction
by roughly half

39. Amount of Each Measure That Is Predicted by
Racial / Ethnic Differences (ω²)
Proportion explained by race differences
0.10
0.09
0.05
0.04
0.03
0.03
0.02 0.02 0.02
0.01 0.01 0.01
0.00
0.00
Common
Math
College
STAT
STAT
STAT
Movies
Cartoons
Written
Stories
Stories
Verbal
Sense
Oral
Life
SAT Analytic Creative Practical
39

40. In Sum
40
 In the Rainbow sample:
 Rainbow measures reduce ethnic-group
differences relative to the SAT alone
 The new measures reduce differences
because different ethnic groups show
different average patterns
 Differences are not eliminated, however

41. The Kaleidoscope Project
41
 Project at Tufts University over a period of
five years involving over 30,000 applicants
to the freshman class. Additional
admissions exercises were optional.
Questions were placed as an optional
Tufts-specific supplement to the Common
Application. The study was done in
collaboration with Dean of Admissions Lee
Coffin and his staff.

42. Essay Prompts: Year 1
42
 The late scholar James O. Freedman referred to
libraries as “essential harbors on the voyage
toward understanding ourselves.” What work of
fiction or non-fiction would you include in
personal library? Why? (Analytical)
 An American adage states that “curiosity killed
the cat.” If that is correct, why do we celebrate
people like Galileo, Lincoln, and
Gandhi, individuals who imagined longstanding
problems in new ways or who defied
conventional thinking to achieve great results?
(Analytical)

43. Essay Prompts: Year 1
43
 History’s great events often turn on small
moments. For example, what if Rosa Parks
had given up her seat on that bus? What if
Pope John Paul I had not died after a
month in office in 1978? What if Gore had
beaten Bush in Florida and won the 2000
U.S. Presidential election? Using your
knowledge of American or world history,
choose a defining moment and imagine an
alternate historical scenario if that key
event had played out differently. (Creative)

44. Essay Prompts: Year 1
44
 Create a short story using one of the following
topics:
 The end of MTV
 Confessions of a Middle School Bully
 The Professor Disappeared
 The Mysterious Lab (Creative)
 Using an 8.5x11 inch sheet of paper, illustrate an ad
for a movie, design a house, make an object
better, or illustrate an ad for an object of your
choice. (Creative)
 Do a creative YouTube Video and send us the web
address (Creative)

45. Essay Prompts: Year 1
45
 Describe a moment in which you took a risk and
achieved an unexpected goal. How did you
persuade others to follow your lead? What
lessons do you draw from this experience? You
may reflect on examples from your
academic, extracurricular or athletic
experiences. (Practical)
 A high school curriculum does not always afford
much intellectual freedom. Describe one of your
unsatisfied intellectual passions. How might you
apply this interest to serve the common good
and make a difference in society? (Wisdom)

46. Creative Essay: “What if…”
 If the Trojans had heeded Laocoon’s advice and thrown
Odysseus’ wooden horse into the sea, they would have
defeated the Greeks at Troy. Aeneas would then never
have had reason to flee the city, and he would never have
ventured to Italy to found Rome. Without Rome, neither the
Roman Republic nor a Roman Empire would have existed.
Concrete, the arch, plumbing, and the sauna might never
have been invented. The modern implications of Rome
never having existed are indeed drastic. Lacking even
concrete floors, people would resort to sleeping in the
mud, and, without plumbing or saunas, they would be
perpetually filthy and, generally, quite chilly. France could
not have built the base of the Eiffel Tower without
arches, so tourists would be unable to purchase miniature
collectible Towers in Parisian convenience stores.

47. Good but Uncreative Essay:
“What if…”
 What if the ratification of the nineteenth amendment did
not pass and women were never given the right to vote?
What would life for women, like me, be like in the United
States? For one thing, I probably would not be writing
this essay. If women were not given their right to vote, I
probably would stop going to school after this year and it
would be unlikely that I would receive a college
education. Without suffrage, my career options would be
limited, if a career were a possibility at all. My accepted
practices would be limited to staying home and taking
care of the family. Rather than being equals, women
would be subservient to men. I might not drive, I might
not dress in the way in which I choose to, and I might
not be able to live my life the way that I can in the
twenty-first century.

48. Pilot Study Data
48
 Number of applications rose
 Bottom third of old application pool greatly
diminished; many more top applicants
 Average SATs rose slightly
 African-American applications up 25%,
acceptances up 30%
 Hispanic-American applications and
acceptances up 15%

49. Pilot Study Data
49
 There were no significant ethnic-group
differences on Kaleidoscope
 Kaleidoscope correlated moderately with
rated leadership/extracurricular activities
(.44)
 Being rated for Kaleidoscope was
associated with higher freshman
GPA, holding constant high school GPA
and SATs

50. Pilot Study Data
50
 Kaleidoscope predicted
extracurricular, leadership, active-
citizenship participation
 Greater customer satisfaction
 Message to
students, parents, teachers, and
counselors that Tufts is looking for more
than just the high-SAT, high-GPA student

51. The Panorama Project
51
 This is a new project at Oklahoma State
University. All applicants are being given the
opportunity optionally to answer questions that
assess creative, analytical, practical, and
wisdom-based skills. Each applicant who
participates answers three questions, similar
to those in Kaleidoscope. Scoring will be
similar to that for Kaleidoscope. Ratings will
be used for those evaluated by holistic
admissions and for scholarship consideration.

52. Panorama Sample Analytical
52
Question
 An army colonel once stated, "Leadership is
about comforting the disturbed and disturbing
the comfortable." What did he mean? What do
you think true leadership is? When have you
been a leader, and how did you exercise
leadership?

53. Panorama Sample Practical
53
Question
 If you were able to open a local charity or
business of your choice, what type of
organization would it be and whom would it
benefit? Describe your start-up process.

54. Panorama Sample Creative
54
Question
 Write a story or poem that includes one of the
following sets of words:
 Purple, panic, panda, petunia, and popcorn
 A stick, a light bulb, the Great Wall of
China, and water
 A bicycle, a clock, the Wild West, and duct
tape

55. Panorama Sample Wisdom
55
Question
 After submitting a class project, you realize
one of your partners committed plagiarism.
Your teacher previously announced that if he
or she learned that cheating had occurred, all
members of the work group would receive an
F grade. How would you handle the situation
and what would be your ideal outcome?

56. The Advanced Placement
56
Project
 This is a project to infuse measurement of
creative, analytical, and practical skills into
tests of achievement given in high schools.
Study has been run so far with
Psychology, Statistics, Physics.

57. An Example from Psychology:
57
Creative
 Imagine that you had to produce a TV sitcom
to illustrate Freud’s personality theory.
Which of the following characters would best
represent the superego?
 (a) A Fire Fighter
 (b) An action-movie hero
 (c) A nurse
 (d) An artist
 *(e) A Supreme Court judge

58. An Example from Statistics:
58
Practical
 Mr. Smith, a politician, argues that the average family
income in his state is $4,500 a month and that therefore
complaints about massive poverty are ill-founded. What
is wrong with Mr. Smiths’ claim?
 (a) $4,500 a month is not very high.
 (b) Mr. Smith’s statement is obviously not true.
 *(c) Mr. Smith does not take into account the standard
deviation of incomes.
 (d) Mr. Smith is not an expert on poverty and hence has
no credentials to make any claim about it.

59. Choate Rosemary Hall Project
59
 This battery included a variety of measures to
enhance prediction of academic success in the
environment of Choate Rosemary Hall
 The samples that follow are from the School
Life Questionnaire

60. School-Life Questionnaire
1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
In day schools you rarely see your teachers
outside of class. Some of them might be engaged
in sports or other extra-curricular activities, but
mostly you only see them in school-related
circumstances. At boarding school the situation
is quite different, because many teachers live on
campus and you get to see them outside the
classroom a lot.

61. School-Life Questionnaire
1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
Given this situation, rate the quality of the
following behavior choices:
____ (a) Always greet teachers and smile, but avoid
seeing them outside of class.
____ (b) Take advantage of this situation to talk to
teachers about your school-related problems.
____ (c) Wait and see if teachers approach you, and
if so, what kinds of things they talk to you about.

62. School-Life Questionnaire
1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
____ (d) Talk to your teachers but avoid discussing your
problems as this might give them a negative impression of
you.
____ (e) Try to be sensitive and make a distinction between
situations when teachers are available and unavailable to you.
____ (f) Always try to be noticed—the more teachers talk to you,
the better your grades will be.
____ (g) Always ask whether it is a good time or not to discuss
your problems with teachers.

63. 1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
You are taking a math class that gives you a
lot of trouble. On the first two tests you did
poorly, and for tomorrow you have a
homework problem that you are not quite
sure how to solve. .

64. 1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
Given this situation, rate the quality of the
following behavior choices:
____ (a) Try to find a solution, or at least some
explanation of how to arrive at a solution, in
the book .

65. 1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
____ (b) Give it a try; if you can’t solve the
problem you’ll just have to tell the teacher you
didn’t understand it.
____ (c) Ask some of your classmates if they’ve
found the solution, telling them you want to
compare it with yours’ (although you don’t
actually have one).

66. 1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
____ (d) Try hard, and if it doesn’t work, give up
on it; you can always pretend you had
forgotten you had homework.
____ (e) Suggest to some of your classmates
that you study together.

67. 1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
____ (f) Go to see the one student in the class
who you know is really smart and ask him for
help.
____ (g) Try hard, take a lot of notes, then
come to the teacher before class and tell
her/him that you tried and failed.

68. Predicting Choate GPA: Hierarchical
Regression
Pre-Choate GPA, SSAT (all) Yale indicators (all)
60 50.9
50
40 31.3
30
20
10
0
Choate GPA 12/99 (adj Rsq)

69. Predicting Choate GPA:
Hierarchical/Stepwise Regression
Pre-Choate GPA SSAT(Q) Yale indicators (3)
60 52.7
50
40 30.9
30 24.7
20
10
0
Choate GPA 12/99 (adj Rsq)

70. Predicting Choate GPA: Hierarchical
Regression
Locus of Control Choate Tacit Knowledge Self-confidence
58
60 52.8
50 42.5
40
30
20
10
0
Choate GPA 12/99 (adj Rsq)

71. Main Results
71
 Adding analytical, creative, and practical
questions:
 Increases content validity
 Increases face validity
 Decreases ethnic-group differences

72. Putting Theory into Practice:
72
Creating Rubrics
 Test users are busy people. So it is
preferable to use “holistic” ratings rather
than trying to be excessively micro-
analytic.
 You need to decide what you value in
responses
 Some rubrics we have used are the
following:

73. Analytical Rating
73
 The extent to which the response is
Analytical
Organized
Logical
Balanced

74. Creative Rating
74
 The extent to which the response is
Novel
Compelling
Task-appropriate (either yes or no)

75. Practical Rating
75
 The extent to which the response is
compelling
 The extent to which the response is
practical with respect to
 Human resources
 Material resources
 Time and place

76. Wisdom Rating
76
 The extent to which the response reflects
A common good
 A balance of one’s own, others’, and
larger interests
 Thinking over the long-term as well as
the short-term
 Positive ethical values

77. A Common Error to Avoid
77
 In scoring, you do not want merely to
measure general academic skills;
therefore, do not take into account
grammar, punctuation, capitalization, and
spelling except for the analytical rating.
Otherwise, you create a general factor
(halo effect) that has little to do with what
you want to rate and that will increase
correlation of the non-traditional
assessments with conventional
standardized tests

78. Conclusion
78
 Creative, practical, and wisdom-based
skills, like memory and analytical
skills, can be rigorously assessed
 Measurements such as those described
here provide a means for rigorous
assessment
 Assessing such additional skills provides
incremental prediction of academic and
personal success and reduces ethnic-
group differences

79. Conclusion
79
 In order to develop the active citizens and
leaders of tomorrow, who will make a
positive, meaningful, and enduring
difference to the world, one reliably and
validly can assess in school
creative, analytical, practical, and wisdom-
based skills.

80. 80