Non traditional measures in assessment 090712 (1)


Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Non traditional measures in assessment 090712 (1)

  1. 1. 1BEYONDSTANDARDIZEDTESTS:Assessing Creativity, CommonSense, Wisdom, and More in Admissions
  2. 2. Contact Information2 Robert J. Sternberg Provost and Senior Vice President George Kaiser Family Foundation Chair in Ethical Leadership Regents Professor of Psychology and Education Oklahoma State University; Honorary Professor, Heidelberg University President, Federation of Associations in the Behavioral and Brain Sciences;
  3. 3. 3
  4. 4. Goals of the Presentation4  To show that, in admissions, in addition to memory and analytical skills, it is important rigorously to assess creative, practical, and wisdom-based/ethical skills of students  To show how to assess these additional skills in a rigorous way  To show data regarding what happens when an assessment process rigorously assesses these additional skills
  5. 5. Organization of Talk5  Introduction  Five Projects with Non-Traditional Measures  The Rainbow Project  The Kaleidoscope Project  The Panorama Project  Advanced Placement Project  The Choate Rosemary Hall Project  Implementing Non-Traditional Measures in Your Institution  Conclusions
  6. 6. The Problem6  Traditional standardized-test measures in assessment, and even school grades, give us good information about some valued skills of students, but practically no information about other valued skills
  7. 7. In Particular…7  If we wish to develop students who will be the active citizens and future leaders of tomorrow, we need to measure a broader range of the skills important to future success—not just the memory and analytical skills measured by standardized tests (such as IQ tests, the SSAT, SAT, and the ACT), but also creative, practical, and wisdom-based skills
  8. 8. What Skills do Standardized Tests8 Measure Well?  Memory skills  Who?  What?  Where?  When?  Why?  How?
  9. 9. What Else Do Standardized Tests9 Measure Well?  Analytical skills  Analyze  Critique  Judge  Compare and contrast  Evaluate
  10. 10. What Skills Don’t Standardized10 Tests Measure Well?  Creative skills  Invention  Exploration  Discovery  Imagination  Supposition
  11. 11. What Skills Don’t Standardized11 Tests Measure Well?  Practical Skills  Application  Utilization  Implementation  Putting ideas into practice  Persuasion
  12. 12. What Skills Don’t Standardized12 Tests Measure Well?  Wisdom-Based Skills  Ethical thinking  Seeing multiple points of view  Understanding how what is true can change over time  Understanding long-term as well as short-term implications  Thinking for the common good
  13. 13. Why These Particular Skills?13  Motivation for the “Theory of Successful Intelligence”  Alice  Barbara  Celia  Diane  Zachary
  14. 14. Assessments on which Talk is14 Based  Rainbow (admissions)  National project involving roughly 1000 diverse seniors in high school and college freshmen  Kaleidoscope (admissions)  Project at Tufts University involving over 30,000 applicants to the freshman class over a period of 5 years
  15. 15. Assessments on which Talk is15 Based  Panorama (admissions)  New project at Oklahoma State University—no data yet  Advanced Placement (achievement)  Project at diverse high schools in the United States measuring (so far) achievement among several hundred students at high levels in psychology, statistics, and physics
  16. 16. Assessments on which Talk is16 Based  Choate Rosemary Hall Assessment  An assessment to help select students who most can profit from, and in turn profit, the environment of the preparatory school
  17. 17. Augmented Theory of17 Successful Intelligence  People are successfully intelligent to the extent that they make the most of their lives  They do so by figuring out their strengths and weaknesses, and (a) capitalizing on the strengths while (b) compensating for or correcting weaknesses  They do so through a mix of analytical, creative, practical, and wisdom- based/ethical skills applied to a knowledge
  18. 18. In particular…18  Individuals are successfully intelligent to the extent they display  Creative skills to generate novel ideas  Analytical skills to ascertain whether the ideas are good ones  Practical skills in order to implement their ideas and persuade others of their value  Wisdom-based/ethical skills in order to ensure the ideas help to achieve a common good based upon positive ethical principles
  19. 19. Source for Details19 Sternberg, R. J. (2010). College Admissions for the 21st Century. Cambridge, MA: Harvard University Press.
  20. 20. The Rainbow Project20  This project was conducted at Yale University in the early 2000s on roughly 1000 students with freshmen from 13 colleges and seniors from 2 high schools. The institutions were geographically very dispersed and were of greatly varying levels of selectivity, from community colleges to highly prestigious ones. There was a high level of ethnic diversity. Assessments were proctored in the
  21. 21. Analytical Assessments21  Learn meanings of words from context  (The blen arose early in the morning on the horizon.)  Complete a number series  (3, 7, 13, 21, ….)  Complete a figural matrix
  22. 22. Practical Assessment Sample Item College Life Tacit-Knowledge Inventory22You are enrolled in a large introductory lecture course. Requirements consist of three term- time exams and a final. Please indicate how characteristic it is of your behavior to spend time doing the following, if your goal is getting an A in the course.
  23. 23. Practical Assessment Sample Item College Life Tacit-Knowledge Inventory23 Rate on a 1 (low) to 9 (high) scale: ___Attending class regularly. ___Attending optional weekly review sessions, if there are any, with the T.A. ___Reading assigned text chapters thoroughly. ___Taking comprehensive class notes. ___Speaking with the Professor after class and during office hours. …
  24. 24. Practical Assessment Sample Item General Workplace/Common Sense Tacit-Knowledge Inventory24 You’ve been assigned to work on a project for a day with a fellow employee whom you really dislike. He is rude, lazy, and rarely does a proper job. What would be the best thing for you to do?
  25. 25. Practical Assessment Sample Item General Workplace/Common Sense Tacit-Knowledge Inventory25 Rate on a 1 (low) to 9 (high) scale: ___Tell the worker that you think he is worthless. ___ Warn the worker that, if he is not “on his toes” today, you will complain to the supervisor. ___ Avoid all conversation and eye contact with the other worker. ___ Be polite to the other worker and try to maintain as business-like a manner as possible so that hopefully he will follow your example for the day. …
  26. 26. Practical Everyday Situational Judgment - Movies26 Examinees see seven digitized movies depicting various real-life situations that college students confront or may confront: The Party: Entering a party where one does not know anyone A Fair Portion: Discussing shares of rental payments for a flat Professor’s Dilemma: Asking for a letter of recommendation from a professor who does not know you
  27. 27. Creative Written Stories27 SHORT STORY TASK: TITLES “A Fifth Chance” “2983” “Beyond the Edge” “The Octopus’s Sneakers” “It’s Moving Backwards” “Not Enough Time”
  28. 28. Creative Oral Stories28 SHORT STORY TASK Students see several pictorial collages. They have to tell a story about one of them.
  29. 29. 29
  30. 30. Creative Cartoon Titles30 Examinees see five cartoons and need to provide titles for three of the five.
  31. 31. ________________________________________________________________________ ________________________________________________________________________31
  32. 32. Exploratory Factor Analysis: Rainbow Tasks Factor 1 Factor 2 Factor 3 Oral Stories 0.57 -0.06 -0.06 Written Stories 0.79 0.01 -0.02 Cartoons 0.20 0.28 -0.08 STAT-creative 0.00 0.73 0.09 STAT-analytic -0.06 0.80 -0.04 STAT-practical 0.03 0.81 -0.02 Movies 0.12 0.05 0.52 College Life -0.13 0.01 1.00 Common Sense 0.12 -0.01 0.92Promax rotation: 62.8% variance explained
  33. 33. Predicting College GPA: SAT +AnalyticalStep 1: 20SAT-Verbal, SAT-Math 15 R squared (%)Step 2: 9.8 9.9Analytic (STAT) 10 5 0 Step 1 Step 2 33
  34. 34. Predicting College GPA: SAT +Practical 20Step 1:SAT-Verbal, SAT-Math 15 R squared (%)Step 2: 9.8 10.7 10Practical (STAT + performance) 5 0 Step 1 Step 2 34
  35. 35. Predicting College GPA: SAT +Creative 20Step 1:SAT-Verbal, SAT-Math 15 R squared (%) 12.8Step 2:Creative (STAT + performance) 9.8 10 5 0 Step 1 Step 2 35
  36. 36. Predicting GPA: SAT + Analytic, Creative, Practical 19.9Step 1: 20SAT-Verbal, SAT-Math 15 R squared (%)Step 2:All Rainbow Project Items 9.8(STAT Analytic, Practical, 10Creative,Practical performance, 5Creative performance) 0 Step 1 Step 2 36
  37. 37. Predicting GPA: All measures (practical beforecreative)* 30 Step 1: SAT-M 24.8 SAT-V 25 HSGPA 20 R squared 15.6 15.2 15.9 15 Step 2: + Analytic 10 Step 3: + Practical 5 Step 4: + Creative 0 *Controlling for school quality in Step Step Step Step 1 2 3 4 dependent variable 37
  38. 38. In sum38  In the Rainbow sample,  Adding Rainbow measures over SAT roughly doubles prediction of college success  Adding Rainbow measures over SAT + High School GPA increases prediction by roughly half
  39. 39. Amount of Each Measure That Is Predicted by Racial / Ethnic Differences (ω²)Proportion explained by race differences 0.10 0.09 0.05 0.04 0.03 0.03 0.02 0.02 0.02 0.01 0.01 0.01 0.00 0.00 Common Math College STAT STAT STAT Movies Cartoons Written Stories Stories Verbal Sense Oral Life SAT Analytic Creative Practical 39
  40. 40. In Sum40  In the Rainbow sample:  Rainbow measures reduce ethnic-group differences relative to the SAT alone  The new measures reduce differences because different ethnic groups show different average patterns  Differences are not eliminated, however
  41. 41. The Kaleidoscope Project41  Project at Tufts University over a period of five years involving over 30,000 applicants to the freshman class. Additional admissions exercises were optional. Questions were placed as an optional Tufts-specific supplement to the Common Application. The study was done in collaboration with Dean of Admissions Lee Coffin and his staff.
  42. 42. Essay Prompts: Year 142  The late scholar James O. Freedman referred to libraries as “essential harbors on the voyage toward understanding ourselves.” What work of fiction or non-fiction would you include in personal library? Why? (Analytical)  An American adage states that “curiosity killed the cat.” If that is correct, why do we celebrate people like Galileo, Lincoln, and Gandhi, individuals who imagined longstanding problems in new ways or who defied conventional thinking to achieve great results? (Analytical)
  43. 43. Essay Prompts: Year 143  History’s great events often turn on small moments. For example, what if Rosa Parks had given up her seat on that bus? What if Pope John Paul I had not died after a month in office in 1978? What if Gore had beaten Bush in Florida and won the 2000 U.S. Presidential election? Using your knowledge of American or world history, choose a defining moment and imagine an alternate historical scenario if that key event had played out differently. (Creative)
  44. 44. Essay Prompts: Year 144  Create a short story using one of the following topics:  The end of MTV  Confessions of a Middle School Bully  The Professor Disappeared  The Mysterious Lab (Creative)  Using an 8.5x11 inch sheet of paper, illustrate an ad for a movie, design a house, make an object better, or illustrate an ad for an object of your choice. (Creative)  Do a creative YouTube Video and send us the web address (Creative)
  45. 45. Essay Prompts: Year 145  Describe a moment in which you took a risk and achieved an unexpected goal. How did you persuade others to follow your lead? What lessons do you draw from this experience? You may reflect on examples from your academic, extracurricular or athletic experiences. (Practical)  A high school curriculum does not always afford much intellectual freedom. Describe one of your unsatisfied intellectual passions. How might you apply this interest to serve the common good and make a difference in society? (Wisdom)
  46. 46. Creative Essay: “What if…” If the Trojans had heeded Laocoon’s advice and thrown Odysseus’ wooden horse into the sea, they would have defeated the Greeks at Troy. Aeneas would then never have had reason to flee the city, and he would never have ventured to Italy to found Rome. Without Rome, neither the Roman Republic nor a Roman Empire would have existed. Concrete, the arch, plumbing, and the sauna might never have been invented. The modern implications of Rome never having existed are indeed drastic. Lacking even concrete floors, people would resort to sleeping in the mud, and, without plumbing or saunas, they would be perpetually filthy and, generally, quite chilly. France could not have built the base of the Eiffel Tower without arches, so tourists would be unable to purchase miniature collectible Towers in Parisian convenience stores.
  47. 47. Good but Uncreative Essay:“What if…” What if the ratification of the nineteenth amendment did not pass and women were never given the right to vote? What would life for women, like me, be like in the United States? For one thing, I probably would not be writing this essay. If women were not given their right to vote, I probably would stop going to school after this year and it would be unlikely that I would receive a college education. Without suffrage, my career options would be limited, if a career were a possibility at all. My accepted practices would be limited to staying home and taking care of the family. Rather than being equals, women would be subservient to men. I might not drive, I might not dress in the way in which I choose to, and I might not be able to live my life the way that I can in the twenty-first century.
  48. 48. Pilot Study Data48  Number of applications rose  Bottom third of old application pool greatly diminished; many more top applicants  Average SATs rose slightly  African-American applications up 25%, acceptances up 30%  Hispanic-American applications and acceptances up 15%
  49. 49. Pilot Study Data49  There were no significant ethnic-group differences on Kaleidoscope  Kaleidoscope correlated moderately with rated leadership/extracurricular activities (.44)  Being rated for Kaleidoscope was associated with higher freshman GPA, holding constant high school GPA and SATs
  50. 50. Pilot Study Data50  Kaleidoscope predicted extracurricular, leadership, active- citizenship participation  Greater customer satisfaction  Message to students, parents, teachers, and counselors that Tufts is looking for more than just the high-SAT, high-GPA student
  51. 51. The Panorama Project51  This is a new project at Oklahoma State University. All applicants are being given the opportunity optionally to answer questions that assess creative, analytical, practical, and wisdom-based skills. Each applicant who participates answers three questions, similar to those in Kaleidoscope. Scoring will be similar to that for Kaleidoscope. Ratings will be used for those evaluated by holistic admissions and for scholarship consideration.
  52. 52. Panorama Sample Analytical52 Question  An army colonel once stated, "Leadership is about comforting the disturbed and disturbing the comfortable." What did he mean? What do you think true leadership is? When have you been a leader, and how did you exercise leadership?
  53. 53. Panorama Sample Practical53 Question  If you were able to open a local charity or business of your choice, what type of organization would it be and whom would it benefit? Describe your start-up process.
  54. 54. Panorama Sample Creative54 Question  Write a story or poem that includes one of the following sets of words:  Purple, panic, panda, petunia, and popcorn  A stick, a light bulb, the Great Wall of China, and water  A bicycle, a clock, the Wild West, and duct tape
  55. 55. Panorama Sample Wisdom55 Question  After submitting a class project, you realize one of your partners committed plagiarism. Your teacher previously announced that if he or she learned that cheating had occurred, all members of the work group would receive an F grade. How would you handle the situation and what would be your ideal outcome?
  56. 56. The Advanced Placement56 Project  This is a project to infuse measurement of creative, analytical, and practical skills into tests of achievement given in high schools. Study has been run so far with Psychology, Statistics, Physics.
  57. 57. An Example from Psychology:57 Creative  Imagine that you had to produce a TV sitcom to illustrate Freud’s personality theory. Which of the following characters would best represent the superego?  (a) A Fire Fighter  (b) An action-movie hero  (c) A nurse  (d) An artist  *(e) A Supreme Court judge
  58. 58. An Example from Statistics:58 Practical  Mr. Smith, a politician, argues that the average family income in his state is $4,500 a month and that therefore complaints about massive poverty are ill-founded. What is wrong with Mr. Smiths’ claim?  (a) $4,500 a month is not very high.  (b) Mr. Smith’s statement is obviously not true.  *(c) Mr. Smith does not take into account the standard deviation of incomes.  (d) Mr. Smith is not an expert on poverty and hence has no credentials to make any claim about it.
  59. 59. Choate Rosemary Hall Project59  This battery included a variety of measures to enhance prediction of academic success in the environment of Choate Rosemary Hall  The samples that follow are from the School Life Questionnaire
  60. 60. School-Life Questionnaire 1 2 3 4 5 6 7Not a very good Average A very good choice choice In day schools you rarely see your teachersoutside of class. Some of them might be engagedin sports or other extra-curricular activities, butmostly you only see them in school-relatedcircumstances. At boarding school the situationis quite different, because many teachers live oncampus and you get to see them outside theclassroom a lot.
  61. 61. School-Life Questionnaire 1 2 3 4 5 6 7 Not a very good Average A very good choice choice Given this situation, rate the quality of the following behavior choices:____ (a) Always greet teachers and smile, but avoid seeing them outside of class.____ (b) Take advantage of this situation to talk to teachers about your school-related problems.____ (c) Wait and see if teachers approach you, and if so, what kinds of things they talk to you about.
  62. 62. School-Life Questionnaire 1 2 3 4 5 6 7 Not a very good Average A very good choice choice____ (d) Talk to your teachers but avoid discussing your problems as this might give them a negative impression of you.____ (e) Try to be sensitive and make a distinction between situations when teachers are available and unavailable to you.____ (f) Always try to be noticed—the more teachers talk to you, the better your grades will be.____ (g) Always ask whether it is a good time or not to discuss your problems with teachers.
  63. 63. 1 2 3 4 5 6 7Not a very good Average A very good choice choiceYou are taking a math class that gives you alot of trouble. On the first two tests you didpoorly, and for tomorrow you have ahomework problem that you are not quitesure how to solve. .
  64. 64. 1 2 3 4 5 6 7 Not a very good Average A very good choice choice Given this situation, rate the quality of the following behavior choices:____ (a) Try to find a solution, or at least some explanation of how to arrive at a solution, in the book .
  65. 65. 1 2 3 4 5 6 7 Not a very good Average A very good choice choice____ (b) Give it a try; if you can’t solve the problem you’ll just have to tell the teacher you didn’t understand it.____ (c) Ask some of your classmates if they’ve found the solution, telling them you want to compare it with yours’ (although you don’t actually have one).
  66. 66. 1 2 3 4 5 6 7 Not a very good Average A very good choice choice____ (d) Try hard, and if it doesn’t work, give up on it; you can always pretend you had forgotten you had homework.____ (e) Suggest to some of your classmates that you study together.
  67. 67. 1 2 3 4 5 6 7 Not a very good Average A very good choice choice____ (f) Go to see the one student in the class who you know is really smart and ask him for help.____ (g) Try hard, take a lot of notes, then come to the teacher before class and tell her/him that you tried and failed.
  68. 68. Predicting Choate GPA: HierarchicalRegression Pre-Choate GPA, SSAT (all) Yale indicators (all) 60 50.9 50 40 31.3 30 20 10 0 Choate GPA 12/99 (adj Rsq)
  69. 69. Predicting Choate GPA: Hierarchical/Stepwise Regression Pre-Choate GPA SSAT(Q) Yale indicators (3)60 52.75040 30.930 24.720100 Choate GPA 12/99 (adj Rsq)
  70. 70. Predicting Choate GPA: HierarchicalRegression Locus of Control Choate Tacit Knowledge Self-confidence 58 60 52.8 50 42.5 40 30 20 10 0 Choate GPA 12/99 (adj Rsq)
  71. 71. Main Results71  Adding analytical, creative, and practical questions:  Increases content validity  Increases face validity  Decreases ethnic-group differences
  72. 72. Putting Theory into Practice:72 Creating Rubrics  Test users are busy people. So it is preferable to use “holistic” ratings rather than trying to be excessively micro- analytic.  You need to decide what you value in responses  Some rubrics we have used are the following:
  73. 73. Analytical Rating73  The extent to which the response is Analytical Organized Logical Balanced
  74. 74. Creative Rating74  The extent to which the response is Novel Compelling Task-appropriate (either yes or no)
  75. 75. Practical Rating75  The extent to which the response is compelling  The extent to which the response is practical with respect to  Human resources  Material resources  Time and place
  76. 76. Wisdom Rating76  The extent to which the response reflects A common good  A balance of one’s own, others’, and larger interests  Thinking over the long-term as well as the short-term  Positive ethical values
  77. 77. A Common Error to Avoid77  In scoring, you do not want merely to measure general academic skills; therefore, do not take into account grammar, punctuation, capitalization, and spelling except for the analytical rating. Otherwise, you create a general factor (halo effect) that has little to do with what you want to rate and that will increase correlation of the non-traditional assessments with conventional standardized tests
  78. 78. Conclusion78  Creative, practical, and wisdom-based skills, like memory and analytical skills, can be rigorously assessed  Measurements such as those described here provide a means for rigorous assessment  Assessing such additional skills provides incremental prediction of academic and personal success and reduces ethnic- group differences
  79. 79. Conclusion79  In order to develop the active citizens and leaders of tomorrow, who will make a positive, meaningful, and enduring difference to the world, one reliably and validly can assess in school creative, analytical, practical, and wisdom- based skills.
  80. 80. 80