Trudy W. Banta                           Professor of Higher Education                                         and        ...
Discipline-Based Assessmentto Provide Convincing Evidence of   Graduate Learning Outcomes                        Presented...
My History• Educational psychology• Program evaluation & measurement• Performance funding in Tennessee• 1990 USDOE effort ...
ASSESSMENTIs like a dancer’s mirror.It improves one’s ability to see andimprove one’s performance.                        ...
ASSESSMENT OF INDIVIDUAL STUDENT DEVELOPMENT•Assessment     of basic skills for use in advising  •Placement  •Counseling•P...
KEY RESULTS OF INDIVIDUAL      ASSESSMENT•Faculty   can assign grades•Students learn their ownstrengths and weaknesses•Stu...
A SECOND LOOK • Across students •Across sections •Across courses     © TWBANTA-IUPUI
•Where   is learning satisfactory?•What   needs to be retaught?•Which approaches produce the mostlearning for which studen...
GROUP ASSESSMENT ACTIVITIES•Classroom   assignments, tests, projects•Questionnaires for students,     graduates, employers...
ASSESSMENT . . .    “a rich conversation   about student learning     informed by data.”        -- Ted Marchese --        ...
USE OF RESULTS OF GROUP      ASSESSMENT•Program    improvement•Institutional   and / or state peerreview•Regional and / or...
ORGANIZATIONAL LEVELS FOR ASSESSMENT             National             Regional                State              Campus   ...
GROUP ASSESSMENT REQUIRES      COLLABORATIONIn setting expected program outcomesIn developing sequence of learningexperi...
BARRIERS TO COLLABORATION     IN THE ACADEMY1. Graduate schools prepare specialists2. Departments hire specialists3. Much ...
TO FOSTER COLLABORATION•Name  interdisciplinary committees•Read and discuss current literature onlearning/assessment•Atten...
MOST FACULTY ARE NOT TRAINED AS            TEACHERS            Faculty DevelopmentCan Help Instructors:•Write clear object...
Taxonomy of Educational Objectives      (Bloom and Others, 1956)Cognitive domain        Sample verbs for outcomescategorie...
SOME GENERIC LEARNING OBJECTIVES•Differentiatebetween fact and opinion•Gather, analyze, and interpret data•Apply ethical p...
PROFESSIONAL PROGRAM          OBJECTIVES     Program Graduates will Demonstrate1.   Professional commitment2.   Communicat...
To Ensure That Concepts Are          Taught          Time management           © TWBANTA-IUPUI
ALVERNO COLLEGE 8 ABILITIESCommunicationAnalysisProblem   SolvingValuing in Decision-MakingInteractingGlobal Perspec...
PRINCIPLES OF UNDERGRADUATE        LEARNING (PULS)1.  Core communication and quantitativeskills2. Critical thinking3. Inte...
PUL #1CORE COMMUNICATION & QUANTITATIVE             SKILLS Demonstrated by student’s ability to: •Express ideas and facts ...
PRINCIPLES OF UNDERGRADUATE          LEARNING•A   distinctive feature of education at IUPUI•Permeate    the entire undergr...
PUL HISTORY AT IUPUI1990 – Study group of faculty and staff1992-98 – Series of task forces1998 – Adoption by Faculty Counc...
Standardized tests        CANinitiate conversation       © TWBANTA-IUPUI
IN USING STANDARDIZED TESTS• Match test with curriculum•Set expected scores on subscales•Discuss results•Determine what is...
Limitations of standardized tests of generic skillscannot cover all a student knowsnarrow coverage, need to supplementdiff...
VOLUNTARY SYSTEM OF ACCOUNTABILITY               Report Scores in  critical thinking, written communication,              ...
TN = MOST PRESCRIPTIVE(5.45% OF            BUDGET FOR INSTRUCTION)1.   Accredit all accreditable programs      (25)2.   Te...
AT THE UNIVERSITY OF TENNESSEE      CAAP      Academic Profile (now MAPP)      COMP (like CLA and withdrawn            by ...
IN TN WE LEARNED1.  No test measured 30% of gen ed skills2. Tests of generic skills measure primarilyprior learning3. Reli...
AN INCONVENIENT TRUTH.9 = the correlation between SAT     and CLA scores of institutions                       thus 81% of...
HOW MUCH OF THE VARIANCE IN SENIOR SCORES IS DUE TO COLLEGE IMPACT?• Student motivation to attend that institution(mission...
HOW MUCH OF THE VARIANCE IN SENIOR SCORES IS DUE TO COLLEGE IMPACT?            (CONTINUED) •Student motivation to do well ...
STUDENT MOTIVATION• Samples of students are being tested• Extrinsic motivators (cash, prizes) are used                We h...
CONCERNS ABOUT VALUE ADDED •Student attrition •Proportion of transfer students •Different methods of calculating •Unreliab...
Recent University of Texas Experience 30 – 40% of seniors at flagships earn    highest CLA score (ceiling effect) flagship...
WORD FROM MEASUREMENT EXPERTS             Given the complexity of  educational settings, we may never besatisfied that val...
Employing currently available  standardized tests of generic   skills to compare the qualityof institutions is not a valid...
OECD’S AHELO        COMPARING HEIS X NATIONS1.   Generic skills (CLA)2.   Disciplines (Engineering and Economics)3.   Valu...
2012K-12 standardized test scores are used to      evaluate and compare schools      assign grades to schools      take ov...
IN FINLAND AND SINGAPORE•No annual testing of students•No high-stakes accountability measures forteachers/schools•Scholars...
SHORT-TERM PERSPECTIVE•Limit degrees to 120 SCH•Penalize students who go beyond     a SCHcap•Reward graduation in 4 years•...
DE-PROFESSIONALIZATION –           IMMEDIATE PAYOFF•Teacher education is first•Industry certifications•Partnerships to fil...
BETTER WAYS TO DEMONSTRATE       ACCOUNTABILITYPerformance Indicators1.Access   (to promote social mobility)2.Engaging stu...
IF WE MUST MEASURE LEARNING               LET’S USE:1.   Standardized tests in major fields               licensure and ce...
START WITH MEASURES YOU           HAVE•Assignments  in courses•Course exams•Work performance•Records of progress through t...
METHODS OF ASSESSMENTPaper  and pencil testsIndividual or group projectsPortfoliosObservation of practiceObservation ...
Direct Measures of Learning    Assignments, exams, projects, papersIndirect Measures    Questionnaires, inventories, inter...
NILOA SURVEYPROGRAM LEVEL                  APPROACHES1.   Portfolios (80% in at least 1 area)2.   Performance assessments3...
STUDENT ELECTRONIC PORTFOLIO •Students   take responsibility for demonstrating core skills •Unique individual skills and a...
More use of RUBRICS     locally developed     VALUE from AAC&U     © TWBANTA-IUPUI
VALUE RUBRICS•Critical thinking•Written communication•Oral communication•Information literacy•Teamwork•Intercultural knowl...
ACCOUNTABILITY REPORT•85% achieve Outstanding ratings in writingas defined . . .•78% are Outstanding in applying knowledge...
FOR EXTERNAL CREDIBILITYCollaborate   on rubricsUse   employers as examinersConduct   process audits                  ©...
E-PORT CHALLENGES•Reliability of rubrics•Student motivation if     used for assessment            (Barrett, 2009)•Differen...
OBSTACLES TO USING  PERFORMANCE-BASED MEASURES•Defining domains and constructs•Obtaining agreement on what to measureand d...
WILL IT TAKE 80 YEARS . . . ?3 Promising Alternatives E portfolios                Rubrics                     Assessmentco...
TEAGLE ASSESSMENT SCHOLARS•study assessment data•visit campuses•talk with 3-4 groups of students•talk with faculty about t...
NATIONAL SURVEY OF STUDENT ENGAGEMENT                   AT          ~ HOPE COLLEGE ~ Seniors               Freshmen 2003 S...
HOPE COLLEGE•Considered data over supper•Proposed solutions•Conducted student focus groups•Shared all data with all facult...
NATIONAL INSTITUTE FORLEARNING OUTCOMES ASSESSMENT•Surveys     2009 CAOs     2011 Departments•Occasional Papers•Website re...
LUMINADegree Qualifications Profile        - Linked student learning outcomes–          AA to MA levelsSuggests transfer b...
NEW LEADERSHIP ALLIANCEFOR STUDENT LEARNING AND ACCOUNTABILITY  - Presidents’ Alliance  - Certification Process          S...
BUILD ASSESSMENT INTO VALUED              PROCESSES1.   Assessment of learning2.   Curriculum review and revision3.   Surv...
Upcoming SlideShare
Loading in …5
×

Trudy Banta AAGLO Forum Melbourne May 2012

811 views
673 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
811
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Trudy Banta AAGLO Forum Melbourne May 2012

  1. 1. Trudy W. Banta Professor of Higher Education and Senior Advisor to the Chancellor for Academic Planning and Evaluation Indiana University-Purdue University Indianapolisclick for short click for audiovideo of Trudy of this © TWBANTA-IUPUI presentation
  2. 2. Discipline-Based Assessmentto Provide Convincing Evidence of Graduate Learning Outcomes Presented in Australia May 2012 by Trudy W. BantaProfessor of Higher Education and Senior Advisor to the Chancellor for Academic Planning and Evaluation Indiana University-Purdue University Indianapolis 355 N. Lansing St., AO 140 Indianapolis, Indiana 46202-2896 tbanta@ iupui.edu http://www.planning.iupui.edu © TWBANTA-IUPUI
  3. 3. My History• Educational psychology• Program evaluation & measurement• Performance funding in Tennessee• 1990 USDOE effort to build a national test• 1992 Initiated evidence-based culture atIUPUI © TWBANTA-IUPUI
  4. 4. ASSESSMENTIs like a dancer’s mirror.It improves one’s ability to see andimprove one’s performance. Alexander Astin 1993 © TWBANTA-IUPUI
  5. 5. ASSESSMENT OF INDIVIDUAL STUDENT DEVELOPMENT•Assessment of basic skills for use in advising •Placement •Counseling•Periodic review of performance with detailedfeedback•End-of-program certification of competence •Licensing exams •External examiners © TWBANTA-IUPUI
  6. 6. KEY RESULTS OF INDIVIDUAL ASSESSMENT•Faculty can assign grades•Students learn their ownstrengths and weaknesses•Studentsbecome self-assessors © TWBANTA-IUPUI
  7. 7. A SECOND LOOK • Across students •Across sections •Across courses © TWBANTA-IUPUI
  8. 8. •Where is learning satisfactory?•What needs to be retaught?•Which approaches produce the mostlearning for which students? © TWBANTA-IUPUI
  9. 9. GROUP ASSESSMENT ACTIVITIES•Classroom assignments, tests, projects•Questionnaires for students, graduates, employers•Interviews, focus groups•Program completion and placement•Awards/recognition for graduates•Monitoring of success in graduate school•Monitoring of success on the job © TWBANTA-IUPUI
  10. 10. ASSESSMENT . . . “a rich conversation about student learning informed by data.” -- Ted Marchese -- AAHE © TWBANTA-IUPUI
  11. 11. USE OF RESULTS OF GROUP ASSESSMENT•Program improvement•Institutional and / or state peerreview•Regional and / or nationalaccreditation © TWBANTA-IUPUI
  12. 12. ORGANIZATIONAL LEVELS FOR ASSESSMENT National Regional State Campus College Discipline Classroom Student © TWBANTA-IUPUI
  13. 13. GROUP ASSESSMENT REQUIRES COLLABORATIONIn setting expected program outcomesIn developing sequence of learningexperiences (curriculum)In choosing measuresIn interpreting assessment findingsIn making responsive improvements © TWBANTA-IUPUI
  14. 14. BARRIERS TO COLLABORATION IN THE ACADEMY1. Graduate schools prepare specialists2. Departments hire specialists3. Much of our scholarship isconducted alone4. Promotion and tenure favorindividual achievements --interdisciplinary work is harder toevaluate © TWBANTA-IUPUI
  15. 15. TO FOSTER COLLABORATION•Name interdisciplinary committees•Read and discuss current literature onlearning/assessment•Attend conferences together•Bring experts to campus•Share good practices•Work together on learning communities © TWBANTA-IUPUI
  16. 16. MOST FACULTY ARE NOT TRAINED AS TEACHERS Faculty DevelopmentCan Help Instructors:•Write clear objectives (outcomes) for studentlearning in courses and curricula•Connect learning outcomes to assignments incourses.•Develop assessment tools that test higher orderintellectual skills © TWBANTA-IUPUI
  17. 17. Taxonomy of Educational Objectives (Bloom and Others, 1956)Cognitive domain Sample verbs for outcomescategories Identifies, defines, describesKnowledge Explains, summarizes, classifiesComprehension Demonstrates, computes, solvesApplication Differentiates, diagrams, estimatesAnalysis Creates, formulates, revisesSynthesis Criticizes, compares, concludesEvaluation © TWBANTA-IUPUI
  18. 18. SOME GENERIC LEARNING OBJECTIVES•Differentiatebetween fact and opinion•Gather, analyze, and interpret data•Apply ethical principles to local,national, global issues•Communicate ideas in writing effectively © TWBANTA-IUPUI
  19. 19. PROFESSIONAL PROGRAM OBJECTIVES Program Graduates will Demonstrate1. Professional commitment2. Communication skills3. Administrative and managerial skills4. Information technology competence5. Research and analytic competence
  20. 20. To Ensure That Concepts Are Taught Time management © TWBANTA-IUPUI
  21. 21. ALVERNO COLLEGE 8 ABILITIESCommunicationAnalysisProblem SolvingValuing in Decision-MakingInteractingGlobal PerspectivesEffective CitizenshipAesthetic Responsiveness
  22. 22. PRINCIPLES OF UNDERGRADUATE LEARNING (PULS)1. Core communication and quantitativeskills2. Critical thinking3. Integration and application of knowledge4. Intellectual depth, breadth, andadaptiveness5. Understanding society and culture6. Values and ethics Approved by IUPUI Faculty Council May 1998
  23. 23. PUL #1CORE COMMUNICATION & QUANTITATIVE SKILLS Demonstrated by student’s ability to: •Express ideas and facts to others effectively in a variety of formats, particularly written, oral, and visual formats •Communicate effectively in a range of settings •Identify and propose solutions for problems using quantitative tools and reasoning •Make effective use of information resources and technology
  24. 24. PRINCIPLES OF UNDERGRADUATE LEARNING•A distinctive feature of education at IUPUI•Permeate the entire undergraduatecurriculum•Are enacted differently in each discipline
  25. 25. PUL HISTORY AT IUPUI1990 – Study group of faculty and staff1992-98 – Series of task forces1998 – Adoption by Faculty Council2007 – Adoption of revised version
  26. 26. Standardized tests CANinitiate conversation © TWBANTA-IUPUI
  27. 27. IN USING STANDARDIZED TESTS• Match test with curriculum•Set expected scores on subscales•Discuss results•Determine what is missing © TWBANTA-IUPUI
  28. 28. Limitations of standardized tests of generic skillscannot cover all a student knowsnarrow coverage, need to supplementdifficult to motivate students to take them! What are they actually measuring? © TWBANTA-IUPUI
  29. 29. VOLUNTARY SYSTEM OF ACCOUNTABILITY Report Scores in critical thinking, written communication, analytic reasoning using•Collegiate Assessment of AcademicProficiency (CAAP)•Measuring Academic Proficiency andProgress (MAPP)•Collegiate Learning Assessment (CLA) © TWBANTA-IUPUI
  30. 30. TN = MOST PRESCRIPTIVE(5.45% OF BUDGET FOR INSTRUCTION)1. Accredit all accreditable programs (25)2. Test all seniors in general education (25)3. Test seniors in 20% of majors (20)4. Give an alumni survey (15)5. Demonstrate use of data to improve (15) ___ 100 © TWBANTA-IUPUI
  31. 31. AT THE UNIVERSITY OF TENNESSEE CAAP Academic Profile (now MAPP) COMP (like CLA and withdrawn by 1990) College BASE © TWBANTA-IUPUI
  32. 32. IN TN WE LEARNED1. No test measured 30% of gen ed skills2. Tests of generic skills measure primarilyprior learning3. Reliability of value added = .14. Test scores give few clues to guideimprovement actions © TWBANTA-IUPUI
  33. 33. AN INCONVENIENT TRUTH.9 = the correlation between SAT and CLA scores of institutions thus 81% of the variance in institutions’ scores is due to prior learning © TWBANTA-IUPUI
  34. 34. HOW MUCH OF THE VARIANCE IN SENIOR SCORES IS DUE TO COLLEGE IMPACT?• Student motivation to attend that institution(mission differences)• Student mix based on • age, gender • socioeconomic status • race/ethnicity • transfer status • college major © TWBANTA-IUPUI
  35. 35. HOW MUCH OF THE VARIANCE IN SENIOR SCORES IS DUE TO COLLEGE IMPACT? (CONTINUED) •Student motivation to do well •Sampling error •Measurement error •Test anxiety •College effects ______ 19 % © TWBANTA-IUPUI
  36. 36. STUDENT MOTIVATION• Samples of students are being tested• Extrinsic motivators (cash, prizes) are used We have learned:• Only a requirement and intrinsic motivationwill bring seniors in to do their best © TWBANTA-IUPUI
  37. 37. CONCERNS ABOUT VALUE ADDED •Student attrition •Proportion of transfer students •Different methods of calculating •Unreliability •Confounding effects of maturation © TWBANTA-IUPUI
  38. 38. Recent University of Texas Experience 30 – 40% of seniors at flagships earn highest CLA score (ceiling effect) flagship campuses have lowest value added scores © TWBANTA-IUPUI
  39. 39. WORD FROM MEASUREMENT EXPERTS Given the complexity of educational settings, we may never besatisfied that value added models can beused to appropriately partition the causaleffects of teacher, school, and student onmeasured changes in standardized testscores. - Henry Braun & Howard Wainer Handbook of Statistics, Vol. 26: Psychometrics Elsevier 2007 © TWBANTA-IUPUI
  40. 40. Employing currently available standardized tests of generic skills to compare the qualityof institutions is not a valid use of those tests. © TWBANTA-IUPUI
  41. 41. OECD’S AHELO COMPARING HEIS X NATIONS1. Generic skills (CLA)2. Disciplines (Engineering and Economics)3. Value added4. Contextual information indicators © TWBANTA-IUPUI
  42. 42. 2012K-12 standardized test scores are used to evaluate and compare schools assign grades to schools take over failing schools evaluate, compare, and fail teachers Yet NAEP scores have stagnated © TWBANTA-IUPUI
  43. 43. IN FINLAND AND SINGAPORE•No annual testing of students•No high-stakes accountability measures forteachers/schools•Scholarships for best and brightest•Starting pay like a doctor•Must complete master’s degree•Teachers are respected professionals © TWBANTA-IUPUI
  44. 44. SHORT-TERM PERSPECTIVE•Limit degrees to 120 SCH•Penalize students who go beyond a SCHcap•Reward graduation in 4 years•Consider earning potential in setting tuition © TWBANTA-IUPUI
  45. 45. DE-PROFESSIONALIZATION – IMMEDIATE PAYOFF•Teacher education is first•Industry certifications•Partnerships to fill employers’ needsDoes apprenticeship model prepare us forglobal leadership in the future? © TWBANTA-IUPUI
  46. 46. BETTER WAYS TO DEMONSTRATE ACCOUNTABILITYPerformance Indicators1.Access (to promote social mobility)2.Engaging student experience3.Workforce development4.Economic development5.Civic contribution of students, faculty,staff, graduates © TWBANTA-IUPUI
  47. 47. IF WE MUST MEASURE LEARNING LET’S USE:1. Standardized tests in major fields licensure and certification tests ETS Major Field Tests2. Internship performance3. Senior projects4. Study abroad performance5. Electronic portfolios6. External examiners © TWBANTA-IUPUI
  48. 48. START WITH MEASURES YOU HAVE•Assignments in courses•Course exams•Work performance•Records of progress through thecurriculum © TWBANTA-IUPUI
  49. 49. METHODS OF ASSESSMENTPaper and pencil testsIndividual or group projectsPortfoliosObservation of practiceObservation of simulated practiceAnalysis of case studiesAttitude or belief inventoriesInterviews and focus groupsSurveys © TWBANTA-IUPUI
  50. 50. Direct Measures of Learning Assignments, exams, projects, papersIndirect Measures Questionnaires, inventories, interviews - Did the course cover these objectives? - How much did your knowledge increase? - Did the teaching method(s) help you learn? - Did the assignments help you learn? GOOD ASSESSMENT INCLUDES BOTH © TWBANTA-IUPUI
  51. 51. NILOA SURVEYPROGRAM LEVEL APPROACHES1. Portfolios (80% in at least 1 area)2. Performance assessments3. Rubrics4. External judges5. Student interviews6. Employer surveys © TBANTA-IUPUI
  52. 52. STUDENT ELECTRONIC PORTFOLIO •Students take responsibility for demonstrating core skills •Unique individual skills and achievements can be emphasized •Multi-media opportunities extend possibilities •Metacognitive thinking is enhanced through reflection on contents - Sharon J. Hamilton IUPUI © TWBANTA-IUPUI
  53. 53. More use of RUBRICS locally developed VALUE from AAC&U © TWBANTA-IUPUI
  54. 54. VALUE RUBRICS•Critical thinking•Written communication•Oral communication•Information literacy•Teamwork•Intercultural knowledge•Ethical reasoning © TWBANTA-IUPUI
  55. 55. ACCOUNTABILITY REPORT•85% achieve Outstanding ratings in writingas defined . . .•78% are Outstanding in applying knowledgeand skills in internships•75% are Outstanding in delivering an oralpresentation © TWBANTA-IUPUI
  56. 56. FOR EXTERNAL CREDIBILITYCollaborate on rubricsUse employers as examinersConduct process audits © TWBANTA-IUPUI
  57. 57. E-PORT CHALLENGES•Reliability of rubrics•Student motivation if used for assessment (Barrett, 2009)•Differences in topics for products to beevaluated (Sekolsky & Wentland, 2010) © TWBANTA-IUPUI
  58. 58. OBSTACLES TO USING PERFORMANCE-BASED MEASURES•Defining domains and constructs•Obtaining agreement on what to measureand definitions•Defining reliability and validity•Creating good measures - Tom Zane WGU © TWBANTA-IUPUI
  59. 59. WILL IT TAKE 80 YEARS . . . ?3 Promising Alternatives E portfolios Rubrics Assessmentcommunities - Banta, Griffin, Flateby,Kahn NILOA Paper #2 (2009) © TWBANTA-IUPUI
  60. 60. TEAGLE ASSESSMENT SCHOLARS•study assessment data•visit campuses•talk with 3-4 groups of students•talk with faculty about their campusassessment data - Charles Blaich Wabash College © TWBANTA-IUPUI
  61. 61. NATIONAL SURVEY OF STUDENT ENGAGEMENT AT ~ HOPE COLLEGE ~ Seniors Freshmen 2003 STUDENTS STUDYING LESS THAN % 38% 39% 2010 1021% /WEEK HOURS 28% © TWBANTA-IUPUI
  62. 62. HOPE COLLEGE•Considered data over supper•Proposed solutions•Conducted student focus groups•Shared all data with all faculty•Departments dedicated a meeting to preparestrategies to increase rigor © TWBANTA-IUPUI
  63. 63. NATIONAL INSTITUTE FORLEARNING OUTCOMES ASSESSMENT•Surveys 2009 CAOs 2011 Departments•Occasional Papers•Website review, standards•Quick comments (monthly)•Calendar of events © TWBANTA-IUPUI
  64. 64. LUMINADegree Qualifications Profile - Linked student learning outcomes– AA to MA levelsSuggests transfer based on assessmentof learning outcomes © TWBANTA-IUPUI
  65. 65. NEW LEADERSHIP ALLIANCEFOR STUDENT LEARNING AND ACCOUNTABILITY - Presidents’ Alliance - Certification Process Set ambitious goals for learning Gather evidence of learning Use evidence to improve learning Report evidence and results © TWBANTA-IUPUI
  66. 66. BUILD ASSESSMENT INTO VALUED PROCESSES1. Assessment of learning2. Curriculum review and revision3. Survey research4. Program review5. Scholarship of Teaching & Learning6. Evaluation of initiatives7. Faculty development8. Promotion & tenure9. Rewards and recognition © TWBANTA-IUPUI

×