The Proof is in the Portfolio: AnArchitecture of the Good, the Bad,        and the Mediocre                    Terrel L Rh...
What is AAC&U?         Founded in 1915, AAC&U is    dedicated to making the aims of    liberal learning a vigorous and   ...
Context:Changing Designs for College         Learning   The Twenty-First Century Academy         A Curriculum in Transitio...
4
US Economy Defined by Greater Workplace       Challenges and DynamismEvery year, more than 1/3 of the entire US labor for...
What Employers Say“[Employers] generally are...frustrated with their inability to find  „360 degree people‟ who have both ...
The Growing Demand for Higher Order Skills    Source: Council on Competitiveness, Competitiveness Index7
Peter D                    HART                              RESEARCH                              ASSOC I A T ESRaising T...
Employers’ Expectations of    Employees Have Increased % who agree with each statementOur company is asking employees to t...
Broad Skills/Knowledge AND Specific Skills/Knowledge Are Needed for Career SuccessWhich is more important for recent colle...
Employers’ Top Priorities For Student      Learning Outcomes In College    % saying two- and four-year colleges should pla...
Higher Level Liberal Education       Skills and Abilities                =          Higher WagesData from Georgetown Unive...
The Salary Premium for Liberal           Education OutcomesFrom a federal database analyzing qualifications for 1,100 diff...
Mean Earnings of Jobs that                  Emphasize Writing                        Mean earnings of writing quintiles   ...
Mean Earnings of Jobs that                      Emphasize Speaking                       Mean earnings of speaking quintil...
Mean Earnings of Jobs that                 Emphasize Originality                  Mean earnings of originality quintiles  ...
Mean Earnings of Jobs that               Emphasize Problem Solving                    Earnings of complex problem solving ...
Mean Earnings of Jobs that                Emphasize Physical Ability                     Mean earnings of physical ability...
Where We Are Now In short, we know “what works” – to foster both learning gains and greater completionBut many students ar...
E-Portfolios Can Capture:Essential Learning OutcomesA Guiding Vision and National Benchmarks for College Learning andLiber...
Narrow Learning is Not Enough   The LEAP Essential Learning Outcomes Knowledge of Human Cultures and the Physical and Nat...
Aims/Outcomes Addressed       Across the CurriculumFirst to Final YearIntegrating Liberal and Professional LearningCo-C...
Employers Assess the Potential Value of   Emerging Educational Practices % saying each would help a lot/fair amount to pre...
High-Impact Practices                                   (HIPs)        Correlate Highly with NSSE Gains on Student        ...
High Impact Practices First-Year Seminars and Experiences Common Intellectual Experiences Learning Communities Writing...
The Eleventh High Impact Practice Electronic portfolios                    26
Impact of Educationally Purposeful Practices on First                        Academic Year GPA by Pre-College Achievement ...
Impact of Educationally Purposeful Practices on the Probability of                                          Returning for ...
Huber, CSU - Northridge                                                                       Figure 4. Impact of Particip...
Participation Levels in High      Impact Practices     First Year (NSSE Data)       Learning Communities 17%       Servic...
Participation Levels in High      Impact Practices    Seniors (NSSE Data)  Research With Faculty     19%   Internship    ...
The Challenge Ahead – Making High Impact   Practices Central Rather than Optional
The Key Elements for a CompellingQuality Framework Already Are in Hand  Consensus Aims and Outcomes  Practices that Fost...
The Power of Rubrics and E-Portfolios as Tools for Both       Assessment and High-Impact Learning • Rubrics to help guide ...
VALUE Projecthttp://www.aacu.org/programs/VALUE
An obvious place to begin:Help Students UnderstandWhat They Are Expected to       Accomplish
VALUE Rubrics & Assessment                  Rubrics BasicsCriteria
VALUE Rubrics & Assessment       Rubrics Basics                             Levels
VALUE Rubrics & Assessment       Rubrics Basics                        Performance                        Descriptors
Validity and Usability• Over 2000 distinct institutions have downloaded one or more  of the VALUE rubrics for use• Over 11...
Reliability Study• 40 Faculty• 4 Traditional Disciplinary Divisions – Humanities,  Social Sciences, STEM, Professions• Thr...
Building the Evidentiary Base                     • University of Kansas – Representing ResultsPercent of Ratings         ...
Building the Evidentiary Base                     • University of Kansas – Representing Results                        Cri...
Building the Evidentiary Base                          “VALUE added” for 4 years - writingPercent of Ratings
Building the Evidentiary Base• University of Kansas –  • “analysis of the data from the AACU VALUE rubrics    affirmed tha...
Building the Evidentiary Base• University of Kansas –  • “Interestingly, the patterns that were visible in the VALUE    ru...
Roanoke College                         WritingReliability = >.8
Using the Results“…we excluded the scores for those instructors and ran  frequencies and descriptive statistics on the cat...
Table 1 Information Literacy Results             Building the Evidentiary Base             University of North Carolina - ...
Building the Evidentiary BaseUniversity of North Carolina - Wilmington      Table 2. Critical Thinking Results            ...
Building the Evidentiary BaseLong Island University – Brooklyn CampusThe results of this initial assessment are early benc...
Implications: VSA and CLA• VSA institutions can select from CAAP, CLA,  or ETS Proficiency Profile• Important consideratio...
One Approach toCapturing and Tracking       Learning     E-Portfolios
Portland State University                                     Freshman Inquiry (FRINQ)                                    ...
Queensborough College• Remedial/Developmental Communications  Course• First Year Writing Course• Disciplinary Course• Web-...
Literature, Language &Communication
We have had our whys, hows, and whats upside-                           down,focusing too much on what should be learned, ...
Curricular and PedagogicalInnovations – Led by Faculty – AreCreating a 21st Century Vision and  Practice for Liberal Educa...
The proof is in the portfolio: An architecture of the good, the bad and the mediocre
The proof is in the portfolio: An architecture of the good, the bad and the mediocre
The proof is in the portfolio: An architecture of the good, the bad and the mediocre
The proof is in the portfolio: An architecture of the good, the bad and the mediocre
The proof is in the portfolio: An architecture of the good, the bad and the mediocre
The proof is in the portfolio: An architecture of the good, the bad and the mediocre
The proof is in the portfolio: An architecture of the good, the bad and the mediocre
Upcoming SlideShare
Loading in …5
×

The proof is in the portfolio: An architecture of the good, the bad and the mediocre

827 views

Published on

Terrel L Rhodes

Published in: Education, Career
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
827
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

The proof is in the portfolio: An architecture of the good, the bad and the mediocre

  1. 1. The Proof is in the Portfolio: AnArchitecture of the Good, the Bad, and the Mediocre Terrel L Rhodes Association of American Colleges and Universities Australia October 18, 2011
  2. 2. What is AAC&U? Founded in 1915, AAC&U is dedicated to making the aims of liberal learning a vigorous and constant influence on institutional planning and educational practice in college. It is a meeting ground for all sectors of higher education and brings together faculty, academic and student affairs leaders and presidents across sectors, divisions, and disciplines to explore the aims of education, the future of the academy, and strategies for institutional change and higher student achievement. 2
  3. 3. Context:Changing Designs for College Learning The Twenty-First Century Academy A Curriculum in Transition: Rethinking educational purposes and practices to better prepare all students for • complexity and contingency • global interdependence • innovation in the workplace • diverse democracy 3
  4. 4. 4
  5. 5. US Economy Defined by Greater Workplace Challenges and DynamismEvery year, more than 1/3 of the entire US labor force changes jobs.Todays Students Will Have 10-14 Jobs by the Time They Are 38.50% of Workers Have Been With Their Company Less Than 5 Years.Every year, more than 30 million Americans are working in jobs that did not exist in the previous quarter. DOL-BLS 5
  6. 6. What Employers Say“[Employers] generally are...frustrated with their inability to find „360 degree people‟ who have both the specific job/technical skills and the broader skills (communication and problem- solving skills, work ethic, and ability to work with others) necessary to promise greater success for both the individual and the employer.” From Peter D. Hart Research Associates, Report of Findings Based on Focus Groups Among Business Executives (AAC&U, 2006) 6
  7. 7. The Growing Demand for Higher Order Skills Source: Council on Competitiveness, Competitiveness Index7
  8. 8. Peter D HART RESEARCH ASSOC I A T ESRaising The BarEmployers’ Views On College LearningIn The Wake Of The Economic Downturn Key findings from survey among 302 employers Conducted October 27 – November 17, 2009 for
  9. 9. Employers’ Expectations of Employees Have Increased % who agree with each statementOur company is asking employees to take on more responsibilitiesand to use a broader set of skills than in the past 91%Employees are expected to work harder to coordinate with otherdepartments than in the past 90%The challenges employees face within our company are more complextoday than they were in the past 88%To succeed in our company, employees need higher levels of learningand knowledge today than they did in the past 88%
  10. 10. Broad Skills/Knowledge AND Specific Skills/Knowledge Are Needed for Career SuccessWhich is more important for recent college graduates whowant to pursue advancement and long-term career success atyour company?BOTH in-depth AND broad range of skills andknowledge 59%Broad range of skills and knowledge that apply to arange of fields or positions 20%In-depth knowledge and skills that apply to a specificfield or position 20%
  11. 11. Employers’ Top Priorities For Student Learning Outcomes In College % saying two- and four-year colleges should place MORE emphasis on helping students develop these skills, qualities, capabilities, knowledge Effective oral/written communication 89% Critical thinking/ analytical reasoning 81% Knowledge/skills applied 79% to real world settings Analyze/solve complex 75% problems Connect choices andactions to ethical decisions 75%Teamwork skills/ ability to collaborate 71% Ability to innovate and be creative 70%Concepts/developments in science/technology 70%
  12. 12. Higher Level Liberal Education Skills and Abilities = Higher WagesData from Georgetown University Center for Education and the Workforce Center on Education and the Workforce 12
  13. 13. The Salary Premium for Liberal Education OutcomesFrom a federal database analyzing qualifications for 1,100 different jobs, there is consistent evidence that the highest salaries apply to positions that call for intensive use of liberal education capabilities, including (random order): Inductive and Deductive Reasoning Judgment and Decision Making Problem Solving Social/Interpersonal Skills Mathematics Originality WritingSource: Georgetown University Center on Education and the Workforce 13
  14. 14. Mean Earnings of Jobs that Emphasize Writing Mean earnings of writing quintiles 70,000earnings 35,000 Earnings 0 q1(low) q2 q3 q4 q5(high) quintilesSource: Georgetown University Center for Education and the Workforce14
  15. 15. Mean Earnings of Jobs that Emphasize Speaking Mean earnings of speaking quintiles 60,000earnings 30,000 Earnin… 0 q1(low) q2 q3 q4 q5(high) quintiles Source: Georgetown University Center for Education and the Workforce15
  16. 16. Mean Earnings of Jobs that Emphasize Originality Mean earnings of originality quintiles 60,000 earnings Earnings 30,000 0 q1(low) q2 q3 q4 q5(high) quintilesSource: Georgetown University Center for Education and the Workforce16
  17. 17. Mean Earnings of Jobs that Emphasize Problem Solving Earnings of complex problem solving quintiles 70,000earnings 35,000 Earnings 0 q1(low) q2 q3 q4 q5(high) quintilesSource: Georgetown University Center for Education and the Workforce17
  18. 18. Mean Earnings of Jobs that Emphasize Physical Ability Mean earnings of physical ability quintiles 60,000 Earnings earnings 30,000 0 q1(low) q2 q3 q4 q5(high) quintilesSource: Georgetown University Center for Education and the Workforce18
  19. 19. Where We Are Now In short, we know “what works” – to foster both learning gains and greater completionBut many students aren‟t doing “what works” 19
  20. 20. E-Portfolios Can Capture:Essential Learning OutcomesA Guiding Vision and National Benchmarks for College Learning andLiberal Education in the 21st CenturyHigh Impact PracticesHelping Students Achieve the Essential Learning OutcomesAuthentic AssessmentsProbing Whether Students Can APPLY Their Learning – to ComplexProblems and Real-World Challenges 20
  21. 21. Narrow Learning is Not Enough The LEAP Essential Learning Outcomes Knowledge of Human Cultures and the Physical and Natural World Focused on engagement with big questions, enduring and contemporary Intellectual and Practical Skills Practiced extensively across the curriculum, in the context of progressively more challenging problems, projects, and standards for performance Personal and Social Responsibility Anchored through active involvement with diverse communities and real-world challenges Integrative Learning Demonstrated through the application of knowledge, skills, and responsibilities to new settings and complex problems 21
  22. 22. Aims/Outcomes Addressed Across the CurriculumFirst to Final YearIntegrating Liberal and Professional LearningCo-Curriculum as WellAssessments that Deepen LearningSustained Focus on Underserved Students
  23. 23. Employers Assess the Potential Value of Emerging Educational Practices % saying each would help a lot/fair amount to prepare college students for success84% Expecting students to complete a significant project before graduation that demonstrates their depth of knowledge in their major AND their acquisition of analytical, problem-solving, and communication skills (62% help a lot)81% Expecting students to complete an internship or community-based field project to connect classroom learning with real-world experiences (66%)81% Ensuring that students develop the skills to research questions in their field and develop evidence-based analyses (57%)73% Expecting students to work through ethical issues and debates to form their own judgments about the issues at stake (48%) Source: Raising the Bar (AAC&U, 2010)
  24. 24. High-Impact Practices (HIPs) Correlate Highly with NSSE Gains on Student Learning Outcomes Provide “Compensatory Benefit” for Students With Lower Test Scores and/or High Drop-Out RatesKuh, High Impact Practices: What They Are, Who Has Access toThem, and Why They Matter (AAC&U 2008)
  25. 25. High Impact Practices First-Year Seminars and Experiences Common Intellectual Experiences Learning Communities Writing-Intensive Courses Collaborative Assignments and Projects Undergraduate Research Diversity/Global Learning Service Learning, Community-Based Learning Internships Capstone Courses and Projects
  26. 26. The Eleventh High Impact Practice Electronic portfolios 26
  27. 27. Impact of Educationally Purposeful Practices on First Academic Year GPA by Pre-College Achievement Level 4.00 3.75 3.50First-year GPA 3.25 3.00 2.75 2.50 ACT 28 ACT 24 2.25 ACT 20 2.00 -2 -1 0 1 2 Educationally Purposeful Activities (standardized)
  28. 28. Impact of Educationally Purposeful Practices on the Probability of Returning for the Second Year of College by Race 1.00 0.95 0.90 Probabilyt of Returning 0.85 0.80 0.75 0.70 0.65 0.60 African American 0.55 White/Caucasian 0.50 -2 -1 0 1 2 Educationally Purposeful Activities (standardized)*Findings developed by LEAP National LeadershipCouncil Member George Kuh and used with permission
  29. 29. Huber, CSU - Northridge Figure 4. Impact of Participation in High Impact Practices on Percentage of Senior NSSE Respondents Graduating on Time by Racial & Ethnic Background 80.0% None 1 HIP 2 HIPs 3 or more HIPs 73% 69% Peratage Graduating "on Time" (i.e., in 2006-07) 70.0% 68% 65% 63% 60.0% 54% 50.0% 48% 40.0% 38% 30.0% 20.0% [V=.255 (.007)] [V=.109 (.094)] Latina/o Respondents Other RespondentsMultiple HIPs distributed through general education and major programs would, we areconvinced, “require only small curricular changes.” Such “modest change” can yield asignificant increase in student success and persistence. 29
  30. 30. Participation Levels in High Impact Practices First Year (NSSE Data) Learning Communities 17% Service Learning 36%
  31. 31. Participation Levels in High Impact Practices Seniors (NSSE Data) Research With Faculty 19% Internship 53% Service Learning 46% Study Abroad 19% Senior Culminating Work 32%
  32. 32. The Challenge Ahead – Making High Impact Practices Central Rather than Optional
  33. 33. The Key Elements for a CompellingQuality Framework Already Are in Hand Consensus Aims and Outcomes Practices that Foster Achievement AND Completion Evidence on “What Works” for Underserved Students Assessments That Raise – and Reveal – the Level of Learning
  34. 34. The Power of Rubrics and E-Portfolios as Tools for Both Assessment and High-Impact Learning • Rubrics to help guide students and faculty • Places individual faculty judgment within national shared experience; national benchmarks • E-portfolios to gather students’ best work, encourage self- assessment, and allow for mining of samples for assessment purposes • E-portfolios are portable, allow for cumulative learning and assessment, can complement other high-impact practices • Can build up from course level to institutional reporting needs AND down from general to specific program/course context 34
  35. 35. VALUE Projecthttp://www.aacu.org/programs/VALUE
  36. 36. An obvious place to begin:Help Students UnderstandWhat They Are Expected to Accomplish
  37. 37. VALUE Rubrics & Assessment Rubrics BasicsCriteria
  38. 38. VALUE Rubrics & Assessment Rubrics Basics Levels
  39. 39. VALUE Rubrics & Assessment Rubrics Basics Performance Descriptors
  40. 40. Validity and Usability• Over 2000 distinct institutions have downloaded one or more of the VALUE rubrics for use• Over 11,000 distinct individuals have downloaded one or more of the VALUE rubrics for use• 3 major consortia are using VALUE rubrics for cross institutional collaboration – Connect2Learning – LaGuardia College/AAEEBL (FIPSE) – 23 campuses; Integrative Portfolio Process – Michigan (FIPSE) – 6 campuses; RAILS – Syracuse (Institute for Museum and Library Studies ACRL) – 10 campuses• Major state university systems are using VALUE rubrics• Being used worldwide, e.g. Japan, Hong Kong, Australia, United Arab Emirates
  41. 41. Reliability Study• 40 Faculty• 4 Traditional Disciplinary Divisions – Humanities, Social Sciences, STEM, Professions• Three VALUE rubrics – Critical Thinking, Civic Engagement, Integrative Learning• Common set of student portfolio work• Agreement = .66 without norming; .8 normed• Another set of 5 campuses, using same set of rubrics with 500 samples of student work – still analyzing
  42. 42. Building the Evidentiary Base • University of Kansas – Representing ResultsPercent of Ratings Critical Thinking: Issues, Analysis, and Conclusions Inter-rater reliability = >.8
  43. 43. Building the Evidentiary Base • University of Kansas – Representing Results Critical Thinking: Evaluation of Sources and EvidencePercent of Ratings
  44. 44. Building the Evidentiary Base “VALUE added” for 4 years - writingPercent of Ratings
  45. 45. Building the Evidentiary Base• University of Kansas – • “analysis of the data from the AACU VALUE rubrics affirmed that a team approach to course design can yield larger improvement in some forms of student writing and thinking” • “We also saw that the rubrics work best when there is close alignment between the nature of the assignment and the dimensions of intellectual skill described in the rubric” • “Finally, at a practical level we are very encouraged that this process is manageable and sustainable”
  46. 46. Building the Evidentiary Base• University of Kansas – • “Interestingly, the patterns that were visible in the VALUE rubric scores were not mirrored in the CLA scores; students in the team-designed and traditional courses performed no differently on the CLA. Students‟ performance on the CLA, moreover, was generally unrelated to the VALUE rubric ratings of their coursework, as well as the instructors‟ grading of the same coursework. In contrast, the latter two measures were highly correlated, suggesting that the VALUE rubrics capture qualities of critical thinking and writing that fit well with what faculty members VALUE in their students‟ work.”
  47. 47. Roanoke College WritingReliability = >.8
  48. 48. Using the Results“…we excluded the scores for those instructors and ran frequencies and descriptive statistics on the categories again. We found the means for the rubric categories of Focus and Thesis and Organization remained close to 2.5 with the scores of 2 and 3 occurring most often (approx. 2.4) while most scores for the categories of Evidence and Reasoning were 2s. The category of Style and Mechanics was in the middle with a mean of 2.3. Our adjusted results support what most faculty believe about the writing of our first-year students, they can learn to develop a thesis and organize a paper more effectively than they can provide convincing evidence and strong reasoning to support the thesis.”
  49. 49. Table 1 Information Literacy Results Building the Evidentiary Base University of North Carolina - Wilmington Table 1. Information Literacy Results Dimension % of Work Products % of Work Products Scored 2 or higher Scored 3 or Higher IL1 Determine Information Needed 87.2% 46.2% IL2 Access Needed Information 89.6% 46.8% IL3 Evaluate Information and Sources 88.5% 39.7% IL4 Use Information Effectively 85.9% 43.6% IL5 Access and Use Information Ethically 93.6% 59.0% Inter-rater reliability = >.8
  50. 50. Building the Evidentiary BaseUniversity of North Carolina - Wilmington Table 2. Critical Thinking Results Dimension % of Work Products % of Work Products Scored 2 or higher Scored 3 or Higher CT1 Explanation of Issues 68.3% 35.5% CT2 Evidence Year1 65.0% 28.2% CT2 Evidence Year 2* Interpreting and Analysis 72.8% 38.6% Questioning viewpoint 40.9% 13.6% CT3 Influence of Context and Assumptions 48.8% 21.2% CT4 Student’s Position 54.5% 24.0% CT5 Conclusions and Related Outcomes 47.7% 17.0%
  51. 51. Building the Evidentiary BaseLong Island University – Brooklyn CampusThe results of this initial assessment are early benchmarks important for all institutions that expect integrative learning to emerge among students‟ upper level accomplishments. These results help us see how students incorporate unmediated experiential learning into deeper transference and comprehension. In fact, students welcome the opportunity to integrate life experience, course work, and texts early in their undergraduate experience, if invited to do so.
  52. 52. Implications: VSA and CLA• VSA institutions can select from CAAP, CLA, or ETS Proficiency Profile• Important considerations for selection – Acceptance by students, faculty, administrators or other policy makers – Trade-offs in cost, ease of administration – Utility of the test for other purposes - supporting campus activities and services or providing guidance on improving learning 55
  53. 53. One Approach toCapturing and Tracking Learning E-Portfolios
  54. 54. Portland State University Freshman Inquiry (FRINQ) UNST 100-level class A year-long, theme-based course with a mentor section that corresponds with the class. Transfer Transition UNST 200- and 300- level classes One-term course designed specifically for transfer students, with a mentor section that corresponds with the class. Sophomore Inquiry (SINQ) UNST 200-level classes One-term courses with a mentor section that corresponds with the class. Sophomore Inquiry Sophomore Inquiry Sophomore Inquiry Upper Division ClusterClasses designated with a "U" offered by academic departments. Three courses in one cluster linked to one of the Sophomore Inquiry courses. Cluster Course One Cluster Course Three Cluster Course Two Senior Capstone UNST 400-level class A 6-credit, community-based learning class.
  55. 55. Queensborough College• Remedial/Developmental Communications Course• First Year Writing Course• Disciplinary Course• Web-based portfolio/wiki 58
  56. 56. Literature, Language &Communication
  57. 57. We have had our whys, hows, and whats upside- down,focusing too much on what should be learned, than on how, and often forgetting the why altogether. In a world of nearly infinite information, we must firstaddress why, facilitate how, and let the what generate naturally from there.Michael Wesch, “From Knowledgeable to Knowledge-able,” Academic Commons, January 2009 (academiccommons.org) 64
  58. 58. Curricular and PedagogicalInnovations – Led by Faculty – AreCreating a 21st Century Vision and Practice for Liberal Education Directly Connected to the Needs and Experiences of Today‟s Diverse Students, our Diverse Democracy, and Interdependent Global Community 65

×