Terrel L Rhodes

1. The Proof is in the Portfolio: An
Architecture of the Good, the Bad,
and the Mediocre
Terrel L Rhodes
Association of American Colleges and Universities
Australia
October 18, 2011

2. What is AAC&U?
 Founded in 1915, AAC&U is
dedicated to making the aims of
liberal learning a vigorous and
constant influence on institutional
planning and educational practice in
college. It is a meeting ground for all
sectors of higher education and
brings together faculty, academic and
student affairs leaders and presidents
across sectors, divisions, and
disciplines to explore the aims of
education, the future of the academy,
and strategies for institutional
change and higher student
achievement.
2

3. Context:
Changing Designs for College
Learning
The Twenty-First Century Academy
A Curriculum in Transition:
Rethinking educational purposes and practices to
better prepare all students for
• complexity and contingency
• global interdependence
• innovation in the workplace
• diverse democracy
3

4. 4

5. US Economy Defined by Greater Workplace
Challenges and Dynamism
Every year, more than 1/3 of the entire US labor force
changes jobs.
Today's Students Will Have 10-14 Jobs by the Time They
Are 38.
50% of Workers Have Been With Their Company Less
Than 5 Years.
Every year, more than 30 million Americans are working in
jobs that did not exist in the previous quarter.
DOL-BLS
5

6. What Employers Say
“[Employers] generally are...frustrated with their inability to find
„360 degree people‟ who have both the specific job/technical
skills and the broader skills (communication and problem-
solving skills, work ethic, and ability to work with others)
necessary to promise greater success for both the individual
and the employer.”
From Peter D. Hart Research Associates, Report
of Findings Based on Focus Groups Among
Business Executives (AAC&U, 2006)
6

7. The Growing Demand for Higher Order Skills
Source: Council on Competitiveness, Competitiveness Index
7

8. Peter D
HART
RESEARCH
ASSOC I A T ES
Raising The Bar
Employers’ Views On College Learning
In The Wake Of The Economic Downturn
Key findings from survey among 302 employers
Conducted October 27 – November 17, 2009
for

9. Employers’ Expectations of
Employees Have Increased
% who agree with each statement
Our company is asking employees to take on more responsibilities
and to use a broader set of skills than in the past
91%
Employees are expected to work harder to coordinate with other
departments than in the past
90%
The challenges employees face within our company are more complex
today than they were in the past
88%
To succeed in our company, employees need higher levels of learning
and knowledge today than they did in the past
88%

10. Broad Skills/Knowledge AND Specific Skills/
Knowledge Are Needed for Career Success
Which is more important for recent college graduates who
want to pursue advancement and long-term career success at
your company?
BOTH in-depth AND broad range of skills and
knowledge
59%
Broad range of skills and knowledge that apply to a
range of fields or positions
20%
In-depth knowledge and skills that apply to a specific
field or position
20%

11. Employers’ Top Priorities For Student
Learning Outcomes In College
% saying two- and four-year colleges should place MORE emphasis on
helping students develop these skills, qualities, capabilities, knowledge
Effective oral/written
communication 89%
Critical thinking/
analytical reasoning 81%
Knowledge/skills applied 79%
to real world settings
Analyze/solve complex 75%
problems
Connect choices and
actions to ethical decisions 75%
Teamwork skills/ ability to
collaborate 71%
Ability to innovate and
be creative 70%
Concepts/developments in
science/technology 70%

12. Higher Level Liberal Education
Skills and Abilities
=
Higher Wages
Data from Georgetown University Center for Education and
the Workforce
Center on Education and
the Workforce
12

13. The Salary Premium for Liberal
Education Outcomes
From a federal database analyzing qualifications for 1,100 different jobs,
there is consistent evidence that the highest salaries apply to positions
that call for intensive use of liberal education capabilities, including
(random order):
Inductive and Deductive Reasoning
Judgment and Decision Making
Problem Solving
Social/Interpersonal Skills
Mathematics
Originality
Writing
Source: Georgetown University Center on Education and the Workforce
13

14. Mean Earnings of Jobs that
Emphasize Writing
Mean earnings of writing quintiles
70,000
earnings
35,000 Earnings
0
q1(low) q2 q3 q4 q5(high)
quintiles
Source: Georgetown University Center for Education and the Workforce14

15. Mean Earnings of Jobs that
Emphasize Speaking
Mean earnings of speaking quintiles
60,000
earnings
30,000
Earnin…
0
q1(low) q2 q3 q4 q5(high)
quintiles
Source: Georgetown University Center for Education and the Workforce15

16. Mean Earnings of Jobs that
Emphasize Originality
Mean earnings of originality quintiles
60,000
earnings
Earnings
30,000
0
q1(low) q2 q3 q4 q5(high)
quintiles
Source: Georgetown University Center for Education and the Workforce16

17. Mean Earnings of Jobs that
Emphasize Problem Solving
Earnings of complex problem solving quintiles
70,000
earnings
35,000 Earnings
0
q1(low) q2 q3 q4 q5(high)
quintiles
Source: Georgetown University Center for Education and the Workforce17

18. Mean Earnings of Jobs that
Emphasize Physical Ability
Mean earnings of physical ability quintiles
60,000
Earnings
earnings
30,000
0
q1(low) q2 q3 q4 q5(high)
quintiles
Source: Georgetown University Center for Education and the Workforce18

19. Where We Are Now
In short, we know “what works” – to foster
both learning gains and greater completion
But many students aren‟t doing “what works”
19

20. E-Portfolios Can Capture:
Essential Learning Outcomes
A Guiding Vision and National Benchmarks for College Learning and
Liberal Education in the 21st Century
High Impact Practices
Helping Students Achieve the Essential Learning Outcomes
Authentic Assessments
Probing Whether Students Can APPLY Their Learning – to Complex
Problems and Real-World Challenges
20

21. Narrow Learning is Not Enough
The LEAP Essential Learning Outcomes
 Knowledge of Human Cultures and the Physical and Natural World
Focused on engagement with big questions, enduring and contemporary
 Intellectual and Practical Skills
Practiced extensively across the curriculum, in the context of progressively more
challenging problems, projects, and standards for performance
 Personal and Social Responsibility
Anchored through active involvement with diverse communities and real-world
challenges
 Integrative Learning
Demonstrated through the application of knowledge, skills, and responsibilities to
new settings and complex problems
21

22. Aims/Outcomes Addressed
Across the Curriculum
First to Final Year
Integrating Liberal and Professional Learning
Co-Curriculum as Well
Assessments that Deepen Learning
Sustained Focus on Underserved Students

23. Employers Assess the Potential Value of
Emerging Educational Practices
% saying each would help a lot/fair amount to prepare college
students for success
84% Expecting students to complete a significant project before graduation
that demonstrates their depth of knowledge in their major AND their
acquisition of analytical, problem-solving, and communication skills
(62% help a lot)
81% Expecting students to complete an internship or community-based
field project to connect classroom learning with real-world experiences
(66%)
81% Ensuring that students develop the skills to research questions in their
field and develop evidence-based analyses (57%)
73% Expecting students to work through ethical issues and debates to form
their own judgments about the issues at stake (48%)
Source: Raising the Bar (AAC&U, 2010)

24. High-Impact Practices
(HIPs)
Correlate Highly with NSSE Gains on Student
Learning Outcomes
Provide “Compensatory Benefit” for Students
With Lower Test Scores and/or High Drop-Out
Rates
Kuh, High Impact Practices: What They Are, Who Has Access to
Them, and Why They Matter (AAC&U 2008)

25. High Impact Practices
 First-Year Seminars and Experiences
 Common Intellectual Experiences
 Learning Communities
 Writing-Intensive Courses
 Collaborative Assignments and Projects
 Undergraduate Research
 Diversity/Global Learning
 Service Learning, Community-Based Learning
 Internships
 Capstone Courses and Projects

26. The Eleventh High
Impact Practice
Electronic portfolios
26

27. Impact of Educationally Purposeful Practices on First
Academic Year GPA by Pre-College Achievement Level
4.00
3.75
3.50
First-year GPA
3.25
3.00
2.75
2.50 ACT 28
ACT 24
2.25
ACT 20
2.00
-2 -1 0 1 2
Educationally Purposeful Activities
(standardized)

28. Impact of Educationally Purposeful Practices on the Probability of
Returning for the Second Year of College by Race
1.00
0.95
0.90
Probabilyt of Returning
0.85
0.80
0.75
0.70
0.65
0.60 African American
0.55 White/Caucasian
0.50
-2 -1 0 1 2
Educationally Purposeful Activities
(standardized)
*Findings developed by LEAP National Leadership
Council Member George Kuh and used with permission

29. Huber, CSU - Northridge
Figure 4. Impact of Participation in High Impact Practices on Percentage of
Senior NSSE Respondents Graduating on Time by Racial & Ethnic Background
80.0%
None 1 HIP 2 HIPs 3 or more HIPs
73%
69%
Peratage Graduating "on Time" (i.e., in 2006-07)
70.0% 68%
65%
63%
60.0%
54%
50.0% 48%
40.0% 38%
30.0%
20.0%
[V=.255 (.007)] [V=.109 (.094)]
Latina/o Respondents Other Respondents
Multiple HIPs distributed through general education and major programs would, we are
convinced, “require only small curricular changes.” Such “modest change” can yield a
significant increase in student success and persistence.
29

30. Participation Levels in High
Impact Practices
First Year (NSSE Data)
Learning Communities 17%
Service Learning 36%

31. Participation Levels in High
Impact Practices
Seniors (NSSE Data)
Research With Faculty 19%
Internship 53%
Service Learning 46%
Study Abroad 19%
Senior Culminating Work 32%

32. The Challenge Ahead –
Making High Impact
Practices Central
Rather than Optional

33. The Key Elements for a Compelling
Quality Framework Already Are in Hand
Consensus Aims and Outcomes
Practices that Foster Achievement AND
Completion
Evidence on “What Works” for
Underserved Students
Assessments That Raise – and Reveal – the
Level of Learning

34. The Power of Rubrics and E-Portfolios as Tools for Both
Assessment and High-Impact Learning
• Rubrics to help guide students and faculty
• Places individual faculty judgment within national shared
experience; national benchmarks
• E-portfolios to gather students’ best work, encourage self-
assessment, and allow for mining of samples for assessment
purposes
• E-portfolios are portable, allow for cumulative learning and
assessment, can complement other high-impact practices
• Can build up from course level to institutional reporting
needs AND down from general to specific program/course
context
34

35. VALUE Project
http://www.aacu.org/programs/VALUE

36. An obvious place to begin:
Help Students Understand
What They Are Expected to
Accomplish

39. VALUE Rubrics & Assessment
Rubrics Basics
Criteria

40. VALUE Rubrics & Assessment
Rubrics Basics
Levels

41. VALUE Rubrics & Assessment
Rubrics Basics
Performance
Descriptors

43. Validity and Usability
• Over 2000 distinct institutions have downloaded one or more
of the VALUE rubrics for use
• Over 11,000 distinct individuals have downloaded one or more
of the VALUE rubrics for use
• 3 major consortia are using VALUE rubrics for cross
institutional collaboration – Connect2Learning – LaGuardia
College/AAEEBL (FIPSE) – 23 campuses; Integrative Portfolio
Process – Michigan (FIPSE) – 6 campuses; RAILS –
Syracuse (Institute for Museum and Library Studies ACRL) –
10 campuses
• Major state university systems are using VALUE rubrics
• Being used worldwide, e.g. Japan, Hong Kong, Australia,
United Arab Emirates

44. Reliability Study
• 40 Faculty
• 4 Traditional Disciplinary Divisions – Humanities,
Social Sciences, STEM, Professions
• Three VALUE rubrics – Critical Thinking, Civic
Engagement, Integrative Learning
• Common set of student portfolio work
• Agreement = .66 without norming; .8 normed
• Another set of 5 campuses, using same set of
rubrics with 500 samples of student work – still
analyzing

45. Building the Evidentiary Base
• University of Kansas – Representing Results
Percent of Ratings
Critical Thinking: Issues, Analysis, and Conclusions
Inter-rater reliability = >.8

46. Building the Evidentiary Base
• University of Kansas – Representing Results
Critical Thinking: Evaluation of Sources and Evidence
Percent of Ratings

47. Building the Evidentiary Base
“VALUE added” for 4 years - writing
Percent of Ratings

48. Building the Evidentiary Base
• University of Kansas –
• “analysis of the data from the AACU VALUE rubrics
affirmed that a team approach to course design can yield
larger improvement in some forms of student writing and
thinking”
• “We also saw that the rubrics work best when there is close
alignment between the nature of the assignment and the
dimensions of intellectual skill described in the rubric”
• “Finally, at a practical level we are very encouraged that this
process is manageable and sustainable”

49. Building the Evidentiary Base
• University of Kansas –
• “Interestingly, the patterns that were visible in the VALUE
rubric scores were not mirrored in the CLA scores; students
in the team-designed and traditional courses performed no
differently on the CLA. Students‟ performance on the CLA,
moreover, was generally unrelated to the VALUE rubric
ratings of their coursework, as well as the instructors‟
grading of the same coursework. In contrast, the latter two
measures were highly correlated, suggesting that the
VALUE rubrics capture qualities of critical thinking and
writing that fit well with what faculty members VALUE in
their students‟ work.”

50. Roanoke College
Writing
Reliability = >.8

51. Using the Results
“…we excluded the scores for those instructors and ran
frequencies and descriptive statistics on the categories again.
We found the means for the rubric categories of Focus and
Thesis and Organization remained close to 2.5 with the scores
of 2 and 3 occurring most often (approx. 2.4) while most
scores for the categories of Evidence and Reasoning were 2s.
The category of Style and Mechanics was in the middle with a
mean of 2.3. Our adjusted results support what most
faculty believe about the writing of our first-year students,
they can learn to develop a thesis and organize a paper
more effectively than they can provide convincing
evidence and strong reasoning to support the thesis.”

52. Table 1 Information Literacy Results
Building the Evidentiary Base
University of North Carolina - Wilmington
Table 1. Information Literacy Results
Dimension % of Work Products % of Work Products
Scored 2 or higher Scored 3 or Higher
IL1 Determine Information Needed 87.2% 46.2%
IL2 Access Needed Information 89.6% 46.8%
IL3 Evaluate Information and Sources 88.5% 39.7%
IL4 Use Information Effectively 85.9% 43.6%
IL5 Access and Use Information Ethically 93.6% 59.0%
Inter-rater reliability = >.8

53. Building the Evidentiary Base
University of North Carolina - Wilmington
Table 2. Critical Thinking Results
Dimension % of Work Products % of Work Products
Scored 2 or higher Scored 3 or Higher
CT1 Explanation of Issues 68.3% 35.5%
CT2 Evidence Year1 65.0% 28.2%
CT2 Evidence Year 2*
Interpreting and Analysis 72.8% 38.6%
Questioning viewpoint 40.9% 13.6%
CT3 Influence of Context and Assumptions 48.8% 21.2%
CT4 Student’s Position 54.5% 24.0%
CT5 Conclusions and Related Outcomes 47.7% 17.0%

54. Building the Evidentiary Base
Long Island University – Brooklyn Campus
The results of this initial assessment are early
benchmarks important for all institutions that expect
integrative learning to emerge among students‟
upper level accomplishments. These results help us
see how students incorporate unmediated
experiential learning into deeper transference
and comprehension. In fact, students welcome
the opportunity to integrate life experience,
course work, and texts early in their
undergraduate experience, if invited to do so.

55. Implications: VSA and CLA
• VSA institutions can select from CAAP, CLA,
or ETS Proficiency Profile
• Important considerations for selection
– Acceptance by students, faculty, administrators
or other policy makers
– Trade-offs in cost, ease of administration
– Utility of the test for other purposes - supporting
campus activities and services or providing
guidance on improving learning
55

56. One Approach to
Capturing and Tracking
Learning
E-Portfolios

57. Portland State University
Freshman Inquiry (FRINQ)
UNST 100-level class
A year-long, theme-based course with a mentor section that corresponds with the class.
Transfer Transition
UNST 200- and 300- level classes
One-term course designed specifically for transfer students,
with a mentor section that corresponds with the class.
Sophomore Inquiry (SINQ)
UNST 200-level classes
One-term courses with a mentor section that corresponds with the class.
Sophomore Inquiry Sophomore Inquiry Sophomore Inquiry
Upper Division Cluster
Classes designated with a "U" offered by academic departments. Three courses in one cluster linked to one of the Sophomore
Inquiry courses.
Cluster Course One Cluster Course Three
Cluster Course Two
Senior Capstone
UNST 400-level class
A 6-credit, community-based learning class.

58. Queensborough College
• Remedial/Developmental Communications
Course
• First Year Writing Course
• Disciplinary Course
• Web-based portfolio/wiki
58

60. Literature,
Language &
Communication

64. We have had our why's, how's, and what's upside-
down,
focusing too much on what should be learned, than on
how, and
often forgetting the why altogether.
In a world of nearly infinite information, we must first
address why, facilitate how, and let the what generate
naturally from there.
Michael Wesch, “From Knowledgeable to Knowledge-able,” Academic
Commons, January 2009 (academiccommons.org)
64

65. Curricular and Pedagogical
Innovations – Led by Faculty – Are
Creating a 21st Century Vision and
Practice for Liberal Education
Directly Connected to the Needs and
Experiences of Today‟s Diverse Students,
our Diverse Democracy, and
Interdependent Global Community
65