The first in the Europortfolio project's series of open webinars, from February 7, 2014. Inter/National Coalition for Electronic Portfolio Research co-directors Darren Cambridge, Barbara Cambridge, and Kathleen Yancey present on the philosophy behind and design of the Coalition, how its results illustrate the principle of "scaling out," and the four propositions about assessment with e-portfolios and their non-negotiable core that Coalition members are currently exploring.

1. Researching e-portfolios:
the current state of play
Darren Cambridge, Barbara Cambridge,
and Kathleen Blake Yancey
Inter/National Coalition for
Electronic Portfolio Research
February 6, 2014

2. Overview
Our research model: Philosophy and process
(Darren)
Some results: Scaling out
(Barbara)
Current questions: The four propositions
(Kathi)

3. Implementation
Research

4. The Importance of Having Problems
In scholarship and research, having a "problem" is at
the heart of the investigative process; it is the
compound of the generative questions around which
all creative and productive activity revolves. But in
one’s teaching, a "problem" is something you don’t
want to have, and if you have one, you probably
want to fix it. … How might we make the
problematization of teaching a matter of regular
communal discourse? How might we think of
teaching practice, and the evidence of student
learning, as problems to be investigated, analyzed,
represented, and debated?
—Randy Bass

6. Three curricula
Kathleen Yancey, Reflection in the Writing Classroom

7. Research into the Swamp
“There is a high, hard ground where practitioners
can make effective use of [traditional] researchbased theory and techniques, and there is a
swampy lowland where situations are confusing
‘messes’ incapable of technical solution. The
difficulty is that the problems of the high ground,
however great their technical interest, are often
relatively unimportant to clients or to the larger
society, while in the swamp are the problems of
greatest human concern.”
Donald Schön

8. Transactional Research
• Practitioners generate research questions
• Goal is to influence practice
• Methodologies chosen based on knowledge
about learning, not exclusively current
disciplinarily-accepted methodologies
• Agency for answering the questions resides in
multiple constituents
– practitioner researchers
– learners
– peer practitioner researchers
• Diversity provides robustness

9. Coalition structure

10. Coalition Exigency
• Rapid growth in use of electronic portfolios in
the United States (and beyond)
• Wide diversity of models
• Considerable potential to impact learning and
engagement
• Evidence uneven and fragmented
• Implementation unconnected to research
Inter/National Coalition for Electronic
Portfolio Research

11. Coalition Structure
• Inter/National Coalition for Electronic Portfolio
Research established in 2003
• Led by Barbara Cambridge (AAHE/NCTE), Kathleen
Yancey (Clemson/FSU), Darren Cambridge
(EDUCAUSE/GMU/AIR)
• Seven cohorts of about ten campuses that work
together for three years
Inter/National Coalition for Electronic
Portfolio Research

12. Coalition Activities
•
•
•
•
•
•
Individual questions and collaborative themes
Two meetings a year
Intervening online communication
Interaction between cohorts
Consultations with Coalition leadership
Coordinated dissemination
Inter/National Coalition for Electronic
Portfolio Research

13. Intra-organizational Practices
•
•
•
•
Diverse team
Space for forming
Narrow but open question
Balance between intellectual and pragmatic
purposes
Inter/National Coalition for Electronic
Portfolio Research

14. Diverse Team
• Both people who have research in their job
title and those who don’t
• Reflective of the range of people involved in
portfolio practice on the campus
– Include administrators
– Include students
• Portland State: Administrators, students,
faculty from multiple disciplines
Inter/National Coalition for Electronic
Portfolio Research

15. Space for Forming
• Need sufficient time and space to develop
– Shared expectations
– Shared conceptual framework
– Personal relationships within team
Inter/National Coalition for Electronic
Portfolio Research

16. Narrow but Open Question
• Well-focused research question
• Openness to the data taking you elsewhere
Inter/National Coalition for Electronic
Portfolio Research

17. Intellectual and
Pragmatic Purposes
• Clear sense of audiences and purposes of research
• Practitioner research doesn’t have to be just
evaluation
• Balance between what you need to justify your work
and what’s intellectually meaningful
• Practice as inquiry
Inter/National Coalition for Electronic
Portfolio Research

18. Diversity and Balance
• Who might you ask to join your team you’ve
not previously considered?
• What aspects of your project can you expand
or emphasize to balance intellectual and
pragmatic value?
Inter/National Coalition for Electronic
Portfolio Research

19. Inter-organizational Practices
• Senior leadership sponsorship
• Triangulation rather than replication
• Collaborative exploration of
methodologies
• Regular conversations with neutral
experts
• Multiple genres of reporting out
Inter/National Coalition for Electronic
Portfolio Research

20. Senior Leadership Sponsorship
• Three-year commitment of travel funding
from institutional budget
– Confirmation of commitment to portfolio practice
• Regular updates and notes of thanks
• Ideally, member of the team
Inter/National Coalition for Electronic
Portfolio Research

21. Triangulation
• Triangulation rather than replication
• Enough structure to focus and connect, but not
restrict
– No one strict definition of “research”
– Shared themes but not a mandated research question
• Cohorts One and Two: Catalog and taxonomy of reflective artifacts
• Critical friends
Inter/National Coalition for Electronic
Portfolio Research

22. Collaborative
Exploration of Methodology
• Guided exploration of research methodologies and
methods
• Both a way to plan the project and a way to develop
shared understanding of research
• Breaking out of received notions of research through
conversations
– Across disciplines / practice areas
– Across campuses / organizations
Inter/National Coalition for Electronic
Portfolio Research

23. Conversations with Experts
• Quarterly conference calls with a Coalition
leader
• Periodic occasions for reviewing and asking
questions
• The questioning is probably more important
than the advice
Inter/National Coalition for Electronic
Portfolio Research

24. Multiple Reporting Genres
• Variety of forms of reporting
– One-pagers
– Blue Skies questions
– Thick descriptions of artifacts
– Presentations of evidence
– Chats
• Helps to stimulate creativity and
accommodate multiple styles
Inter/National Coalition for Electronic
Portfolio Research

25. Scaling out
A local site implements a practice, does
research on its effects, and shares the findings
with another site that adopts or adapts the
inquiry question.
The process continues.

26. Over 60 campus research teams are linked through
the Coalition in a network in which some teams
have scaled out from others.
Scaling out can be around category of inquiry
question, the inquiry question itself,
methodology, purpose, audience, or emergent
findings.
Today we’ll look at category of inquiry question as
an illustration of the value of scaled out research
having been done or being done in the Coalition.

27. 1. Institutional mission
Goshen College, a small college in mid-America,
strives to provide an education for the whole
person to create leaders for the church and
for the world. To that end, it chooses to focus
on experiential learning and connections
between academic affairs and student life.
Specifically, each student spends a semesterlong study abroad with classes and service.

28. Content analysis of reflective statements in
eportfolios that address each of the college’s student
learning outcomes
Coding of fifty reflective
statements about ways
students had changed and
about ways students had
been strengthened in
relation to learning
outcomes:
–
–
–
–
Knowledge
Skills
Responsibilities
Integration
Six themes emerged, the
strongest being
- Affecting identity (values,
beliefs, and assumptions)
- Increased self-awareness

29. Finding: Eportfolios enable students to turn experience
into evidence of learning through reflection and
through multiple representations of experience.
Implication for Goshen College:
“We have a way to assess if we are acting out our
mission as a college through the structure of learning
for students.”

30. Institutional mission
Loyola University, a medium-sized private
university, in the middle-part of the US. It
emphasizes the Jesuit value of social justice. It
plays that out through experiential learning.
The university strives to engage students in
their own learning as they work to embody
the university’s mission.

31. Inquiry into the Effectiveness of Introductions to Engaged
Learning Assignments
In Fall 2013 a group of faculty gave a reflection assignment that
generated an artefact for the students’ portfolio. Some
faculty within the group used a common prompt; others did
not.
Using activity theory, constructivism, and flow theory, another
group of faculty and administrators (some in the group who
gave the assignment and others not) catalogued the artefacts,
used a rubric to judge the artefacts (norming), will review a
random sample of the artefacts, and will continue the process
during Spring 2014. Comparison and contrast will be made
between the quality of the artefacts generated from the
common prompt versus other prompts.

32. Relation to Mission
If the common prompt helped yield more
effective reflective artefacts, ones that
according to the rubric demonstrate
engagement in experiential learning related to
social justice, the university will consider use
of that prompt across more classes in the
university.

33. 2. Professional identity
Clemson University-Cohort II
University of Michigan-Cohort VI
Teams from two large universities in different parts of
the United States, one private and one public, chose
to investigate professional identity, one of majors in
psychology and one of minors in writing.

34. Clemson: How do eportfolios contribute to
development of identity as a psychologist?
Psychological Assessment Survey, 169-item
survey based on American Psychological
Association’s learning goals and outcomes.
Five psychological factors:
Five general factors;
Knowledge base
Information and technical literacy
Research methods
Communication skills
Critical thinking skills
Sociocultural/international awareness
Application
Personal development
Values in psychology
Career planning/development

35. Methodology and Finding
PAS at beginning of the semester; eportfolio-based PAS
at end of the semester where each item tapped how
their eportfolio affected the learning outcomes
(n=95)
Finding: Both set of factors improved significantly over
the semester between 0.4 and 0.8 on a 6-point Likert
scale. Psychological factors improved at a greater
rate than general competencies.
Instructor, self-rated, and peer-rated PAS scores were
similar, confirming reliability of the instrument
revealed by the initial confirmatory factor analysis.

36. Michigan: How do affordances in eportfolios of multiple connections,
spaces for reflection, linkages, and navigations contribute to the ways
students develop and come to think about themselves as writers?
Four cohorts of writing minors (N=66) and comparison group (N=113)
Mixed methods:
Writing minors:
Statistical analyses on entry
• Instructional and demographic data
and exit surveys
• Entry surveys in semester students entered the program Coded for emergent themes
• Exit surveys from semester they graduated
related to writerly self-interest
• Entry interviews during second semester of study
and frameworks for thinking
• Exit interviews as they graduate
about writing
• Application letters to Minor in Writing
• Archived gateway and capstone eportfolios
Comparison group:
Rhetorical move analysis and
• Writing samples from each semester in the study
corpus linguistics
• Directed self-placement essays
• Writing samples from first-year writing classes

37. Five major findings
•
•
•
•
•
Minors express stronger sense of themselves as writers than in the
comparison group.
Minors demonstrate greater genre awareness than students in the
comparison group.
Minors demonstrate a more developed sense of rhetorical situation than
students in the comparison group.
Minors demonstrate a more developed practice of reflection on writing
than students in the comparison group.
Eportfolios play an important role in helping students in the minor group
understand their development as writers.

38. • According to the research of Clemson
University and the University of Michigan,
eportfolios contribute to the professional
identity development of students in the
respective professions of psychology and
writing.

39. 3. Use of eportfolios within a system
Lamar University and DePaul University have
investigated the use of eportfolios across
systems. Lamar, in an on-line graduate
program, asked questions about effects of
eportfolio practice in the university on
teachers use of eportfolios in earlier academic
levels. DePaul is comparing and contrasting an
eportfolio element across disciplines and
learning situations.

40. Lamar research question: Has participation as a master’s
candidate using the eportfolio process contributed to
transference of eportfolio practices with PreK-12 students?
Method: Web-based survey to 271 program graduates in Fall 2011 and to
202 in Spring 2013 with 49% response rate. Based on Crosswell method of
explanatory sequential design using collection of qualitative date to
explain quantitative data results.
Findings about conditions for implementation in PreK-12:
Support factors: administrative support, grading, parental support, internet
safety, eportfolio retention time
Challenges: assessment, eportfolios as official records, eportfolio reflection
time
Next inquiry:
Case study of entire school district to investigate eportfolio implementation,
including contributions of teachers who have and have not studied
through eportfolios themselves

41. DePaul research: The affordances of eportfolios enable more
kinds and opportunities for reflection for all students.
Reflections are different across schools (professional, adult
learners, first-year students). DePaul is studying how and why
the differences.
• Current work includes coding eportfolios. Much discussion about
methodology, but they have developed what they call a “mash-up
methodology” on which they have shared agreement. They affirm
the great benefit of having researchers from different backgrounds
and settings working together. The acceptance of research results
and commitment to application to practice are greater with this
approach.
• Scaling out across an institution will be more possible after this
initial work that signals the possibility of application of eportfolios
in multiple educational contexts.

42. Other sites for further specific information
about Cohort teams’ research
Final reports of all campus teams are posted on
the Inter/National Coalition for Electronic
Portfolio Research website (2008-2013)
Electronic Portfolios 2.0: Emergent Research on
Implementation and Impact (Cambridge,
Cambridge, and Yancey. Stylus Publishing,
2009)
Eportfolios for Lifelong Learning and Assessment
(Cambridge, D., Jossey-Bass, 2010)

43. Scaling out can lead to core practices.
• Scaling out is important, but so is the
evolution of core assumptions and practices
as more and more local entities embody
eportfolio practice and research. The next
section of our webinar will focus on core
propositions regarding eportfolio learning.

44. Cohorts 6 and 7
The Non-Negotiable Core of ePortfolio
and the Four Propositions

45. COHORT SIX
Curtin University of Technology (Australia)
Goshen College
Indiana University Purdue
University Indianapolis
Lamar University
Northeastern University
Portland State University
University of Georgia
University of Michigan
University of Mississippi
Virginia Military Institute
Westminster College
.
COHORT SEVEN
Deakin University (AUS)
Clemson
DePaul
Lamar University
Loyola of Chicago
Santa Clara University
University of Cincinnati
(two campuses)

48. The Non-Negotiable Core of Electronic
Portfolio
--Confusion among definitions
--Confusion among practices
--Interest in identifying what is
distinctive about ePortfolios

49. Where is reflection?
When you complete a "Critical
Task" assignment in a course
you will upload this
assignment to your
Assessment Portfolio (AP).
Chalk and Wire will notify your
instructor that the assignment
is ready to be graded. The
instructor will grade your
Critical Task assignment using
a rubric. All Critical Task
assignments must be
submitted and graded
through Chalk and Wire. At
the end of your program you
will have a complete portfolio
of Critical Task assignments
that document your
professional growth.

50. FOUR PROPOSITIONS
1. Interaction of Pieces of Evidence
2. Relationship between Evidence and Reflection
3. Material Practices
4. Meaningful Comparison without Standardization

51. PROPOSITION ONE
Interaction of Pieces of Evidence
For meaningful assessment, interaction
of pieces of evidence within an
eportfolio is more important than
single pieces of evidence

53. PROPOSITION TWO
Relationship between Evidence and Reflection
Reflection on pieces of evidence within
an eportfolio and on the eportfolio as a
whole provides information for
assessment that isn’t available by
other means

54. IN TEAMS

55. PROPOSITION THREE
Material Practices
The material practice of eportfolio
composition generates distinctive
knowledge about learning

56. New students have
limited experiences
with various media;
and their thinking is
patterned and
routinized in a way
that’s not keyed to
insight: they may
need new
experiences with all
kinds of texts. Also
complicated by
proficiency with
technologies and
faculty facility with
them.
Reading portfolios as a
practice. Confusion of
port reading linked to
media: print as linear
harder to process;
reflection providing role
of narrator; what kinds
of tools/commentary/
navigations do we need
for the design and the
readings of portfolios?

57. PROPOSITION FOUR
Meaningful Comparison without
Standardization
Eportfolios enable meaningful
comparison of student learning across
institutions (and other contexts)
without standardization

58. What is validity?
How do different
perspectives impact
the way we assess
an ePortfolio?
What is the process of assessment, and how does it impact
judgments?
CONSIDER: The decision-making process should be an
interpretive conversation guided by norms.
What criteria and evidence can be used in making a decision?
When, how, and by whom?
And how do you document that decision-making process so
others understand how you got there?

59. Some Tentative Results
1. Students are
often asked to
collect artifacts, but not to consider
what they learn from seeing how they
interact with each other. Considering
the interaction of artifacts=an
opportunity for learning that is often
missed.

60. 2. Cohort members agree that
reflection makes a distinctive
contribution to learning, but we—
faculty, staff, and students—are
frequently vague about what we want
that distinctive contribution to be.
Strong reflection seems to tap at least
two domains of learning (e.g.,
sociolinguistic; psychological;
epistemic).

61. 3. There is some resistance to the idea
that we can have comparability
without standardization. It’s not that
cohort members want
standardization, but they don’t see
how they can manage assessment
without it. It requires a paradigmatic
shift in thinking—much like outcomes
have.

62. Learn More
• Europortfolio: europortfolio.org
• Inter/National Coalition for Electronic
Portfolio Research: ncepr.org
• Eportfolios 2.0: Emergent Findings about
Implementation and Impact (Stylus, 2009)
• Darren: dcambrid@gmail.com, @dcambrid
• Barbara: bcambridge@ncte.org
• Kathi: kyancey@fsu.edu