Researching e-portfolios: The current state of play


Published on

The first in the Europortfolio project's series of open webinars, from February 7, 2014. Inter/National Coalition for Electronic Portfolio Research co-directors Darren Cambridge, Barbara Cambridge, and Kathleen Yancey present on the philosophy behind and design of the Coalition, how its results illustrate the principle of "scaling out," and the four propositions about assessment with e-portfolios and their non-negotiable core that Coalition members are currently exploring.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Implementation as avoiding problems
    Research as producing proof
  • Poll: What characterizes good evidence for the effectiveness of an application of learning technology?
  • Tranferrable to other domains – schools, workplace learning, etc.?
  • Researching e-portfolios: The current state of play

    1. 1. Researching e-portfolios: the current state of play Darren Cambridge, Barbara Cambridge, and Kathleen Blake Yancey Inter/National Coalition for Electronic Portfolio Research February 6, 2014
    2. 2. Overview Our research model: Philosophy and process (Darren) Some results: Scaling out (Barbara) Current questions: The four propositions (Kathi)
    3. 3. Implementation Research
    4. 4. The Importance of Having Problems In scholarship and research, having a "problem" is at the heart of the investigative process; it is the compound of the generative questions around which all creative and productive activity revolves. But in one’s teaching, a "problem" is something you don’t want to have, and if you have one, you probably want to fix it. … How might we make the problematization of teaching a matter of regular communal discourse? How might we think of teaching practice, and the evidence of student learning, as problems to be investigated, analyzed, represented, and debated? —Randy Bass
    5. 5. Three curricula Kathleen Yancey, Reflection in the Writing Classroom
    6. 6. Research into the Swamp “There is a high, hard ground where practitioners can make effective use of [traditional] researchbased theory and techniques, and there is a swampy lowland where situations are confusing ‘messes’ incapable of technical solution. The difficulty is that the problems of the high ground, however great their technical interest, are often relatively unimportant to clients or to the larger society, while in the swamp are the problems of greatest human concern.” Donald Schön
    7. 7. Transactional Research • Practitioners generate research questions • Goal is to influence practice • Methodologies chosen based on knowledge about learning, not exclusively current disciplinarily-accepted methodologies • Agency for answering the questions resides in multiple constituents – practitioner researchers – learners – peer practitioner researchers • Diversity provides robustness
    8. 8. Coalition structure
    9. 9. Coalition Exigency • Rapid growth in use of electronic portfolios in the United States (and beyond) • Wide diversity of models • Considerable potential to impact learning and engagement • Evidence uneven and fragmented • Implementation unconnected to research Inter/National Coalition for Electronic Portfolio Research
    10. 10. Coalition Structure • Inter/National Coalition for Electronic Portfolio Research established in 2003 • Led by Barbara Cambridge (AAHE/NCTE), Kathleen Yancey (Clemson/FSU), Darren Cambridge (EDUCAUSE/GMU/AIR) • Seven cohorts of about ten campuses that work together for three years Inter/National Coalition for Electronic Portfolio Research
    11. 11. Coalition Activities • • • • • • Individual questions and collaborative themes Two meetings a year Intervening online communication Interaction between cohorts Consultations with Coalition leadership Coordinated dissemination Inter/National Coalition for Electronic Portfolio Research
    12. 12. Intra-organizational Practices • • • • Diverse team Space for forming Narrow but open question Balance between intellectual and pragmatic purposes Inter/National Coalition for Electronic Portfolio Research
    13. 13. Diverse Team • Both people who have research in their job title and those who don’t • Reflective of the range of people involved in portfolio practice on the campus – Include administrators – Include students • Portland State: Administrators, students, faculty from multiple disciplines Inter/National Coalition for Electronic Portfolio Research
    14. 14. Space for Forming • Need sufficient time and space to develop – Shared expectations – Shared conceptual framework – Personal relationships within team Inter/National Coalition for Electronic Portfolio Research
    15. 15. Narrow but Open Question • Well-focused research question • Openness to the data taking you elsewhere Inter/National Coalition for Electronic Portfolio Research
    16. 16. Intellectual and Pragmatic Purposes • Clear sense of audiences and purposes of research • Practitioner research doesn’t have to be just evaluation • Balance between what you need to justify your work and what’s intellectually meaningful • Practice as inquiry Inter/National Coalition for Electronic Portfolio Research
    17. 17. Diversity and Balance • Who might you ask to join your team you’ve not previously considered? • What aspects of your project can you expand or emphasize to balance intellectual and pragmatic value? Inter/National Coalition for Electronic Portfolio Research
    18. 18. Inter-organizational Practices • Senior leadership sponsorship • Triangulation rather than replication • Collaborative exploration of methodologies • Regular conversations with neutral experts • Multiple genres of reporting out Inter/National Coalition for Electronic Portfolio Research
    19. 19. Senior Leadership Sponsorship • Three-year commitment of travel funding from institutional budget – Confirmation of commitment to portfolio practice • Regular updates and notes of thanks • Ideally, member of the team Inter/National Coalition for Electronic Portfolio Research
    20. 20. Triangulation • Triangulation rather than replication • Enough structure to focus and connect, but not restrict – No one strict definition of “research” – Shared themes but not a mandated research question • Cohorts One and Two: Catalog and taxonomy of reflective artifacts • Critical friends Inter/National Coalition for Electronic Portfolio Research
    21. 21. Collaborative Exploration of Methodology • Guided exploration of research methodologies and methods • Both a way to plan the project and a way to develop shared understanding of research • Breaking out of received notions of research through conversations – Across disciplines / practice areas – Across campuses / organizations Inter/National Coalition for Electronic Portfolio Research
    22. 22. Conversations with Experts • Quarterly conference calls with a Coalition leader • Periodic occasions for reviewing and asking questions • The questioning is probably more important than the advice Inter/National Coalition for Electronic Portfolio Research
    23. 23. Multiple Reporting Genres • Variety of forms of reporting – One-pagers – Blue Skies questions – Thick descriptions of artifacts – Presentations of evidence – Chats • Helps to stimulate creativity and accommodate multiple styles Inter/National Coalition for Electronic Portfolio Research
    24. 24. Scaling out A local site implements a practice, does research on its effects, and shares the findings with another site that adopts or adapts the inquiry question. The process continues.
    25. 25. Over 60 campus research teams are linked through the Coalition in a network in which some teams have scaled out from others. Scaling out can be around category of inquiry question, the inquiry question itself, methodology, purpose, audience, or emergent findings. Today we’ll look at category of inquiry question as an illustration of the value of scaled out research having been done or being done in the Coalition.
    26. 26. 1. Institutional mission Goshen College, a small college in mid-America, strives to provide an education for the whole person to create leaders for the church and for the world. To that end, it chooses to focus on experiential learning and connections between academic affairs and student life. Specifically, each student spends a semesterlong study abroad with classes and service.
    27. 27. Content analysis of reflective statements in eportfolios that address each of the college’s student learning outcomes Coding of fifty reflective statements about ways students had changed and about ways students had been strengthened in relation to learning outcomes: – – – – Knowledge Skills Responsibilities Integration Six themes emerged, the strongest being - Affecting identity (values, beliefs, and assumptions) - Increased self-awareness
    28. 28. Finding: Eportfolios enable students to turn experience into evidence of learning through reflection and through multiple representations of experience. Implication for Goshen College: “We have a way to assess if we are acting out our mission as a college through the structure of learning for students.”
    29. 29. Institutional mission Loyola University, a medium-sized private university, in the middle-part of the US. It emphasizes the Jesuit value of social justice. It plays that out through experiential learning. The university strives to engage students in their own learning as they work to embody the university’s mission.
    30. 30. Inquiry into the Effectiveness of Introductions to Engaged Learning Assignments In Fall 2013 a group of faculty gave a reflection assignment that generated an artefact for the students’ portfolio. Some faculty within the group used a common prompt; others did not. Using activity theory, constructivism, and flow theory, another group of faculty and administrators (some in the group who gave the assignment and others not) catalogued the artefacts, used a rubric to judge the artefacts (norming), will review a random sample of the artefacts, and will continue the process during Spring 2014. Comparison and contrast will be made between the quality of the artefacts generated from the common prompt versus other prompts.
    31. 31. Relation to Mission If the common prompt helped yield more effective reflective artefacts, ones that according to the rubric demonstrate engagement in experiential learning related to social justice, the university will consider use of that prompt across more classes in the university.
    32. 32. 2. Professional identity Clemson University-Cohort II University of Michigan-Cohort VI Teams from two large universities in different parts of the United States, one private and one public, chose to investigate professional identity, one of majors in psychology and one of minors in writing.
    33. 33. Clemson: How do eportfolios contribute to development of identity as a psychologist? Psychological Assessment Survey, 169-item survey based on American Psychological Association’s learning goals and outcomes. Five psychological factors: Five general factors; Knowledge base Information and technical literacy Research methods Communication skills Critical thinking skills Sociocultural/international awareness Application Personal development Values in psychology Career planning/development
    34. 34. Methodology and Finding PAS at beginning of the semester; eportfolio-based PAS at end of the semester where each item tapped how their eportfolio affected the learning outcomes (n=95) Finding: Both set of factors improved significantly over the semester between 0.4 and 0.8 on a 6-point Likert scale. Psychological factors improved at a greater rate than general competencies. Instructor, self-rated, and peer-rated PAS scores were similar, confirming reliability of the instrument revealed by the initial confirmatory factor analysis.
    35. 35. Michigan: How do affordances in eportfolios of multiple connections, spaces for reflection, linkages, and navigations contribute to the ways students develop and come to think about themselves as writers? Four cohorts of writing minors (N=66) and comparison group (N=113) Mixed methods: Writing minors: Statistical analyses on entry • Instructional and demographic data and exit surveys • Entry surveys in semester students entered the program Coded for emergent themes • Exit surveys from semester they graduated related to writerly self-interest • Entry interviews during second semester of study and frameworks for thinking • Exit interviews as they graduate about writing • Application letters to Minor in Writing • Archived gateway and capstone eportfolios Comparison group: Rhetorical move analysis and • Writing samples from each semester in the study corpus linguistics • Directed self-placement essays • Writing samples from first-year writing classes
    36. 36. Five major findings • • • • • Minors express stronger sense of themselves as writers than in the comparison group. Minors demonstrate greater genre awareness than students in the comparison group. Minors demonstrate a more developed sense of rhetorical situation than students in the comparison group. Minors demonstrate a more developed practice of reflection on writing than students in the comparison group. Eportfolios play an important role in helping students in the minor group understand their development as writers.
    37. 37. • According to the research of Clemson University and the University of Michigan, eportfolios contribute to the professional identity development of students in the respective professions of psychology and writing.
    38. 38. 3. Use of eportfolios within a system Lamar University and DePaul University have investigated the use of eportfolios across systems. Lamar, in an on-line graduate program, asked questions about effects of eportfolio practice in the university on teachers use of eportfolios in earlier academic levels. DePaul is comparing and contrasting an eportfolio element across disciplines and learning situations.
    39. 39. Lamar research question: Has participation as a master’s candidate using the eportfolio process contributed to transference of eportfolio practices with PreK-12 students? Method: Web-based survey to 271 program graduates in Fall 2011 and to 202 in Spring 2013 with 49% response rate. Based on Crosswell method of explanatory sequential design using collection of qualitative date to explain quantitative data results. Findings about conditions for implementation in PreK-12: Support factors: administrative support, grading, parental support, internet safety, eportfolio retention time Challenges: assessment, eportfolios as official records, eportfolio reflection time Next inquiry: Case study of entire school district to investigate eportfolio implementation, including contributions of teachers who have and have not studied through eportfolios themselves
    40. 40. DePaul research: The affordances of eportfolios enable more kinds and opportunities for reflection for all students. Reflections are different across schools (professional, adult learners, first-year students). DePaul is studying how and why the differences. • Current work includes coding eportfolios. Much discussion about methodology, but they have developed what they call a “mash-up methodology” on which they have shared agreement. They affirm the great benefit of having researchers from different backgrounds and settings working together. The acceptance of research results and commitment to application to practice are greater with this approach. • Scaling out across an institution will be more possible after this initial work that signals the possibility of application of eportfolios in multiple educational contexts.
    41. 41. Other sites for further specific information about Cohort teams’ research Final reports of all campus teams are posted on the Inter/National Coalition for Electronic Portfolio Research website (2008-2013) Electronic Portfolios 2.0: Emergent Research on Implementation and Impact (Cambridge, Cambridge, and Yancey. Stylus Publishing, 2009) Eportfolios for Lifelong Learning and Assessment (Cambridge, D., Jossey-Bass, 2010)
    42. 42. Scaling out can lead to core practices. • Scaling out is important, but so is the evolution of core assumptions and practices as more and more local entities embody eportfolio practice and research. The next section of our webinar will focus on core propositions regarding eportfolio learning.
    43. 43. Cohorts 6 and 7 The Non-Negotiable Core of ePortfolio and the Four Propositions
    44. 44. COHORT SIX Curtin University of Technology (Australia) Goshen College Indiana University Purdue University Indianapolis Lamar University Northeastern University Portland State University University of Georgia University of Michigan University of Mississippi Virginia Military Institute Westminster College   .     COHORT SEVEN Deakin University (AUS) Clemson DePaul Lamar University Loyola of Chicago Santa Clara University University of Cincinnati (two campuses)
    45. 45. The Non-Negotiable Core of Electronic Portfolio --Confusion among definitions --Confusion among practices --Interest in identifying what is distinctive about ePortfolios
    46. 46. Where is reflection? When you complete a "Critical Task" assignment in a course you will upload this assignment to your Assessment Portfolio (AP). Chalk and Wire will notify your instructor that the assignment is ready to be graded. The instructor will grade your Critical Task assignment using a rubric. All Critical Task assignments must be submitted and graded through Chalk and Wire. At the end of your program you will have a complete portfolio of Critical Task assignments that document your professional growth.
    47. 47. FOUR PROPOSITIONS 1. Interaction of Pieces of Evidence 2. Relationship between Evidence and Reflection 3. Material Practices 4. Meaningful Comparison without Standardization
    48. 48. PROPOSITION ONE Interaction of Pieces of Evidence For meaningful assessment, interaction of pieces of evidence within an eportfolio is more important than single pieces of evidence
    49. 49. PROPOSITION TWO Relationship between Evidence and Reflection Reflection on pieces of evidence within an eportfolio and on the eportfolio as a whole provides information for assessment that isn’t available by other means
    50. 50. IN TEAMS
    51. 51. PROPOSITION THREE Material Practices The material practice of eportfolio composition generates distinctive knowledge about learning
    52. 52. New students have limited experiences with various media; and their thinking is patterned and routinized in a way that’s not keyed to insight: they may need new experiences with all kinds of texts. Also complicated by proficiency with technologies and faculty facility with them. Reading portfolios as a practice. Confusion of port reading linked to media: print as linear harder to process; reflection providing role of narrator; what kinds of tools/commentary/ navigations do we need for the design and the readings of portfolios?
    53. 53. PROPOSITION FOUR Meaningful Comparison without Standardization Eportfolios enable meaningful comparison of student learning across institutions (and other contexts) without standardization
    54. 54. What is validity? How do different perspectives impact the way we assess an ePortfolio? What is the process of assessment, and how does it impact judgments? CONSIDER: The decision-making process should be an interpretive conversation guided by norms. What criteria and evidence can be used in making a decision? When, how, and by whom? And how do you document that decision-making process so others understand how you got there?
    55. 55. Some Tentative Results 1. Students are often asked to collect artifacts, but not to consider what they learn from seeing how they interact with each other. Considering the interaction of artifacts=an opportunity for learning that is often missed.
    56. 56. 2. Cohort members agree that reflection makes a distinctive contribution to learning, but we— faculty, staff, and students—are frequently vague about what we want that distinctive contribution to be. Strong reflection seems to tap at least two domains of learning (e.g., sociolinguistic; psychological; epistemic).
    57. 57. 3. There is some resistance to the idea that we can have comparability without standardization. It’s not that cohort members want standardization, but they don’t see how they can manage assessment without it. It requires a paradigmatic shift in thinking—much like outcomes have.
    58. 58. Learn More • Europortfolio: • Inter/National Coalition for Electronic Portfolio Research: • Eportfolios 2.0: Emergent Findings about Implementation and Impact (Stylus, 2009) • Darren:, @dcambrid • Barbara: • Kathi: