Download to read offline
![Research into the Swamp
“There is a high, hard ground where practitioners
can make effective use of [traditional] researchbased theory and techniques, and there is a
swampy lowland where situations are confusing
‘messes’ incapable of technical solution. The
difficulty is that the problems of the high ground,
however great their technical interest, are often
relatively unimportant to clients or to the larger
society, while in the swamp are the problems of
greatest human concern.”
Donald Schön](https://image.slidesharecdn.com/epnetcoalitionresearchwebinar-140207102505-phpapp02/85/Researching-e-portfolios-The-current-state-of-play-7-320.jpg)
More Related Content
Similar to Researching e-portfolios: The current state of play
Researching e-portfolios: The current state of play
- 1. Researching e-portfolios: the currentstate of play Darren Cambridge, Barbara Cambridge, and Kathleen Blake Yancey Inter/National Coalition for Electronic Portfolio Research February 6, 2014
- 2. Overview Our research model:Philosophy and process (Darren) Some results: Scaling out (Barbara) Current questions: The four propositions (Kathi)
- 3.
- 4. The Importance ofHaving Problems In scholarship and research, having a "problem" is at the heart of the investigative process; it is the compound of the generative questions around which all creative and productive activity revolves. But in one’s teaching, a "problem" is something you don’t want to have, and if you have one, you probably want to fix it. … How might we make the problematization of teaching a matter of regular communal discourse? How might we think of teaching practice, and the evidence of student learning, as problems to be investigated, analyzed, represented, and debated? —Randy Bass
- 6. Three curricula Kathleen Yancey,Reflection in the Writing Classroom
- 7. Research into theSwamp “There is a high, hard ground where practitioners can make effective use of [traditional] researchbased theory and techniques, and there is a swampy lowland where situations are confusing ‘messes’ incapable of technical solution. The difficulty is that the problems of the high ground, however great their technical interest, are often relatively unimportant to clients or to the larger society, while in the swamp are the problems of greatest human concern.” Donald Schön
- 8. Transactional Research • Practitionersgenerate research questions • Goal is to influence practice • Methodologies chosen based on knowledge about learning, not exclusively current disciplinarily-accepted methodologies • Agency for answering the questions resides in multiple constituents – practitioner researchers – learners – peer practitioner researchers • Diversity provides robustness
- 9.
- 10. Coalition Exigency • Rapidgrowth in use of electronic portfolios in the United States (and beyond) • Wide diversity of models • Considerable potential to impact learning and engagement • Evidence uneven and fragmented • Implementation unconnected to research Inter/National Coalition for Electronic Portfolio Research
- 11. Coalition Structure • Inter/NationalCoalition for Electronic Portfolio Research established in 2003 • Led by Barbara Cambridge (AAHE/NCTE), Kathleen Yancey (Clemson/FSU), Darren Cambridge (EDUCAUSE/GMU/AIR) • Seven cohorts of about ten campuses that work together for three years Inter/National Coalition for Electronic Portfolio Research
- 12. Coalition Activities • • • • • • Individual questionsand collaborative themes Two meetings a year Intervening online communication Interaction between cohorts Consultations with Coalition leadership Coordinated dissemination Inter/National Coalition for Electronic Portfolio Research
- 13. Intra-organizational Practices • • • • Diverse team Spacefor forming Narrow but open question Balance between intellectual and pragmatic purposes Inter/National Coalition for Electronic Portfolio Research
- 14. Diverse Team • Bothpeople who have research in their job title and those who don’t • Reflective of the range of people involved in portfolio practice on the campus – Include administrators – Include students • Portland State: Administrators, students, faculty from multiple disciplines Inter/National Coalition for Electronic Portfolio Research
- 15. Space for Forming •Need sufficient time and space to develop – Shared expectations – Shared conceptual framework – Personal relationships within team Inter/National Coalition for Electronic Portfolio Research
- 16. Narrow but OpenQuestion • Well-focused research question • Openness to the data taking you elsewhere Inter/National Coalition for Electronic Portfolio Research
- 17. Intellectual and Pragmatic Purposes •Clear sense of audiences and purposes of research • Practitioner research doesn’t have to be just evaluation • Balance between what you need to justify your work and what’s intellectually meaningful • Practice as inquiry Inter/National Coalition for Electronic Portfolio Research
- 18. Diversity and Balance •Who might you ask to join your team you’ve not previously considered? • What aspects of your project can you expand or emphasize to balance intellectual and pragmatic value? Inter/National Coalition for Electronic Portfolio Research
- 19. Inter-organizational Practices • Seniorleadership sponsorship • Triangulation rather than replication • Collaborative exploration of methodologies • Regular conversations with neutral experts • Multiple genres of reporting out Inter/National Coalition for Electronic Portfolio Research
- 20. Senior Leadership Sponsorship •Three-year commitment of travel funding from institutional budget – Confirmation of commitment to portfolio practice • Regular updates and notes of thanks • Ideally, member of the team Inter/National Coalition for Electronic Portfolio Research
- 21. Triangulation • Triangulation ratherthan replication • Enough structure to focus and connect, but not restrict – No one strict definition of “research” – Shared themes but not a mandated research question • Cohorts One and Two: Catalog and taxonomy of reflective artifacts • Critical friends Inter/National Coalition for Electronic Portfolio Research
- 22. Collaborative Exploration of Methodology •Guided exploration of research methodologies and methods • Both a way to plan the project and a way to develop shared understanding of research • Breaking out of received notions of research through conversations – Across disciplines / practice areas – Across campuses / organizations Inter/National Coalition for Electronic Portfolio Research
- 23. Conversations with Experts •Quarterly conference calls with a Coalition leader • Periodic occasions for reviewing and asking questions • The questioning is probably more important than the advice Inter/National Coalition for Electronic Portfolio Research
- 24. Multiple Reporting Genres •Variety of forms of reporting – One-pagers – Blue Skies questions – Thick descriptions of artifacts – Presentations of evidence – Chats • Helps to stimulate creativity and accommodate multiple styles Inter/National Coalition for Electronic Portfolio Research
- 25. Scaling out A localsite implements a practice, does research on its effects, and shares the findings with another site that adopts or adapts the inquiry question. The process continues.
- 26. Over 60 campusresearch teams are linked through the Coalition in a network in which some teams have scaled out from others. Scaling out can be around category of inquiry question, the inquiry question itself, methodology, purpose, audience, or emergent findings. Today we’ll look at category of inquiry question as an illustration of the value of scaled out research having been done or being done in the Coalition.
- 27. 1. Institutional mission GoshenCollege, a small college in mid-America, strives to provide an education for the whole person to create leaders for the church and for the world. To that end, it chooses to focus on experiential learning and connections between academic affairs and student life. Specifically, each student spends a semesterlong study abroad with classes and service.
- 28. Content analysis ofreflective statements in eportfolios that address each of the college’s student learning outcomes Coding of fifty reflective statements about ways students had changed and about ways students had been strengthened in relation to learning outcomes: – – – – Knowledge Skills Responsibilities Integration Six themes emerged, the strongest being - Affecting identity (values, beliefs, and assumptions) - Increased self-awareness
- 29. Finding: Eportfolios enablestudents to turn experience into evidence of learning through reflection and through multiple representations of experience. Implication for Goshen College: “We have a way to assess if we are acting out our mission as a college through the structure of learning for students.”
- 30. Institutional mission Loyola University,a medium-sized private university, in the middle-part of the US. It emphasizes the Jesuit value of social justice. It plays that out through experiential learning. The university strives to engage students in their own learning as they work to embody the university’s mission.
- 31. Inquiry into theEffectiveness of Introductions to Engaged Learning Assignments In Fall 2013 a group of faculty gave a reflection assignment that generated an artefact for the students’ portfolio. Some faculty within the group used a common prompt; others did not. Using activity theory, constructivism, and flow theory, another group of faculty and administrators (some in the group who gave the assignment and others not) catalogued the artefacts, used a rubric to judge the artefacts (norming), will review a random sample of the artefacts, and will continue the process during Spring 2014. Comparison and contrast will be made between the quality of the artefacts generated from the common prompt versus other prompts.
- 32. Relation to Mission Ifthe common prompt helped yield more effective reflective artefacts, ones that according to the rubric demonstrate engagement in experiential learning related to social justice, the university will consider use of that prompt across more classes in the university.
- 33. 2. Professional identity ClemsonUniversity-Cohort II University of Michigan-Cohort VI Teams from two large universities in different parts of the United States, one private and one public, chose to investigate professional identity, one of majors in psychology and one of minors in writing.
- 34. Clemson: How doeportfolios contribute to development of identity as a psychologist? Psychological Assessment Survey, 169-item survey based on American Psychological Association’s learning goals and outcomes. Five psychological factors: Five general factors; Knowledge base Information and technical literacy Research methods Communication skills Critical thinking skills Sociocultural/international awareness Application Personal development Values in psychology Career planning/development
- 35. Methodology and Finding PASat beginning of the semester; eportfolio-based PAS at end of the semester where each item tapped how their eportfolio affected the learning outcomes (n=95) Finding: Both set of factors improved significantly over the semester between 0.4 and 0.8 on a 6-point Likert scale. Psychological factors improved at a greater rate than general competencies. Instructor, self-rated, and peer-rated PAS scores were similar, confirming reliability of the instrument revealed by the initial confirmatory factor analysis.
- 36. Michigan: How doaffordances in eportfolios of multiple connections, spaces for reflection, linkages, and navigations contribute to the ways students develop and come to think about themselves as writers? Four cohorts of writing minors (N=66) and comparison group (N=113) Mixed methods: Writing minors: Statistical analyses on entry • Instructional and demographic data and exit surveys • Entry surveys in semester students entered the program Coded for emergent themes • Exit surveys from semester they graduated related to writerly self-interest • Entry interviews during second semester of study and frameworks for thinking • Exit interviews as they graduate about writing • Application letters to Minor in Writing • Archived gateway and capstone eportfolios Comparison group: Rhetorical move analysis and • Writing samples from each semester in the study corpus linguistics • Directed self-placement essays • Writing samples from first-year writing classes
- 37. Five major findings • • • • • Minorsexpress stronger sense of themselves as writers than in the comparison group. Minors demonstrate greater genre awareness than students in the comparison group. Minors demonstrate a more developed sense of rhetorical situation than students in the comparison group. Minors demonstrate a more developed practice of reflection on writing than students in the comparison group. Eportfolios play an important role in helping students in the minor group understand their development as writers.
- 38. • According tothe research of Clemson University and the University of Michigan, eportfolios contribute to the professional identity development of students in the respective professions of psychology and writing.
- 39. 3. Use ofeportfolios within a system Lamar University and DePaul University have investigated the use of eportfolios across systems. Lamar, in an on-line graduate program, asked questions about effects of eportfolio practice in the university on teachers use of eportfolios in earlier academic levels. DePaul is comparing and contrasting an eportfolio element across disciplines and learning situations.
- 40. Lamar research question:Has participation as a master’s candidate using the eportfolio process contributed to transference of eportfolio practices with PreK-12 students? Method: Web-based survey to 271 program graduates in Fall 2011 and to 202 in Spring 2013 with 49% response rate. Based on Crosswell method of explanatory sequential design using collection of qualitative date to explain quantitative data results. Findings about conditions for implementation in PreK-12: Support factors: administrative support, grading, parental support, internet safety, eportfolio retention time Challenges: assessment, eportfolios as official records, eportfolio reflection time Next inquiry: Case study of entire school district to investigate eportfolio implementation, including contributions of teachers who have and have not studied through eportfolios themselves
- 41. DePaul research: Theaffordances of eportfolios enable more kinds and opportunities for reflection for all students. Reflections are different across schools (professional, adult learners, first-year students). DePaul is studying how and why the differences. • Current work includes coding eportfolios. Much discussion about methodology, but they have developed what they call a “mash-up methodology” on which they have shared agreement. They affirm the great benefit of having researchers from different backgrounds and settings working together. The acceptance of research results and commitment to application to practice are greater with this approach. • Scaling out across an institution will be more possible after this initial work that signals the possibility of application of eportfolios in multiple educational contexts.
- 42. Other sites forfurther specific information about Cohort teams’ research Final reports of all campus teams are posted on the Inter/National Coalition for Electronic Portfolio Research website (2008-2013) Electronic Portfolios 2.0: Emergent Research on Implementation and Impact (Cambridge, Cambridge, and Yancey. Stylus Publishing, 2009) Eportfolios for Lifelong Learning and Assessment (Cambridge, D., Jossey-Bass, 2010)
- 43. Scaling out canlead to core practices. • Scaling out is important, but so is the evolution of core assumptions and practices as more and more local entities embody eportfolio practice and research. The next section of our webinar will focus on core propositions regarding eportfolio learning.
- 44. Cohorts 6 and7 The Non-Negotiable Core of ePortfolio and the Four Propositions
- 45. COHORT SIX Curtin Universityof Technology (Australia) Goshen College Indiana University Purdue University Indianapolis Lamar University Northeastern University Portland State University University of Georgia University of Michigan University of Mississippi Virginia Military Institute Westminster College . COHORT SEVEN Deakin University (AUS) Clemson DePaul Lamar University Loyola of Chicago Santa Clara University University of Cincinnati (two campuses)
- 48. The Non-Negotiable Coreof Electronic Portfolio --Confusion among definitions --Confusion among practices --Interest in identifying what is distinctive about ePortfolios
- 49. Where is reflection? Whenyou complete a "Critical Task" assignment in a course you will upload this assignment to your Assessment Portfolio (AP). Chalk and Wire will notify your instructor that the assignment is ready to be graded. The instructor will grade your Critical Task assignment using a rubric. All Critical Task assignments must be submitted and graded through Chalk and Wire. At the end of your program you will have a complete portfolio of Critical Task assignments that document your professional growth.
- 50. FOUR PROPOSITIONS 1. Interactionof Pieces of Evidence 2. Relationship between Evidence and Reflection 3. Material Practices 4. Meaningful Comparison without Standardization
- 51. PROPOSITION ONE Interaction ofPieces of Evidence For meaningful assessment, interaction of pieces of evidence within an eportfolio is more important than single pieces of evidence
- 53. PROPOSITION TWO Relationship betweenEvidence and Reflection Reflection on pieces of evidence within an eportfolio and on the eportfolio as a whole provides information for assessment that isn’t available by other means
- 54.
- 55. PROPOSITION THREE Material Practices Thematerial practice of eportfolio composition generates distinctive knowledge about learning
- 56. New students have limitedexperiences with various media; and their thinking is patterned and routinized in a way that’s not keyed to insight: they may need new experiences with all kinds of texts. Also complicated by proficiency with technologies and faculty facility with them. Reading portfolios as a practice. Confusion of port reading linked to media: print as linear harder to process; reflection providing role of narrator; what kinds of tools/commentary/ navigations do we need for the design and the readings of portfolios?
- 57. PROPOSITION FOUR Meaningful Comparisonwithout Standardization Eportfolios enable meaningful comparison of student learning across institutions (and other contexts) without standardization
- 58. What is validity? Howdo different perspectives impact the way we assess an ePortfolio? What is the process of assessment, and how does it impact judgments? CONSIDER: The decision-making process should be an interpretive conversation guided by norms. What criteria and evidence can be used in making a decision? When, how, and by whom? And how do you document that decision-making process so others understand how you got there?
- 59. Some Tentative Results 1.Students are often asked to collect artifacts, but not to consider what they learn from seeing how they interact with each other. Considering the interaction of artifacts=an opportunity for learning that is often missed.
- 60. 2. Cohort membersagree that reflection makes a distinctive contribution to learning, but we— faculty, staff, and students—are frequently vague about what we want that distinctive contribution to be. Strong reflection seems to tap at least two domains of learning (e.g., sociolinguistic; psychological; epistemic).
- 61. 3. There issome resistance to the idea that we can have comparability without standardization. It’s not that cohort members want standardization, but they don’t see how they can manage assessment without it. It requires a paradigmatic shift in thinking—much like outcomes have.
- 62. Learn More • Europortfolio:europortfolio.org • Inter/National Coalition for Electronic Portfolio Research: ncepr.org • Eportfolios 2.0: Emergent Findings about Implementation and Impact (Stylus, 2009) • Darren: dcambrid@gmail.com, @dcambrid • Barbara: bcambridge@ncte.org • Kathi: kyancey@fsu.edu