Demonstrating Competencies with E-Portfolios: The Carolina MPA
•Download as PPTX, PDF•
1 like•1,019 views
Presentation at E-Learn 2014 International Conference. describes the conceptual design, instructional development and organizational implementation involved with the transition from a traditional end of program capstone project to a competence-oriented portfolio and oral exam assessment in a public administration graduate program.
Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Demonstrating Competencies with E-Portfolios: The Carolina MPA
Similar to Demonstrating Competencies with E-Portfolios: The Carolina MPA (20)
More from Stefanie Panke
More from Stefanie Panke (20)
Recently uploaded
Recently uploaded (20)
Demonstrating Competencies with E-Portfolios: The Carolina MPA
- 1. Demonstrating Competencies with E-Portfolios: The Carolina MPA Stefanie Panke & John Stephens University of North Carolina at Chapel Hill AACE E-Learn 2014 October 27th New Orleans
- 3. Why Portfolios? Learning and assessment are two sides of the same coin, and they strongly influence each other. Gulikers, Bastiaens & Kirschner (2004)
- 4. From ‘assessment of learning’ towards ‘assessment for learning’ Knowledge imparted by the instructor (input) Competencies students can apply (output).
- 5. Integrated, complex construct of knowledge, skills and attitudes that can be used in order to solve arising problems and succeed in handling (new) situations (Baartman et al., 2007). Combination of knowledge, skills, understanding, values, attitudes and desires, which lead to effective, embodied human action in the world, in a particular domain. (Buckingham et al., 2012)
- 7. Competencies
- 8. Reflective Leadership and Professionalism • Enter shared-goal conversation that values the leader and his or her expertise (Hyland & Kranzow, 2012) • Fostering critical reflection as means of developing expertise, critical self-surveillance whereby professional experiences are revisited and explored (McNeill, Brown & Shaw, 2010).
- 9. Types of Portfolios (Hewett, 2004) • Documentation portfolios: growth toward learning goals • Process portfolios: phases of the learning process • Showcase portfolios: accomplishments and competencies
- 10. Showcase & Process Portfolio
- 11. Carolina MPA E-Portfolios • Graduation requirement (portfolio/ oral exam) • Campus-wide WordPress Multisite • PUBA 746 portfolio course (peer feedback/ milestones) • Portfolio Guide • Competence Memo Template • Portfolio Assessment Rubric
- 13. Benefits of e-portfolios • Encouraging reflection (Roberts, Maor & Herrington, 2013). • Promoting self-regulation (Abrami et al, 2008, Meyer at al, 2010). • Improving knowledge management (Chang, Tseng, Liang & Chen, 2013). • Acknowledging diversity and transfer learning (O’Toole, 2013).
- 14. Benefits of e-portfolios • Fostering digital literacy /multimedia storytelling (Wakimoto & Lewis, 2014). • Supporting career development (Reese & Levy, 2009). • Strengthening organizational ties (cf. Lorenzo & Ittelson, 2005; Reese & Levy, 2009).
- 15. Barriers to Implementation & Use • Implementing e-portfolios is a complex process fraught with challenges and dilemmas (Chau & Cheng, 2010). • The promotion of reflective thinking and practice are not an automatic result of creating a portfolio (Wray, 2007).
- 16. Barriers to Implementation & Use • Tensions between personal reflection and institutional requirements (Tosh, Light, Fleming & Haywood, 2005) • Written reflection vs. metacognitive processes (McNeill, Brown & Shaw, 2010) • Unclear expectations and assessment strategies • Conflicting portfolio goals
- 17. Portfolio Evaluation • Faculty focus group • Course evaluation results PUBA 746 • Student online survey. • Participants: • 18 students (15 female, 3 male) • 17 faculty members (10 female, 7 male).
- 18. Student Online Survey • Administered 2 month after graduation • Survey software Qualtrics • Incentive $10 gift certificate • 23 Likert scale items: Strongly Disagree (1) - Strongly Agree (6) • 3 open-ended questions • 15/18 students (83%) http://tinyurl.com/nzk7lr6
- 19. Learning Trajectories and Transfer Learning • 60% included products created outside class. • 73% agree that the portfolio brings together classroom learning with professional experiences and personal background. • 2% plan to use the portfolio in the future.
- 20. Foster Digital Literacy % who agree or strongly agree • Web publishing is an important skill: 73% • The portfolio has improved the general technical abilities to develop a professional website: 1% • It was easy to set up the portfolio in WordPress: 53%
- 21. Support Career Prospects • Only one student shared the portfolio with a potential employer. • ‘It's too personal. Changing the portfolio to be less introspective would make it a less useful assignment, though’. • ‘By the time I could share the portfolio, I had a job’.
- 22. Assessment Students diagreed: • that the assessment process was fair: 40% • that they received helpful feedback: 33% • that the committee reviewed thoroughly: 33% • ‘a bit of inconsistency with the expectations of different faculty committees made the process seem unfair’. • ‘by allowing us to put as much material in the portfolio as we wanted, it was simply too much for them to review’.
- 23. Reflection • 80% were proud of their portfolio. • 93% agreed that the portfolio process prompted them to reflect their competencies as public service leaders. • ‘extremely worthwhile exercise’. • ‘excellent opportunity to increase self-awareness and plan for future development’ • ‘Overall, the portfolio did help me reflect on what I learned in the program’. • ‘I appreciated the flexibility we were given to design our portfolios to reflect our strengths’.
- 24. Summary Positive Outcomes • Dialogue on how to interpret central learning outcomes of the program • Bridge between classroom reality and professional experiences • Reflective leadership Areas for Improvement • Consistency in assessment: clear and shared expectations • Increased ownership: Roadmap for future use of portfolio • Balance between structure and autonomy during portfolio porcess
- 25. Outlook • Currently: 2 online students are completing portfolio process • Next spring: Cohort of 35 online and residential students • Faculty Learning Communities • Sample Portfolios
Editor's Notes
- An e-portfolio is a systematically curated exhibition of learning products. Students use their portfolios to collect their work, select and highlight examples they want to showcase, and to reflect, discuss and potentially even advance their learning. The collection as a whole presents the student’s learning goals, learning processes, and learning outcomes. These processes are basically the same for printed portfolios or e-portfolios. The electronic format makes it easy to link artifacts and reflections and allows to include a richer variety of content such as multimedia elements or datasets.
- Learning and assessment are two sides of the same coin. That means that learning goals and assessment techniques need to be aligned. If we want to change student learning in the direction of competency development, we need to align authentic, competency-oriented instruction with authentic, competency-oriented assessment.
- Portfolios are a tool for supporting authentic assessment, they allow a shift from ‘assessment of learning’ towards ‘assessment for learning’. Instead of assessing how well students can reproduce knowledge imparted by the instructor (input), the focus is on the competencies students can apply (output).
- Competencies are an integrated, complex construct of knowledge, skills and attitudes. They describe what we can do with what we know and how we can transfer it to handling new situations. Instructors often assume a smooth learning trajectory as students move through the curriculum from course to course, from year to year and eventually into the workplace. However, accomplishments in authentic settings require more than knowledge. Learners need skills, understanding, values, attitudes and desires to apply their knowledge in a particular context.
- Prompted by a shift in accreditation standards that call for a competency-focused curriculum, the Carolina MPA has undergone a complete curriculum redesign. The faculty committee developed a set of eight broad competencies with 24 intermediary competencies that together define the learning outcomes the program seeks to impart. Starting in fall 2012, all program requirements and core course content are built around these competencies.
- The eight central learning outcomes are obviously specific to the MPA program and include for example to lead manage and engage others in public service, to articulate and apply public service values, to understand law and legal process or to manage financial resources. Just to exemplify how the main competencies are broken down, I have selected one area that is perhaps of more broader interest: Analyze information for decision making.
- Hewett (2004) distinguishes three basic types of portfolios that support different assessment purposes: Documentation portfolios show the growth toward achieving specific, pre-defined learning goals. They support diagnostic assessment and allow students and instructors to both plan and check how the learner is progressing. Process portfolios document the phases of the learning process and reflect upon the students’ journey towards mastery. They make students cognizant of how they learn best and support self-assessment of learning strategies. Showcase portfolios focus on the students' accomplishments and competences. They include the students' best works and reflections on how and why the work products were selected. They support summative assessment of students’ competences and learning outcomes. We are currently in the process of getting IRB approval for investigating how well our portfolio approach meet the needs of program leadership, faculty and students. Our research focus is twofold: First of all, we want to understand how well different types of portfolio enable specific benefits – and inhibit others. Second, we want to understand what are effective techniques for assessing portfolios. We do this through a series of online questionnaires, focus groups and interview. We are eager to share our instruments and discuss our results.
- My motivation to delve deeper into the topic of e-portfolios stems from my work as an Instructional Analyst at UNC School of Government. In the past 2 years, our Master of Public Administration Program shifted from a capstone project to an e-portfolio and oral exam process for graduation. In the Carolina MPA program the student portfolio is a pivotal piece of the new, competence-oriented curriculum. Students use the portfolio to document their level of competence in central learning outcomes of the program. It needs to pass a three person faculty review committee, before students can move on to the final oral exam. This spring, the first cohort completed the portfolio process. When it came to evaluating and reflecting our portfolio experiences, we asked ourselves – what are the benchmarks of success? Is it enough if our students and our faculty are fairly happy with the process? What experiences do other campuses have?
- To this end, I did a lit review of portfolio outcomes so we can see of we achieve this. Benefits of e-portfolios documented in the literature include: Fostering digital literacy and multimedia storytelling: By creating digital exhibitions spaces of their work, students gain technology, writing and multimedia communication skills en passant (Wakimoto & Lewis, 2014). Supporting career development: The digital collection of work samples and skill demonstrations can be easily shared with potential employers (Reese & Levy, 2009). Strengthening organizational ties: E-Portfolios may link students to their alma mater even after graduation and can be used to connect alumni and prospective students (cf. Lorenzo & Ittelson, 2005; Reese & Levy, 2009). Encouraging reflection: E-portfolios enable learners to review their learning processes and outcomes by self-reflection and comments from peers (Roberts, Maor & Herrington, 2013). Promoting self-regulation: Portfolios that stress the reflective component have the potential to raise students’ metacognitive awareness (Abrami et al, 2008, Meyer at al, 2010). Improving knowledge management. The process of conceptualizing, implementing and developing an e-portfolio involves collecting, arranging, re-organizing and presenting information. E-portfolios can thereby facilitate knowledge management performance (Chang, Tseng, Liang & Chen, 2013). Acknowledging diversity and transfer learning: Portfolios are a natural fit for assessing networked learners in their personal learning environments Panke, 2013). The e-portfolio brings diverse student outputs from a range of different learning and working contexts into one common format and thus facilitates assessment (O’Toole, 2013). This makes portfolios sound like the proverbial jackalope. However, not all of the benefits will manifest in every instance of a portfolio centered assessment. It is important to keep in mind that there are different portfolios for different purposes.
- To this end, I did a lit review of portfolio outcomes so we can see of we achieve this. Benefits of e-portfolios documented in the literature include: Fostering digital literacy and multimedia storytelling: By creating digital exhibitions spaces of their work, students gain technology, writing and multimedia communication skills en passant (Wakimoto & Lewis, 2014). Supporting career development: The digital collection of work samples and skill demonstrations can be easily shared with potential employers (Reese & Levy, 2009). Strengthening organizational ties: E-Portfolios may link students to their alma mater even after graduation and can be used to connect alumni and prospective students (cf. Lorenzo & Ittelson, 2005; Reese & Levy, 2009). Encouraging reflection: E-portfolios enable learners to review their learning processes and outcomes by self-reflection and comments from peers (Roberts, Maor & Herrington, 2013). Promoting self-regulation: Portfolios that stress the reflective component have the potential to raise students’ metacognitive awareness (Abrami et al, 2008, Meyer at al, 2010). Improving knowledge management. The process of conceptualizing, implementing and developing an e-portfolio involves collecting, arranging, re-organizing and presenting information. E-portfolios can thereby facilitate knowledge management performance (Chang, Tseng, Liang & Chen, 2013). Acknowledging diversity and transfer learning: Portfolios are a natural fit for assessing networked learners in their personal learning environments Panke, 2013). The e-portfolio brings diverse student outputs from a range of different learning and working contexts into one common format and thus facilitates assessment (O’Toole, 2013). This makes portfolios sound like the proverbial jackalope. However, not all of the benefits will manifest in every instance of a portfolio centered assessment. It is important to keep in mind that there are different portfolios for different purposes.
- Tensions between personal reflection and institutional requirements: Tosh, Light, Fleming & Haywood (2005) criticized that oftentimes portfolios offer pseudo-authentic student voices, and are viewed as ‘just another assignment’. Instead of valuable reflection, students produce what they think the instructor or program requires. As Chau & Cheng (2010) observed: “students may conceive and shape written messages based on institutional requirements” (475). Discrepancy between written reflection and metacognitive processes: McNeill, Brown & Shaw (2010) questioned whether recorded reflection is a true picture of the cognitive process involved. Specifically, the authors raised the question as to whether participants who did not have fluent writing skills were able to fully convey the cognitive processes involved in reflection, or indeed, whether participants who were accomplished writers could appear to attain deeper levels of reflection. Difficulties with portfolio expectations and assessment: Although portfolios are often praised as an authentic assessment strategy, little is known about how to establish fair and transparent processes for assessing the portfolio itself. As Fitch et al. (2008) reported, faculty may struggle with the idea why one should assess assignments already graded. Conflicting portfolio goals: It remains an open question how well different types of portfolio enable specific benefits – and inhibit others. For instance, a portfolio that emphasizes reflection and fosters metacognitive awareness may not be the best tool to support career development – and vice versa.
- We obtained IRB approval for evaluating the portfolio process in a mixed method approach that included (1) a faculty focus group, (2) the course evaluation results of PUBA 746 and (3) online survey for students, administered 2 month after graduation.
- Two month after graduation, the research team administered an online survey with the survey software Qualtrics. We reached out to the 18 students in the first MPA cohort in the portfolio condition to learn about how they viewed the process in retrospect. We offered a $10 gift certificate to enhance the response rate and as a token of appreciation for the students’ willingness to provide ongoing feedback and endure uncertainty and change as the portfolio pilot group. The survey comprised a total of 25 items, most of them with a six point likert scale in order to obtain clear trends. In addition, we included 3 open-ended questions to get suggestions for improvement and other qualitative input.
- Both for students and faculty members, assessing portfolios is a crucial and difficult aspect of the portfolio process. The faculty reviewers have to develop a shared approach to qualify an appropriate level of reflection and adequate selection of learning products in the portfolios. Though the portfolio assessment rubric was a useful instrument for assessing the portfolios, it did not prevent both students and faculty from unrealistic expectations about the quality learning products, the level of reflection and the detail and comprehensiveness of faculty feedback. Training and discussion of portfolio samples allow both sides to clarify expectations
- Finally, I don’t want to come across as overly critical of standardized, multiple choice testing – in fact, this is really not my area of expertise. I completed my education up to my PhD in Germany, back ‘in the good old days’, prior to Bologna Reform. Thinking back on my personal educational experience, I have to say that the only standardized multiple choice quiz I ever took was the theoretical part of my driver’s license test. And it might be worth mentioning that I passed that one with flying colors. However, I did fail the practical part the first time around. And, while on second try I eventually did get my permit, I still remember the tester saying, ‘Young lady, I would never let you drive my car’.