Presentation: "Process and product assessment with e-portfolios in the first academic year at the university of teacher education", Andrea Christen, Martin Hofmann, University of teacher education, St.Gallen, Switzerland.
In this session we will report on the results of an empircal study with a focus on process and product assessment of e-Portfolios in the first academic year at the university of teacher education in Rorschach. The participating students of this study have completed their first year of study . Our College is situated in an monastery, built in 1487.
Associated with e-assessment, we examined formative and summative assessment methods. Summative assessment methods are related to a more cognitive comprehension of learning, which can be made precisely within one core point: What has been learnt by the student at a certain point in time? This assessing method collects results,preferably, at one point in time in the form of multiple choice tasks or short answer essays, for instance. Formative assessment methods in contrast are mostly related to a more constructive comprehension of learning, which can be determined within two core points: First: what are learners‘ needs ? And Second: how can learners‘ be supported by tracking their learning goals? This assessment method demands open evaluation over different points of time. 2. The well-known critc, Ayla, concentrates on the inherent weaknessesof formative assessment methods at colleges: high-flying aims combined with e-Portfolios and their assessment often ignore learners‘ needs. 300 students: nur 5% Miteinbezug der Studierenden bei der Festlegung der Inhalte: Our sittuation is not free as well: Our students do not have total freedom to choose the contents of the e-Portfolio. They are restricted by the 5 learning subjects discipline areas or subject areas is better here) What about the 5% of the participants above out of 300 hundred? Winter (2006): contradiction in awarding ECTS points could be solved through oral e-portfolio tests
The students made15 postings on an E-Portfolio Blog over an investigation period of 21 weeks.
The content of curriculum at our college contain, besides mathematics, languages, arts, social sciences and sports a discipline called: professional skills. This discipline lasts over three years. It contains these five topics in the first study year : the teaching profession, my learning, my motivation, personal and social skills, our learning group At the end of the year, students complete an assessement which verifies these praciticed skills. If students pass, they can continue their studies, if they fail, they have to leave the school or to repeat the first year. ( More precise: (nur, falls zeitlich möglich): exams in all subjects, exam in practical classes, assessment, exam in learning group)
Here, some contents of the five topics can bee seen. For instance: Within The learning field the teaching profession, the students have to read and to discuss the standards of the swiss teaching union. Do these standards refer to their Vorstellung of the profession? Within the personal and social skills topic, students have to work in different learning tasks together, where for instance the skill to contact other persons is trained. Or, within the learning topic „our learning group“, we train by video training some central skills of the profession: these are: assertive behavior, contact care and sympathy care.
Here , you can read, what a student worte in her E-Portfolio about the personal-skill training how to presentate.
Or, here, you can see the comment of the student to the standards of the swiss teachers union.
We started the work with e-portfolios in 2005 with 24 students and 2 professors at the first level of the beginning phase of the studies. These contained reflections about the just presented topics. We continued in 2006/07 with 53 students and 4 professors. : Since 2007/08, all studentes of the first academic year and all professors of the subject „professional and study skills“ participated (twelve in all), 130 students You can see one gain out of this table: 1. the sum of 133 students works can be managed by e-portfolios on a weblog. This would not be the case if we still worked with paper. 2. Our aim is weekly assessed contributions from the students. By technical means, this aim can easily be achieved. The assignment of professors to process or product group took place following the initial participation of professors in 2005. Newly participating professors still assessed the old form product group, at the end of the semester. E-Portfolio experienced professors started with the weekly assessment (process group). By the way: since autumn 2008, all professors assess weekly through direct feedback on the e-portfolio. The results of this study can be presented next year.
We used the following instruments: Questionnaire: we surveyed the students at the beginning of the investigation period in week 40/07 and at the end of the investigation period, Week 08/08. All surveyed students had started their studies of becoming a teacher. The E-Portfolio has been written on Mahara. Mr Hofmann will introduce you to Mahara after this presentation. The students had to achieve 70 percent of of the total points, which are 315 points out of 450 points, in order to get 1.5 ECTS. For one complete posting, students got 30 Points.
Here, you see the feedback sheet, that has been given to students twice during the investigation period. Points have been given weekly by the correcting professors. You can see here clearly, That the choice of material , which means the choice of important contents, has become very well after a short time. The green reflectivity line although becomes better, but declines after some weeks.
For our empirical research, We compared the means of both groups at the intragroup level and the intergroup levels. We have in the first column the interesting variable, for instance here: attitude; and the two investigation groups. We have in the second column the number of completed questionnaires. We have in the third column the means of the two groups before the students started to write in their E-Portfolio. We have in the forth column the probabiltiy, with which the two groups differ from each other by counting the intergroup level at the beginning. We have in the next column the means of the two groups after the students wrote their E-Portfolio. After 15 weeks of work. We have in the last column the means probability of the intragroup comparison, which indicates the probability, with which the variables changed inside of the two groups. We will report now the main results for each variable. For the attitude towards the E-Portfolio, we have a significant better attitude of the students of the Process-group after 15 weeks of work. For the interest about writing an E-Portfolio, we have for both groups a significant and compabarable decline. We had this result in each of the last for years. The means of about 3 points are still at a high level.
For the use of the E-Portfolio for the teaching profession, students of the process group report significantly higher data on a medium level. For the relevance of the E-Portfolio, the students of the process group report a significantly higher appraisal on a medium level. The students of both groups indicate high strain for writing their E-Portfolio. Why do students of the product group show lower appraisal of the use and relevance of E-Portfolios? We suppose two possibel reasons: Students of the product group may not write their E-Portfolio during the semester. They may write it just shortly before the closing date. So, they deal far less with the E-Portfolio and may afterward not perceive the use or relevance,of it but just the urgency to finish the E-Portfolio before the closing date. Another reason may be seen on the relationship level: the students of the product-group may interprete the absence of the weekly assessment as negative and may link this to a lack of evaluation from the professors.
There are no reported differences between the two groups concerning learning progress . But for both groups a high level of reported learning progress between 3.45 and 3. 7. There is an absolutely not expected tendency for the process-group, not to realize real learning progress, if you look at the intragroup comparison. This result is diffcult to explain. One should expect that the weekly working with the E-Portfolio increases learning progress. In the case that we will find similar results in the next years, the assumption of elaborated knowledge-processing when working with an E-Portfolio has to be put into question. Investigation for this variable has clearly to be enforced. It also has to be considered, for instance, whether the perceived lower learning progress of the students with weekly assessment can be explained with a lack of considerations of the real needs of the students. This is despite the fact that students are free to write within the predetermined five learning topics. Concerning the increments of learning contents, there can be found no differences between the two assessing groups. But for both groups, there is a high level of reported increments of learning contents (3.88 to 3.97).
All students would clearly prefer a weekly assessment. The data reveals this at a very high level: 3. to 4.63. The motivation for writing an E-Portfolio is ,after the 21 working weeks for the students of the process-group, significantly higher. This significantly higher motivation still has to be cleared up by the following questions: is this motivation really sustained through the weekly assessement? Or is it sustained through the independent choice of learning contents with the five learning fields? Or is it sustained through the independent choice of working time or working place? Is it a result of the interest in learning content or is it just a result of the relief to finish E-Portfolio work?
We also asked students of this semester, why other students should utilizie an E-Portofolio by an multiple response item. The percents indicate answers of all anserwing students to a single part of the item.
The amount of work for the assessing professors is not different in a significant way. We have an average work load for the professors of the product group of 20h hours for assessing 12 E-Portfolios with 15 posts. And we have an average work load for the professors of the process group of 28h hours respectively 22h without the outlier. So it is just slightly more of work load. This is an important result of this study. We derive from these results, that the weekly assessing method, which is clearly wished by the students, can be maintained. We derive also out of the outliner, that professors have to be carefully introduced to technical knowledge, which is needed for working with E-Portfolios.
5. This result gave rise to a change. 8. We continued our work this year by weekly assessiment of all posts. But we distinguished the weekly assessment by weekly assessment by scale and by weekly assessment by an open feedback of the professors by the feedback function of Mahara. We are working on the results this summer. 9. Even when working with E-Portfolios, it might be necessary to think about a distinction of training- monitoring and assessing function. This topic has been highlighted by the Austrian authors ,Dorninger and Schrack. These authors propose a combined assessment method, which allows students to assess the postings of other students. Students and professors could assess only half of the E-Portfolio postings . The future teachers could thus train their competences for giving feedback to written texts.
Ab hier: Martin-
ePortfolio 2009 London
<ul><li>Process and Product assessment with E-Portfolios in the first academic year at the university of teacher education, Rorschach (Switzerland), 2007/08 </li></ul>
1. Theoretical background: E-Assessment <ul><li>Nothing new…. </li></ul><ul><li>summative and formative assessment methods (Reinmann-Rothmeier & Mandl, 2001; Barret, 2008; Seufert, 2007) </li></ul><ul><li>high-flying aims combined with E-Portfolios and their assessment at colleges often ignore learners needs (Ayla, 2006 ) </li></ul><ul><li>However, E-Portfolios open up multifacited insights into the learning process of students </li></ul><ul><li>This situation requiers a change in assessment methods, tending towards techniques of assessing teachers on the basis of narratives or microcontents (Rhim, 2006, Barrett, 2008,) </li></ul>
2. E-Portfolio at the high school of teacher education (St.Gallen, Switzerland) <ul><li>level IV - last semester </li></ul><ul><li>professional skills </li></ul><ul><li>relation to profession </li></ul><ul><li>level III - last semesters </li></ul><ul><li>empirical research </li></ul><ul><li>special field of study </li></ul><ul><li>level II - all semesters </li></ul><ul><li>Mathematics, Languages, Arts, Music, Natural Sciences, Humanities, Educational Sciences </li></ul><ul><li>project orientated activities </li></ul><ul><li>level I - start phase of studies </li></ul><ul><li>reflection about future profession </li></ul><ul><li>Professional and Study Skills </li></ul>
3. Course of the study Week /year number of postings instruction Feedback to students 40/07 start semester 07 Technical instruction: 2h 42-51/07 process group: 1 post/week (10 posts) product group: 15 posts until week12/08 Technical support on individual demand W42-44 process group: feedback about weekly indiviual points (twice: week 46/50) Product group: no feedback 01-08/08 semesterbreak 01-08/08 08-12/08 Process group: 1 post/week (5 posts) Product group: 15 posts until week12/08 process group: no feedback product group: no feedback 15/08 15/08 feedback to both groups: passed/failed
the teaching profession teaching standards (Swiss teaching union) standards of the govern-ment my learning cognitive skills meta -cognitive skills resource management my motivation qualifications for teaching teachers of tomorrow personal and social skills contact presentation cooperation argumentation reliability management sensitivity emotional competence our learning-group group processes assertive behavior contact care sympathy care
<ul><li>Professional skills: presentations </li></ul><ul><li>These are the cards, I used for my presentation. </li></ul><ul><li>My teacher told me that I should try to keep in </li></ul><ul><li>eye-contact with the public. </li></ul><ul><li>I can learn from this that the public is more </li></ul><ul><li>important than my PowerPoint presentation. </li></ul><ul><li>I think I can use this knoweldge when I have </li></ul><ul><li>to present the organisation of our ski-camp </li></ul><ul><li>to the parents. </li></ul>
<ul><li>The teaching profession: </li></ul><ul><li>Teacher‘s standards </li></ul><ul><li>(Swiss teaching union) </li></ul><ul><li>For most of these rules, we agreed . </li></ul><ul><li>That‘s clear! </li></ul><ul><li>However, I think, one has to discuss and to </li></ul><ul><li>think about teacher‘s ethics and one has to know </li></ul><ul><li>where to seek them. </li></ul>
“ The students in the next new semester should utilize an e-portfolio because:” <ul><li> they will acquire new PC skills as well (13.1%) </li></ul><ul><li>they will then already know Mahara and can possibly later use the system with their students (17.7%) </li></ul><ul><li>the composition of postings supports the consolidation of BSK content (68.5%) </li></ul><ul><li>the composition of postings in some ways makes it enjoyable (4.6%) </li></ul><ul><li>the composition of postings necessitates the processing of weekly content (65.4%) </li></ul><ul><li>one can also read the blogs of others (43.1%) </li></ul><ul><li>one learns something easily by doing it (20%) </li></ul><ul><li>the feedback from lecturers brings something to it (30.8%) </li></ul><ul><li>it is a good method of assessment (43.1%) </li></ul>
8. Results: workload approach of assessing professors <ul><li>12 E-Portfolios/ assessment end of semester : Ø20h </li></ul><ul><li>12 E-Portfolios/ weekly assessment: Ø28h (Ø22 h without outlier) </li></ul>
9. Discussion <ul><li>Weekly assessment maps: </li></ul><ul><li>More positive attitude towards the E-Portfolio </li></ul><ul><li>Higher appraisel for the relevance and use of an E-Portfolio </li></ul><ul><li>Higher motivation </li></ul><ul><li>Clear preference of the weekly assessment method </li></ul><ul><li>No significant differences between assessment methods concerning the workload of the professors </li></ul><ul><li>Independent of assessment method: </li></ul><ul><li>6. Interest, strain </li></ul><ul><li>7. Learning progress </li></ul><ul><li>8. Increments of learning contents </li></ul><ul><li>Perspective: </li></ul><ul><li>9. Weekly assessed posts by open feedback function </li></ul><ul><li>10. Distinction of training-, monitoring- and assessing function (Dorninger & Schrack, 2008) </li></ul>
10. E-Portfolios: Guided Tour London 09 www.eportfolio-phsg.ch 1 2