Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Definite - paper curses A&E, 2006-2007




Evaluation and assessment in the VU Medical Centre:
Support or threat of valid ...
Introduction
Initial nursing education aims to educate nurses to continuously provide patient or client care of high
quali...
Spencer and Spencer (1993) define competences as the underlying characteristics linked to
performance in a (work) situatio...
approach will require a rigorous performance assessment system and less reluctance to fail those
students not meeting the ...
interpretation (Gipps, 1994). Reliability is the extent to which one can rely on test results as a
measurement (Van Berkel...
Table 1 Model of educational assessment for use in validation and planning of assessments (Crooks et al., 2000).
Link     ...
value of the consequences depends on the quality of the process of judgement and on the use of
concepts. By making specifi...
individual nurse and nurse educator as well. This is not good for inter-rater reliability. For judging
objectively, requir...
Thirdly, nurse educators and practioners need to improve themselves in judging and coaching students
in a positive objecti...
Appendix1: Original forms of the VUMC of Amsterdam

-   VUMC - richtlijn voor reflectieverslag, tussenbeoordeling en eindb...
Appendix1: Original forms of the VUMC of Amsterdam

Richtlijn voor reflectieverslag, tussenbeoordeling en eindbeoordeling
...
Tussenbeoordelings- en eindbeoordelingsgesprek

Doelstelling                    De tussenbeoordeling en eindbeoordeling zi...
Tussenbeoordeling
Naam student                          :
Groep                                 :
Praktijkleerperiode     ...
2) Patiëntgerichtheid, de student:
        Criteria                                                                       ...
5) Coachen, de student: (alleen indien van toepassing )
       Criteria                                                   ...
Eindbeoordeling
Naam student                          :
Groep                                 :
Praktijkleerperiode       ...
Criteria                                                                                    Beoordeling
                  ...
Toelichting (werkbegeleider vult dit altijd in):



Reactie van student op bovenstaande:



5) Coachen, de student (alleen...
Appendix 2
         -    Reflectie op de verkregen feedback en het gebruik van deze feedback
         -    Ervaringen bij ...
Assessment & Evaluation, definite paper, june 2007.   19
Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
Upcoming SlideShare
Loading in …5
×

Evaluatie en assessment binnen de opleiding Verpleegkunde VUMC

2,666 views

Published on

  • Be the first to comment

  • Be the first to like this

Evaluatie en assessment binnen de opleiding Verpleegkunde VUMC

  1. 1. Definite - paper curses A&E, 2006-2007 Evaluation and assessment in the VU Medical Centre: Support or threat of valid use of competence-based education in initial nurse practical education A.W.Duit 3090469 I. de Lijser F053640 F.Vasse F053636 Group 26 b June 2007
  2. 2. Introduction Initial nursing education aims to educate nurses to continuously provide patient or client care of high quality. In the last few years higher education has innovated its curricula to competence-based education. Hospitals on the other hand, take care of the practical education. Students have to apply what they learn at school to practical situations in the hospital. At college they get competence-based education at different levels (beginner, junior and bachelor) to develop themselves as a professional in nursing. Official bodies in higher education defined five different roles of nursing, matching ten competencies, which have been adopted by colleges and hospitals. These roles are care provider, manager, designer, coach and professional. At the `Vrije Universiteit Medisch Centrum‟ (VUMC) hospital of Amsterdam theoretical and practical knowledge of competence-based education is growing, but it has not been completely implemented by all nurses and educators who take care of students. There is a discrepancy between the interests of college and hospital concerning learning goals. Innovation of education in a competence-based curriculum means a different kind of evaluation and assessment in both settings. At college they have developed several competence-based evaluation and assessment instruments, but the hospital has not yet changed its instruments to a competence-based assessment tool even though the way of thinking is getting more competence-based. Students come to the hospital with competence- based goals, but will be confronted with evaluation and assessment instruments, which are not competence-based. The main question is: are the current evaluation and assessment instruments of the VUMC hospital valid and reliable enough to measure what they need to measure, in alignment with competence-based education for developing a professional performance as beginner, junior or bachelor? First, there is a literature study on the concepts of competence and competence-based evaluation and assessment. This leads to criteria of evaluation of competence-based assessment and an analysis of the current evaluation and assessment instruments of the VUMC. This will be followed by a summary of the outcomes in the conclusion and recommendations section. Competence. Competence is an internationally known concept, but the meaning differs in various countries. The concept competency in the USA for example, refers to characteristics of behaviour, whereas the concept competence in the UK refers to the standards for assessing the behaviour. The meaning is also influenced by the learning theory the educators and scientists use for the concept of competences. In the behaviouristic and cognitivistic approach motivation, standards and values have little meaning in the notion of learning, but they have a bigger role in the constructivists‟ approach to learning (Van Merriënboer, Van der Klink & Hendriks, 2002). Van Merriënboer and colleagues (2002) describe eleven definitions of competence, concluding that three aspects influence the view on competence: the geographic context, the learning theoretical perspective and the purpose of the definition. They propose to change the concept of feature of competence into the concept of „dimension‟, because of the diversity from high to low, in contradiction to feature which imply dichotomy. The authors also describe three dimensions for defining the concept of competence. First, competence can be more or less specific to a generic context or more applicable to a specific job environment. Second, competence comprises the integration of knowledge skills, attitudes and characteristics necessary for solving complex problems. Third, competence is more or less stable, which does not mean that it cannot be changed. They also distinguish three dimensions of competence that are specifically relevant to an educational setting. First, a prerequisite for development and assessment of competence is a focus on action of performances, which can provide understanding of underlying mental activities. Second, development of competence requires practice, effort and time. Third, different competences are related to each other, though not in a strict hierarchical way. Some competences are necessary for the development of others. For example to be able to make a diagnosis, competence in keeping proper patient history is required for a doctor (Baartman, Bastiaens & Kirschner, 2004). Besides the features mentioned above, motivation, ambition and curiosity are crucial to develop abilities (Van Merriënboer et al, 2002). Assessment & Evaluation, definite paper, june 2007. 1 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  3. 3. Spencer and Spencer (1993) define competences as the underlying characteristics linked to performance in a (work) situation and they distinguish five characteristics, visualised by an iceberg. This metaphor makes clear that a limited part of the competences is directly visible and assessable; the part above the water, consisting of knowledge and skills. The visible characteristics are easy to test, in contrast to the characteristics below the waterline: motivation, self-concept and character profile. Scott (1991) notices the process of problem construction. A person needs time to think about the situation which makes the problem come to the surface (see figure 1). This process is constructive and when a person finishes this process, he is able to select appropriate performance. This creative thinking process and acting in real and vocational life, needs a certain attitude. A competent person needs to reflect and learn from his experiences. On the other hand there is a difference between what an individual should be able to do at an expected level of performance and what they actually do in the real-life setting. Competence is not always highly correlated with performance in practice, and greater attention should be paid to assessing the performance when the practitioner is moved to an unsupervised situation (While, 1994 So, the concept of competence has many definitions, is an influenced ability and is not always visible in performance. The most important elements of competence are the abilities to perform effectively. Underlying competences or characteristics like motivation and self-concept are not seen, but influence the action of performance. Job Specific Performance Skills Way of Thinking Attitude Figure 1. Elements of competences by Scott (1991). Competence and performance Competence is concerned with the ability to perform effectively in different occasions and contexts, dealing with unexpected occurrences. It is about what a person potentially knows and can do in ideal circumstances. Performance is real behaviour: what is actually done in the real-life context. A high quality performance means choosing the best course of action for each particular situation by using „knowledge for the selection of alternatives‟. Students should be competent to practice after they have finished their educational programmes. Competence is a necessary, but not sufficient requirement for adequate professional practice. Quality assurance in health care concerns care actually received, regardless the context or difficulties in situation. The context in which care has to be given may be an explanation for poor performance; it is not a sufficient excuse for poor professional practice. Nursing of good quality depends on nurses who are dedicated, sincere, devoted and enthusiastic. Nurses who possess the scientific knowledge and technical skills necessary to carry out their professional responsibilities efficiently en effectively. (While, 1994) It is important that nurse education prepares practitioners who are able to use their knowledge and skills, for taking their responsibilities in providing high quality care. Several factors affect the performance: expectations of the health care institution, colleagues, reference groups such as professional organisations and own expectations or own roll image. There are three additional factors: attitudes and expectations of superiors, the nature of the work assignment and the quality and amount of work-related communication. Other intermediaries between competence and high quality performance are honesty, judgement, work habits and maturity. Psychological stability and adaptive capacities, anxiety, low motivation and fatigue affect performance as well. Apparently competent registered practitioners do not always perform at an adequate level. A more relevant focus of nurse education should be the performance of students in real-life situations, being alert to those potential nurses who will not perform adequately when registered, despite their apparent competence. This Assessment & Evaluation, definite paper, june 2007. 2 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  4. 4. approach will require a rigorous performance assessment system and less reluctance to fail those students not meeting the designated standard. (While, 1994) So, in the practice of nursing education it is necessary to look at the performance of the student several times in supervised situations to make sure the student is competent in the same kind of situations, when unsupervised. The aim of competence nursing education is to deliver professional registered nurses, who have learned to argument their way of thinking, are motivated to have a learning attitude and function at an adequate level of job performance. This also means that, as Van Merriënboer and colleagues (2002) appoint, the specified level of dimension, integration and stability of competences have to be practiced in complex situations to create more integration of knowledge, skills and attitude to deliver professionals who own job performances skills for the rest of their life. As Biggs (1999) says, if the goal of education is the acquisition of competence, assessment must be based on competence. Competence-based evaluation and assessment When education is based on competence instruction, learning and assessment should be aligned (Biggs, 1999). Competence-based education and traditional evaluation forms are in contradiction and will obstruct the development of students‟ competences. Students adjust their behaviour to the method of assessment. The purpose of competence-based education is to focus on development of competences. Assessment is supposed to link and integrate learning, instruction and evaluation with respect to principles of equality. Traditionally, evaluation is an instrument of assessment to measure achievement after performance and discussion. The condition to guarantee psychometric request of reliability is to standardise as much as possible. Within this approach assessment is supposed to measure the knowledge of the student and to classify him efficiently. To realise development of competence, an efficient assessment method within the education environment is important. Competence-based education and learning could impossibly be received without competence-based assessments. Changing the methods of assessment is crucial for changing the students‟ behaviour to learning. (Gibbs, 2006). To realise the purpose of competence-based assessment it is important to adjust proper methods of assessment. Not only measure knowledge, skills and attitude but complete integration and interaction of those as well. The term assessment means that the overall performance of the person has priority. Dochy and Nickmans (2005) appoint measuring competences to Assessment Of Learning (AOL) and driving students behaviour into desired competences to Assessment For Learning (AFL). The first aims to the outcome of the learning process; the second aims to the learning process itself and connects assessment to the shift to student-centred, competence-based education and new methods of learning. New methods of assessment include product and process, they are never exclusively directed to evaluation of product. AFL provides information about methods to guide, support and drive the students learning process. The focus is to gather all kinds of information about the progression of the student. Feedback helps the student to close the gap between current and desired competences and current knowledge, conception en skills. It enables the teacher to align instruction and supervision to each specific student. The purpose is insight and optimisation of the learning process and includes optimisation of the education process (Dochy and Nickmans, 2005). Changing from traditional to competence-based education means aligning education and assessments of competences by not only measure AOL but AFL as well. Requirements to assess competences Measurements can be as productive and accurate as the competences selected to be measured (While, 1994). But what are requirements to assess competence? Van Berkel and Bax (2006) give justified requirements to judge test questions in higher education. These questions assess control of knowledge as a competence of the student and will lead to a certificate. The requirements serve as a guideline and are validity, reliability, objectivity, transparency and standardisation. The traditional definition of validity is the extent to which a test measures that which it was designed to measure. Early writing mentions predictive, concurrent, construct and content validity: does a test predict future performance, correlate with a similar test, have an adequate measure of construct and covers an appropriate and necessary content? A more extensive definition of validity is consequential validity and concerns the consideration of the consequences of test use and Assessment & Evaluation, definite paper, june 2007. 3 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  5. 5. interpretation (Gipps, 1994). Reliability is the extent to which one can rely on test results as a measurement (Van Berkel and Bax, 2006). Gipps (1994) also stresses that a score needs to be independent of circumstances or persons; comparability and repeatability is necessary, just as similar scoring by all markers (inter-rater reliability) and by the same marker (intra-rater reliability). Van Berkel and Bax (2006) say objectivity is most commonly seen as not influenced by aspects, independent of disorders or resistances. For transparency availability of all information needed for good prepare-and-answer strategies is essential. Standardisation is important in a limiting case and opposite decisions of failure or success have to be made. There are two ways to justify: predictive ability of a test or concerning content arguments. Because the first is problematic, it is necessary to show which objectives have been tested and why they are relevant. If there are bad job practitioners the education selects wrong and has a bad pass or fail limit in parts of the education (Van Berkel en Bax, 2006). To assess competence (consequential) validity, reliability, objectivity, transparency and standardisation, as written above, are important for judging job specific performance. The student is also part of the work force and judging on wrong basis with the consequence of losing the job and hurting somebody‟s self-respect can hurt for life. Therefore the scoring of assessment is a difficult situation, which needs to be done very carefully and objective by educated markers to respect the quality requirement of assessment. This is why the next part is specifically about validity and assessment of competence. Validity and assessment of competence Validity is the most important quality requirement of an assessment, but its evaluation is often neglected (Crooks, Kane & Cohen, 2000). It is impossible to measure competence reliable. Being accurate and independent of factors by chance is a restriction for being valid, but competence is not a homogeneous or stable concept and judgements of competence are subjective. Within the psychometric approach homogeneous is an important aspect of reliability and because competence is not a homogeneous concept this approach is not suitable. The same holds for the stability or repeatability of measurements within the system theory and for the intra- rater reliability within the social-constructive vision (Luken, 2004). Van Merriënboer and colleagues (2002) mention that the concept of competence includes context, its bound context. Luken (2004) states that assessment of competence is assigned to a person in a context and takes a certification to all kinds of different contexts in future situations. Professions consist of many varying and quickly changing functions. Competence must be defined generally for including all the situations and must be defined specific for being really legitimate, which is a contradiction. Objectives of assessments are necessary, but within a profession primary focus is the reality of the job and learning within education. Lack of motivation and not being well can lead to not using a present ability. Competence is integrated, bound by context and holistic of nature, but students need an analytic approach of detailed feedback. This means it is hard to measure competence and impossible to measure it in a valid way (Luken, 2004). There is a difference between affirming a primary focus on validity and acting upon it. In practice, validity has often received less attention than reliability or generality. Validity examination relies heavily on human judgement and is therefore hard to carry out, report and defend. It is also vulnerable to new negative evidence, which may lower confidence in the assessment. A threat is the continuous neglect of validity in monitoring the quality of assessments (Crooks et al., 2000). The authors present a structured model of educational assessment (see table 1, page 5), as a framework for evaluating validity of assessment use and interpretations. It tries to build validity into assessments, which are under development as well. The model identifies eight steps in assessment use and interpretation to indicate where it can go wrong. Each step or link represents issues that need to be addressed in any validation effort. Examining each link looking for weaknesses in the chain of inference provides a systematic approach to validation. The relative influence of each link on validity can vary in different cases. Listed threats need consideration in each case, so is any further threat associated with the particular context of assessment. Validation requires careful consideration of the strength of each of the links, observing the assessment process in action and evaluating the threats to validity associated with each of the links (Crooks et al, 2000). This means that the reality of the job is essential in nursing education and the model of Crooks and colleagues (2000) can serve as a guideline for validity of assessment of competence. Assessment & Evaluation, definite paper, june 2007. 4 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  6. 6. Table 1 Model of educational assessment for use in validation and planning of assessments (Crooks et al., 2000). Link Threat Admini- Low motivation; Assessment anxiety; Inappropriate assessment conditions & Stration Task or response not communicated. Scoring Scoring fails to capture important qualities of task performance; Indue emphasis on some criteria, forms or styles of response; Lack of intra-ratter or inter-ratter consistency; Scoring too analytic & Scoring too holistic. Aggre- Aggregated tasks too diverse & Gation Inappropriate weights given to different aspects of performance. Genera- Conditions of assessment too variable; Lization Inconsistency in scoring criteria for different tasks & Too few tasks. Extra- Conditions of assessment too constrained & Polation Parts of the target domain not assessed or given little weight. Evalua- Poor grasp of assessment information and its limitations; Tion Inadequately supported construct interpretation & Biased interpretation or explanation. Decision Inappropriate standards & Poor pedagogical decisions. Impact Positive consequences not achieved & Serious negative impact occurs. Approaches and methods to assess competences The goal of assessment of competence is to evaluate developed competences by judging performance, effective application of knowledge and skills in a practical setting. Used techniques address psychomotor, cognitive, and affective domains. Competences can be generic to clinical practice in any setting and specific to a clinical specialty (Redman, Lenburg & Walker, 1999). Starting point can be a list of criteria, based on valid performance of competent practitioners according to fixed objectives or to standards of level of control (for example failure, adequate or excellent). Criterion referenced evaluation methods evaluate performance against a set of criteria, known to learner and assessor. Standards referenced methods translate the level of performance to a scale, which has benefits: measuring more valid en reliable, quality can be judged by scale analysis, it can be measured more accurate by putting items together and distinctions can be made. The approach can be analytic, assessing all parts separately, or holistic by total assessment. Judgement can have the possibility to compensate or to be conjunctive, which means a minimum level must have been achieved for any part. Glaser (1963) makes a distinction between relative norm referenced testing and absolute criterion referenced testing as a result of educational assessment. The first is a psychometrics and psychological measurement based on classical test theory, build on requirements of prediction and correlation. The second gives information of the degree of competences attained by a particular student independent of the performance of others. It is an indication of current degree of performance in relation to criteria of optimal performance ranging from no proficiency at all to perfect performance. Making the content of testing measurement explicit leads to better interpretation of test scores, which has an effect on consequences of test interpretation (Gipps, 1994). Performance of an examinee in a real or realistic setting is the basis of assessment of competence. Because of the nature of development of competence there can be different measurement points for linking judgement. Grotendorst and colleagues (2006) give a model which measures the achieved point of development of competence. The measurement points are awareness of not knowing or not being able, showing observable behaviour results, relapse in results is sometimes a progression, first success experience, awareness of control at results, finale of practise and at the end automatic and integration. Assessing a student‟s achievement is important to verify the criterion (Grotendorst, Van Aken, Heida, & Sino, 2006). To develop tools of assessment for higher education Luken (2004) takes knowledge, attitude and skills as a starting point. Luken (2004) appoints these elements of competence are more measurable than the total concept of competence, but they are less valid and hence there is a dilemma. A matrix of competence measures competence through a dimension of objectives and a dimension of previously named elements of competence. The first dimension gives performance of tasks with descriptions of examples; the other gives accomplishment of the task and is legitimate outside the field of activity. After an objective analytical judgement a judgement of the total integration takes place and gives outputs, saying something about the results and the person in that situation. Tools of assessment are of value to develop competence. Luken (2004) emphasizes it is wise not to focus on psychometric reliability and validity only, but on consequential validity as well. The Assessment & Evaluation, definite paper, june 2007. 5 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  7. 7. value of the consequences depends on the quality of the process of judgement and on the use of concepts. By making specific output of jobs connecting matrices will arise and the concept of competence becomes useful (Luken, 2004). The redesigned nursing curriculum at the University of Colorado presents a systematically practice-oriented model: Competency Outcomes and Performance Assessment (COPA). (Lenburg, 1999). Regulatory and professional bodies define standards of acceptable competent performance. Criterion based performance measurements measure competence and identify existing needs for skills or knowledge deficiencies. The model is a framework for assessing competencies essential for nursing practice, including psychometrical techniques of assessment. Used methods are Competency Performance Assessment (CPA), to assess all class assignments, and Competency Performance Examination (CPE) used in clinical situations to evaluate clinical outcomes. Both comprise critical elements collectively defining competence for particular skills or abilities. The elements are single, discrete, observable behaviours mandatory for the specific competencies being evaluated and specify exactly acceptable performance in clinical and non-clinical courses; the course grade is based on this pre-determined foundation. The elements are created within the context of accepted standards of practice, evidence-based research literature, and course objectives. Students are continuously reminded of the performance expectations required within the context of contemporary practice and of prerequisite competencies necessary for a module. This enhances ultimate success, confidence and competence (Redman et al., 1999). The model of COPA is based on four questions: what are essential competencies and outcomes of contemporary practice?; what are indicators to define those competencies?; what are the most effective ways to learn those competencies? And what are the most effective ways to document achievement of the required competencies? Eight core practice competencies are: assessment and intervention; communication; critical thinking; teaching; human caring relationships; management; leadership and knowledge integration. The model can be applied to all kinds of environments, levels and types of nurses in different practice roles and to continuing education. Essential requirements are specification of competency outcomes, specific indicators of competence, the most effective ways they can be learned and the most effective methods by which competence will be tested against the pre- established standards (Lenburg, 1999) So, to measure competences there are several tools available, such as a matrix described by Luken (2004) and the model of COPA. To assess competence a criteria referenced approach is suitable, with a list of criteria based on valid performance of competent practitioners according to standards of control level. These evaluation methods evaluate performance against a set of known criteria. Analyse of the current evaluation and assessment instruments VUMC Comparing all the theory with current instruments of the VUMC leads to the analysis described here. At the VUMC hospital of Amsterdam there are three kinds of instruments of assessment for making a final judgement of a student: a paper of self-reflection, a between-judgement and a final judgement. The guidelines of these instruments are shown in appendix 1. Before and after the between- judgement there are several consultations of progress. The instruments of between- and final judgements are almost identical and both define the five items as requirements of development in nursing practice. One item is about learning process, the others about different roles of nursing practice. The criteria on the list measure the appearance of each item. The appearance of a student within the five different items says something about the performance. The outputs of the current instruments do not give much information about the achieved results and the minimal requirements to achieve; criterion transparency cannot be reached. If the results are not clearly defined, it is difficult to support the student to achieve those competence goals effectively. Nurse educators of the domain formulated the criteria. There are general but bound context formulated objectives of education, showing criterion-referenced assessment. They are suitable for competence education, but these objectives are not complete. The instruments are identical for each stage of the education, which is beginner, junior and bachelor. There is no specification of the expected level to be reached at each stage and no required minimum for each criterion. The expected level of performance and judgement depends on requirements of the college of the student, which differs from college to college; the individual department culture of the hospital and the capability of Assessment & Evaluation, definite paper, june 2007. 6 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  8. 8. individual nurse and nurse educator as well. This is not good for inter-rater reliability. For judging objectively, requirements of the list do not clearly enough define inadequate, adequate and excellent performance. Gibbs (2006) appoints this makes a construct less valid and standardization less obvious. By using the same instruments for between- and final judgement predictive and concurrent validity have been taken care of. The between-judgement is an indication of the final-judgement and gives points to improve. Without positive achievements the results of the same student will be approximately the same. Validity of content has been taken care of by having nurses and nurse educators setting the criteria. Validity of construct needs further investigation: does judging the five items and criteria give a reliable picture of need competence and performance? Nurse educators of the hospital formulated the criteria and must be judged adequately at a minimum. Consequential validity and reliability needs to be taken care of as well. If a different marker gives a different interpretation of the criteria it will effect the judgement and may lead to different total judgement. The value of the consequences depends on the quality of the process of judgement and the use of concepts. Essential requirements are specification of competency outcomes, specific indicators of competence and single interpretation of criteria. Objectivity is hard to reach, because the competence is bound context and disturbing influences are possible. However, if outcomes are clearly defined, there will be less difference between a bad and an excellent professional for judging. The guidelines state that the counselling, evaluation and judgement will be reviewed at the meeting of introduction, which will lead to transparency. But it is not specifically written down, because all units of care are different and it depends on the context. This means less transparency for the student. The purpose of the paper of reflection is evaluation of process and self-judgement of the student. The requirements for the paper are written down in the guidelines. The output of the paper is judging and part of the between- and final- judgement, which is suitable for competence based education assessment. The approach within the judgement is analytical with a scale of three points, which means inadequate, adequate and excellent. The assessment is conjunctive: to achieve a positive judgement the minimum score of adequateness for any element has to be reached. So, there is a combination of an analytic and holistic approach. The student can give a self- judgement, but it is the assessor who decides. Another aspect of the assessment instruments of the VUMC is that the items are not covering the national by official bodies adopted five roles of a nursing professional: the role of designer is really missing. This role covers the development of evidence based practice, best practice and innovation. At the same time one of the hospitals core competences is outcome centered practise. This implies quality improvement; innovation and management alignment with the role of designer. The role of patient centred care provider, the role of coach and the role of manager are apparently present in the assessment instrument. The role of professional is not mentioned in the assessment instrument: within the five items of the assessment instrument some criteria are aligned with a professional attitude, but it is not explicitly present as an item. The role of patient centred care provider is not complete. The outcome does not represent all the actions to take care of. Conclusions and recommendations. The current assessment instruments of the VUMC have been analysed after research of different kinds of aspects of competence and competence based education. Examples are performance, the changing environment the practical nursing education has to deal with and the consequence of assessing differently. The conclusion is that the current evaluation and assessment instruments of the VUMC do not measure competence-based outcomes, which it would like to measure on different levels. The main reason is the lack of critical based elements, which has to be achieved before going to a next level. The purpose of an assessment instrument must be to give insight in and optimalisation of the learning process, including optimalisation of the practical education process with clearly defined outcomes. The instrument itself is a threat to the valid use of assessment. There are too many weaknesses in the chain of validity (see table 1, page 5). Well-defined requirements are serving as a they have to satisfy the validity, reliability, objectivity, transparency and standardisation objectives much more. The consequential validity and reliability need to be taken care of. Firstly, it is recommended to innovate the current assessment instrument of the VUMC, because of its own organisational goals. Secondly, it is necessary to innovate the current assessment instruments in accordance with its environment to support and not being a threat in the valid use. Assessment & Evaluation, definite paper, june 2007. 7 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  9. 9. Thirdly, nurse educators and practioners need to improve themselves in judging and coaching students in a positive objective, transparent and standardised way. A possible way to improving nurse practioners capabilities is to develop a different assessment instrument, for example a matrix of competences as described by Luken. Fourthly, the core-competence of the VUMC is outcome- centred and it is recommended to adopt the aligned ideas of the COPA model (Competence Outcome Performance Assessment). Finally, the valid use of Crooks‟ model of assessment and its planning of assessments is recommended. References Baartman, L. K. J., Bastiaens, T. J. & Kirschner, P. A. (2004). Requirements for competency assessment programmes. Paper presented at the Onderwijs Research Dagen 2004. Utrecht, The Netherlands. Gevonden op 30 mei 2007, op http://www.fss.uu.nl/edsci/index.php?option=com_content&task=view&id=55&Itemid=39 Berkel, H., Bax, A. (2006). Toetsen in het hoger onderwijs. Houten: Bohn Stafleu Van Loghum. Crooks, T. J., Kane, M. T. & Cohen, A. S. (1996). Threats to the valid use of assessments. Assessment in Education: principles, policy & practice, 3 (3), 265-285 Dochy, F., Nickmans, G. (2005). Competentiegericht opleiden en toetsen. .Theorie en praktijk van flexibel leren. Utrecht: Lemma. Gibbs, G. (2006). Innovative assessment in higher education. London & New York: Routledge. Gipps, C. V. (1994). Beyond Testing. Towards a theory of educational assessment. Abingdon: Routledge Falmer Glaser, R. (1963). Instructional technology and the measurement of learning outcomes: Some questions. American Psychologist, 18, 519-521. Grotendorst, A., Van Aken, I., Heida, A. & Sino, C. (2006). Bekwaamheid op de proef gesteld. Perspectieven op competentiegericht beoordelen. Houten: Bohn Stafleu van Loghum. Lenburg, C. B. (1999). Redesigning expectations for initial and continuing competence for contemporary nursing practice. Online Journal of Issues in Nursing, 4 (2). Gevonden op 1 juni 2007, op http://www.nursingworld.org/ojin/topic10/tpc10_1.htm Luken, T. (2004). Zijn competenties meetbaar? Dilemma en uitweg bij een werkbaar maken van het competentiebegrip. Tijdschrift voor Hoger Onderwijs, 22 (1), 38-53. Redman, R. W., Lenburg, C. B. & Hinton Walker, P. (1999). Competency assessment: Methods for development and implementation in nursing education. Online Journal of Issues in Nursing, 4 (2). Gevonden op 1 juni 2007, op http://www.nursingworld.org/ojin/topic10/tpc10_3.htm Scott, G. (1991). How clever are we in the way we train our workers? The Great Australian Competence Caper. Training and development in Australia, 18 (2), 7-12. Spencer, L. M., Spencer, S. M. (1993). Competence at work. Models for superior performance. New York: Wiley & Sons. Van Merriënboer, J. J. G., Van der Klink, M. R., & Hendriks, M. (2002). Competenties: van complicaties tot compromis. Over schuifjes en begrenzers. Een studie in opdracht van de onderwijsraad. Den Haag, the Netherlands: Onderwijsraad. While, A. E. (1994). Competence versus performance: which is more important? Journal of Advanced Nursing, 20 (3), 525-531. Assessment & Evaluation, definite paper, june 2007. 8 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  10. 10. Appendix1: Original forms of the VUMC of Amsterdam - VUMC - richtlijn voor reflectieverslag, tussenbeoordeling en eindbeoordeling - VUMC- formulier voor tussen beoordeling - VUMC- formulier voor eindbeoordeling Appendix 2: Report of reflection - Reflectie op de verkregen feedback en het gebruik van deze feedback - Ervaringen bij het zelf geven van feedback Assessment & Evaluation, definite paper, june 2007. 9 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  11. 11. Appendix1: Original forms of the VUMC of Amsterdam Richtlijn voor reflectieverslag, tussenbeoordeling en eindbeoordeling Reflectieverslag Op de verschillende zorgeenheden kan de wijze van begeleiding, evaluatie en beoordeling anders georganiseerd zijn. Dit wordt besproken tijdens het introductiegesprek. De student krijgt aanwijzingen en feedback gericht op het niveau van functioneren. Om het leerproces goed te kunnen volgen is het noodzakelijk regelmatig te evalueren. De student wordt gevraagd voortdurend terug te blikken op zijn handelen, om te leren analyseren, evalueren en waar nodig bij te stellen. De student is verantwoordelijk voor zijn leerproces en hierin past het beschrijven van de ontwikkeling die de student doormaakt en tevens het beschrijven van de zelfbeoordeling. De student geeft met het schrijven van een reflectieverslag een terugblik op de praktijkleerperiode, de stand van zaken tot nu toe, persoonlijke ervaringen, inzicht in het beroep, conclusies en voornemens weer. Benoem de vorderingen die je maakt en om beschrijf deze in het reflectieverslag. Als hulpmiddel zijn de volgende vragen geformuleerd. Hoe heb je tijdens de praktijkleerperiode gewerkt aan je leerdoelen? Wat heb je per leerdoel en item bereikt, waar ben je goed (competent) in geworden? Waar moet je nog per leerdoel en item aandacht aan besteden of waarin kun je jezelf nog ontwikkelen? Hoe kun je aantonen dat je de leerdoelen hebt behaald? Waar ben je enthousiast over en wat vind je leuk in het werk? Waar zie je tegenop of vind je moeilijk/vervelend? Beschrijf waardoor dat komt. Wat kun je eraan doen om dit te verbeteren? Hoe ervaar je de rol als leer- werkstudent? Hoe geef je deze rol vorm? Hoe ervaar je het leren en werken op de zorgeenheid? Wat leer je van de zorgvragers, medestudenten en begeleiders, andere disciplines, enzovoorts. Beantwoord de volgende vraag: Mijn persoonlijke inschatting is dat ik deze praktijkleerperiode wel / niet met succes zal afleggen. Motiveer je antwoord. Het reflectieverslag is Max. 4 A4-tjes en dient tijdig, minimaal twee dagen vóór het gesprek of volgens afspraak van de zorgeenheid, ingeleverd te worden bij de werkbegeleider(s) en praktijkopleider. Assessment & Evaluation, definite paper, june 2007. 10 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  12. 12. Tussenbeoordelings- en eindbeoordelingsgesprek Doelstelling De tussenbeoordeling en eindbeoordeling zijn formele besprekingen van het leerproces aan de hand van het reflectieverslag (evt. portfolio) van de student en de gespreksformulieren. In deze gesprekken wordt: vastgesteld welke resultaten zijn bereikt een tussentijdse (tussenbeoordeling) en uiteindelijke conclusie (eindbeoordeling) getrokken leerpunten en afspraken gemaakt en vastgelegd het leerproces en de voortgang geëvalueerd de begeleiding geëvalueerd Wanneer en - De tussenbeoordeling vindt halverwege en de eindbeoordeling aan het eind van de praktijkleerperiode plaats. met wie - Begeleiders/praktijkopleiders van de zorgeenheid en de student. Voorbereiding De student schrijft voor de tussenbeoordeling-/eindbeoordeling een reflectieverslag en levert dit tijdig (volgens afspraak ZE) in bij de begeleiders en praktijkopleider. Zij ondertekenen het reflectieverslag als voldaan. De student levert het ondertekende reflectieverslag in bij de onderwijsinstelling en geeft een kopie aan de praktijkopleider. De praktijkopleider zorgt ervoor dat deze kopie in het studentendeel van het studentenarchief van het VUMC wordt opgeborgen. Verslaglegging De student en begeleider/praktijkopleider spreken af wie aanvullingen en de reactie van de student op het gespreksformulier notuleert tijdens het gesprek. Beoordelingsformulier De begeleiders vullen van tevoren het gespreksformulier van de tussenbeoordeling en de eindbeoordeling in en leggen deze tijdens het gesprek aan de student voor. Het beoordelingsformulier van de onderwijsinstelling wordt ingevuld en ondertekend door begeleiders/praktijkopleider van de zorgeenheid. De student levert deze in bij de onderwijsinstelling en geeft een kopie aan de praktijkopleider. De praktijkopleider zorgt ervoor dat deze kopieën in het studentendeel van het studentenarchief van het VUMC worden opgeborgen. Conclusie De behaalde resultaten worden op schrift gezet en deze worden altijd zowel mondeling als schriftelijk toegelicht tijdens het gesprek. Er wordt per onderdeel een conclusie beschreven (Goed, Voldoende of Onvoldoende). Tenslotte wordt de eindconclusie getrokken aan de hand van de verdeling van de diverse onderdelen (zie criteria op het formulier). Assessment & Evaluation, definite paper, june 2007. 11 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  13. 13. Tussenbeoordeling Naam student : Groep : Praktijkleerperiode : van tot Zorgeenheid / leerplaats : Werkbegeleider(s) : Datum gesprek : Toelichting: De werkbegeleider beoordeelt de gemaakte vorderingen in het leerproces conform het leerjaar waarin de student zich bevindt en licht zijn/haar bevindingen altijd toe. Voorbereiding van beoordelingsgesprek: - De werkbegeleider vult van te voren dit beoordelingsformulier in en legt dit tijdens het beoordelingsgesprek voor aan de student - De student levert van te voren bij de werkbegeleider en praktijkopleider een getypt reflectieverslag in (zie richtlijn). 1) Leerproces, de student: Criteria Beoordeling G/V/O  Voert het leeractiviteitenplan volgens voorgenomen schema uit  Kiest leersituaties (in overleg met WB) die in relatie staan tot leerdoelen  Vraagt regelmatig of hij/zij gedrag vertoond dat erop gericht is om leerdoelen te bereiken: - toont initiatief en interesse, is leergierig en enthousiast - staat open voor feedback en kritiek, vraagt naar concrete voorbeelden en accepteert suggesties voor verbetering - neemt verantwoordelijkheid voor het eigen handelen - kan theorie en opgedane kennis toepassen in de praktijk - geeft blijk van analytisch vermogen en kan klinisch redeneren Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: Assessment & Evaluation, definite paper, june 2007. 12 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  14. 14. 2) Patiëntgerichtheid, de student: Criteria Beoordeling G/V/O  Handelt zoveel mogelijk vanuit het belang van de patiënt  Geeft duidelijk en onderbouwd aan wat wel en niet mogelijk is aan de patiënt  Behandelt de patiënt met respect en neemt daarbij omgangsvormen in acht  Onderzoekt de vraag van de patiënt en vraagt goed door  Maakt duidelijke afspraken met de patiënt over de te leveren zorg en handelt hiernaar  Evalueert met de patiënt, gaat na of deze tevreden is en neemt acties tot verbetering Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: 3) Samenwerken, de student: Criteria Beoordeling G/V/O  Overlegt en stemt werkzaamheden, wensen en verwachtingen af met de ander(en)  Deelt actief informatie en opgedane kennis  Geeft door gedrag en houding aan open te staan voor wat gesprekspartners inbrengen  Vraagt door op gegeven informatie of vraagt opheldering als iets niet duidelijk is Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: 4) Coördinatie, planning en continuïteit van zorg, de student: Criteria Beoordeling G/V/O  Formuleert concrete en meetbare doelstellingen  Stelt prioriteiten, handelt anticiperend en uit eigen beweging  Brengt structuur aan binnen de eigen werkzaamheden en maakt een eigen werkindeling  Creëert en benut mogelijkheden en middelen om doel te bereiken  Wisselt van gedrag, aanpak of wijze van communiceren wanneer de situatie daarom vraagt  Kan inspelen op veranderende omstandigheden en wijzigende inzichten Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: Assessment & Evaluation, definite paper, june 2007. 13 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  15. 15. 5) Coachen, de student: (alleen indien van toepassing ) Criteria Beoordeling G/V/O  Geeft specifieke constructieve feedback aan medestudenten over het functioneren en de ontwikkeling die men nastreeft  Stimuleert medestudenten om eigen verantwoordelijkheid voor hun werk en ontwikkeling te nemen  Stimuleert initiatieven van medestudenten tot vergroten en verbreden van hun kennis en vaardigheden  Toetst of geleerde zaken goed begrepen zijn en of het lukt deze in praktijk te brengen Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: Opmerkingen n.a.v. het reflectieverslag Aandachtspunten: Eindconclusie: Criteria: mét onderdeel 5 zonder onderdeel 5 Beoordeling  4 of meer items G, rest V 3 of meer items G, rest V Goed  3 of meer items V en rest G 2 of meer items V en rest G Voldoende  1 maal O 1 maal O Onvoldoende  G = Goed  V = Voldoende  O = Onvoldoende Ondertekenen Naam aanwezigen Functie Handtekening Gezien / akkoord* *) invullen wat van toepassing is Assessment & Evaluation, definite paper, june 2007. 14 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  16. 16. Eindbeoordeling Naam student : Groep : Praktijkleerperiode : van tot Zorgeenheid / leerplaats : Werkbegeleider(s) : Datum gesprek : Toelichting: De werkbegeleider beoordeelt de gemaakte vorderingen in het leerproces conform het leerjaar waarin de student zich bevindt en licht zijn/haar bevindingen altijd toe. Voorbereiding van beoordelingsgesprek: - De werkbegeleider vult van te voren dit beoordelingsformulier in en legt dit tijdens het beoordelingsgesprek voor aan de student - De student levert van te voren bij de werkbegeleider en praktijkopleider een getypt reflectieverslag in (zie richtlijn). 1) Leerproces, de student Criteria Beoordeling G/V/O Voert het leeractiviteitenplan volgens voorgenomen schema uit  Kiest leersituaties (in overleg WB) die in relatie staan tot de leerdoelen  Vraagt regelmatig of hij/zij gedrag vertoond dat erop gericht is om leerdoelen te bereiken: - Toont initiatief en interesse, is leergierig en enthousiast - Staat open voor feedback en kritiek, vraagt naar concrete voorbeelden en accepteert suggesties voor verbetering - Neemt verantwoordelijkheid voor het eigen handelen - Kan theorie en opgedane kennis toepassen in de praktijk - Geeft blijk van analytisch vermogen en kan klinisch redeneren Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: 2) Patiëntgerichtheid, de student Assessment & Evaluation, definite paper, june 2007. 15 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  17. 17. Criteria Beoordeling G/V/O  Handelt zoveel mogelijk vanuit het belang van de patiënt  Geeft duidelijk en onderbouwd aan wat wel en niet mogelijk is aan de patiënt  Behandelt de patiënt met respect en neemt daarbij omgangsvormen in acht  Onderzoekt de vraag van de patiënt en vraagt goed door  Maakt duidelijke afspraken met de patiënt over de te leveren zorg en handelt hiernaar  Evalueert met de patiënt, gaat na of deze tevreden is en neemt acties tot verbetering Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: 3) Samenwerken, de student Criteria Beoordeling G/V/O  Overlegt en stemt werkzaamheden, wensen en verwachtingen af met de ander(en)  Deelt actief informatie en opgedane kennis  Geeft door gedrag en houding aan open te staan voor wat gesprekspartners inbrengen  Vraagt door op gegeven informatie of vraagt opheldering als iets niet duidelijk is Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: 4) Coördinatie, planning en continuïteit van zorg, de student Criteria Beoordeling G/V/O  Formuleert concrete en meetbare doelstellingen  Stelt prioriteiten, handelt anticiperend en uit eigen beweging  Brengt structuur aan binnen de eigen werkzaamheden en maakt een eigen werkindeling  Creëert en benut mogelijkheden en middelen om doel te bereiken  Wisselt van gedrag, aanpak of wijze van communiceren wanneer de situatie daarom vraagt  Kan inspelen op veranderende omstandigheden en wijzigende inzichten Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Assessment & Evaluation, definite paper, june 2007. 16 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  18. 18. Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: 5) Coachen, de student (alleen indien van toepassing) Criteria Beoordeling G/V/O  Geeft specifieke constructieve feedback aan medestudenten over het functioneren en de ontwikkeling die men nastreeft  Stimuleert medestudenten om eigen verantwoordelijkheid voor hun werk en ontwikkeling te nemen  Stimuleert initiatieven van medestudenten tot vergroten en verbreden van hun kennis en vaardigheden  Toetst of geleerde zaken goed begrepen zijn en of het lukt deze in praktijk te brengen Conclusie voor dit onderdeel:  G = Goed  V = Voldoende  O = Onvoldoende Toelichting (werkbegeleider vult dit altijd in): Reactie van student op bovenstaande: Opmerkingen n.a.v. het reflectieverslag Aandachtspunten: Eindconclusie: Criteria: mét onderdeel 5 zonder onderdeel 5 Beoordeling  4 of meer items G, rest V 3 of meer items G, rest V Goed  3 of meer items V en rest G 2 of meer items V en rest G Voldoende  1 maal O 1 maal O Onvoldoende  G = Goed  V = Voldoende  O = Onvoldoende Ondertekenen Naam aanwezigen Functie Handtekening Gezien / akkoord* *) invullen wat van toepassing is Assessment & Evaluation, definite paper, june 2007. 17 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  19. 19. Appendix 2 - Reflectie op de verkregen feedback en het gebruik van deze feedback - Ervaringen bij het zelf geven van feedback Reflectie op de verkregen feedback en het gebruik van deze feedback - Het krijgen van feedback geeft informatie over de kwaliteit m.b.t. de diverse aspecten en t.a.v. het geheel. Sommige aspecten zijn door ons wel gezien, zoals het feit dat het een actueel onderwerp is dat aandacht verdiend. Andere dingen echter niet, omdat je als schrijver een „blinde vlek‟ krijgt. Het voordeel was dat het groepje uit drie mensen bestond en ieder vanuit eigen perspectief naar het geheel en de onderdelen keek, maar dan nog ontstaat dat fenomeen. Het is daarom goed om van anderen een „fris‟oordeel te krijgen, om vervolgens de kwaliteit te kunnen verhogen. En te ervaren dat het als (toekomstige) professional goed is om een deskundige naar gemaakt werk te laten kijken. - Het is prettig om te merken dat ons oordeel over het werk qua niveau overeenkomt met het oordeel van de feedback. Dat betekent dat we in staat zijn dat realistisch in te schatten. - Door de opbouw van het feedbackformulier is het mogelijk op systematische manier de feedback te verwerken in de paperdoor de aspecten puntsgewijs bij langs te gaan. Ervaringen bij het zelf geven van feedback - Leerzaam om het werk van mede - studenten te bekijken en te kijken naar sterke en zwakke punten van het geheel. Wat is goed gedaan en wat is minder goed? - Goede oefening om deze sterke en zwakke punten onder woorden te brengen: Waarom is bijvoorbeeld de organisatie van het stuk goed, maar de inhoud nog van minder niveau? Waar ligt dat aan, hoe kan dat verbeterd worden? - Het bekijken van de beoordelingscriteria voor het geven van de peer- feedback bewerkstelligde diepere bewustwording van deze criteria. - Het bekijken van de criteria leverde ook meer bewustwording van de weging van de aspecten van de criteria: inhoud (terecht) 40%, andere onderdelen 10 tot 15%. - Eigen werk werd n.a.v. de peer -feedback m.b.v. het beoordelingsschema nog eens nader bekeken: verbetering van eigen werk niet alleen door feedback van anderen, maar ook door het zelf feedback geven en daardoor diepere bewustwording van bewustwording van beoordelingscriteria. - Het lezen van artikelen en feedback van anderen binnen de groep 24-26 was ook een eye- opener: bijna iedereen van onze groep beoordeelde in het midden met een aantal bolletjes aan de linkerzijde en een enkel bolletje helemaal rechts ingevuld. Het is opvallend dat de artikelen gemiddeld dus als “in ontwikkeling” beoordeeld werden, terwijl de artikelen onderling duidelijk van niveau varieerden! - Omdat ons groepje26 b bestaat uit 3 mensen gingen we overleggen of iets uit het te beoordelen artikel van beginnend, in ontwikkeling of gevorderd niveau is en we bleken hier behoorlijk eensgezind over. Bij de totale eindbeoordeling van het geheel bleek de een toch strenger dan de ander: de een vond het geheel nog lang niet voldoende, de ander nog niet voldoende. De verdeling 40%, 10 en 15% bleek hierbij een goede hulp: door 40% te vertalen in 4 punten, 10% in 1 punt en 15 % in 1,5 punten is een maximum score van 10 mogelijk en kon er makkelijker via een cijferindicatie tot een (on)voldoende indicatie gekomen worden. Dit hielp dus de grens tussen voldoende en onvoldoende te bepalen. - Het beoordelingschema is een goed hulpmiddel bij het bekijken en beoordelen van eigen artikelen en die van anderen, goed is wel om daarna nog onder woorden te brengen waarom iets goed is of beter kan. Dit helpt bij het lezen en zelf schrijven van artikelen en de ontwikkeling van deze vaardigheid. Assessment & Evaluation, definite paper, june 2007. 18 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.
  20. 20. Assessment & Evaluation, definite paper, june 2007. 19 Group 26b: A. W. .Duit, I. de Lijser, F. Vasse.

×