There are many theories on how to develop a good examination paper. The type of examination paper you create depends on the specific needs of the examining body, school, organization or nation - their philosophy, their classification/statistical needs, if any, and the needs of the students and the institutions in which they would like to continue their education, or the jobs into which they would like to enter. It also depends on the subject and what is suitable to test the skills and knowledge for that subject. There are many aspects of the examination to take into consideration, each one playing off the other, working with the other to create the perfect examination paper for a certain situation. I was asked what the essential ingredients for an excellent Literature paper in an international context would be. We may, therefore be able to make a few assumptions on the needs and philosophy of the organization giving the exam and the students taken them, given the examination is to take place in an international context and would therefore have certain requirements, for example - the questions will need to be applicable to any culture it is taken in and the skills and concepts learned must be transferable as well as innately promote intercultural awareness and contemplation from the content of the examination. In this presentation I would like to look at different decisions that would need to be made when designing an examination, important concepts which would apply to any examination and give an example as to how these concepts might come together to create a strong literature examination in an intercultural context.
There are quite a few aspects to consider when putting together an examination paper. One of those is the type of skills you wish to assess in the examination. This can be determined by the subject and also by what the results of the examination will be used for. Presently there are two main types of assessment used today. They are psychometric assessment and performance assessment. In both types of assessment questions and tasks are carefully chosen and put together in order to create an examination which will fulfils the needs of the examining body. It is important that they try to get the right balance between the construction of the exam and the reliability of the marking. However they go about this in two different ways, and each method produces different results, both positive and negative depending on what the examination results will be used for.
Psychometric testing is a form of standardized test, usually multiple choice, used to test, what was often seen as innate qualities in a person - such as intelligence, aptitude and capability. It was later used also to test academic achievement. It is valued as an assessment tool due to its ability to provide accurate measurements and precise results for student ranking. Questions can be carefully chosen to isolate a certain skill or type of knowledge which could be used to differentiate between students. This type of exam offers a high level of accuracy of measurement and as exams are automatically marked a high level of equity. Needless to say this method of assessment is also cost and time effective as there is no need for an external examiner.
Performance assessment differs quite dramatically from psychometric assessment. Instead of demonstrating knowledge by choosing from among a list of possible answers the students are required to actually use the skills and knowledge they have learned in the classroom to demonstrate their abilities. The types of assessment used are varied and can include many tasks such as &quot;problem solving, essays, project work and examinations” and push the students to use and demonstrate a wide range of abilities and skills, not just regurgitating knowledge.
In the DP Assessment principles and practice document, a document for an international education organization, it states, “It is often stated that we now live in a knowledge society. This should not be taken to mean that the acquisition and retention of factual information is of prime importance. The explosion of information in recent times makes it impossible for individuals to achieve mastery of knowledge in many areas. The more valued academic skills of today are in accessing, ordering sifting, synthesizing and evaluating information, and creatively constructing knowledge (16).” Performance assessment is often preferred as an assessment option because it can adequately account for this change in society and the skills needed to succeed in this developing society.
Performance assessment incorporates and allows to assess skills such as Bloom’s higher-order cognitive skills in order to adapt to this different way of gathering and using information and knowledge. This type of assessment does have its drawbacks, however, as it requires the professional judgment of an examiner who will mark the exams and determine the level of achievement. This can be time consuming and costly, whereas psychometric testing is relatively cost and time effective with a high reliability of results. There is also the consequence that with performance assessment, whereas the construct validty of the examination may be maintained, the scores will be less reliable as different marker's results/opinions may vary. However, levels of reliability can be raised with the use of a detailed markscheme and high levels of quality control.
Another important aspect of examinations, is their effect on teaching and learning. The effect on teaching of examinations is often called Washback or Backwash. This can be positive or negative. Examinations can be powerful entities, especially high-stakes examinations where the futures of the students and the teachers will be effected. Teachers’ positions in some countries may rely on the results of their students, and the students' futures, which direction they follow in school and after may also depend on the outcomes of these exams. Therefore the nature of the exam will effect how the teacher teaches the necessary information in the classroom. For example, if the exam is a multiple choice examination, testing a narrowly defined construct or concept, then the teaching might reflect this, leaving the students with limited knowledge and skills. However, if the examination is based on performance assessment, proliferate with varying assessment types and skill use, then the teacher would be rewarded for teaching a proliferation of skills and knowledge, and practice in using these skills, which would result in a more well-rounded and skilled student.
Another important aspect of preparing an examination paper is the design. The question of whether an exam is well constructed is often a question of balance. I have already alluded in some cases to the concepts of validity and reliability. It is important to carefully construct the examination to have a good balance of these two concepts. There are several types of validity and reliability, however, simply put, a high level of validity occurs when the construct and questions assess what the exam has been developed to assess, and reliability occurs when the measurement of the assessment results are accurate and can be reproduced in another similar environment. These concepts are an important aspect to any examination, especially a high-stakes examination. The DP Assessment principles and practices document states, “Validity and reliability are widely regarded as essential characteristics of any assessment system, particularly a high-stakes one where the outcome is of great importance to the student or the teacher (8).” It must be mentioned here that performance assessment requires the use of professional judgement and therefore marker reliability may be reduced if measure are not put into place to preserve this. For example, by arefully choosing markers, having a detailed markscheme, and a strong moderation system. These definitions broaden when speaking about psychometric testing and performance assessment. When discussing psychometrics, validity refers to identifying the single construct being tested in the examination and reliability “how consistently the different items in the test behave, in terms of the correlation of student responses given to these different items (8).” Validity and reliability are defined differently when discussing performance assessment. Important to validity in this case is also what the results of the examination will be used for and how they will be perceived as well as the washback effect on teaching and learning. With performance assessment, the scope of validity is greatly widened to include the wide range of skills and knowledge assessed, the way it is assessed and even the results of the assessment on teaching.
We have looked at several different major aspects involved in putting together an examination: the type of examination, whether it be psychometric or performance, the effect of negative or positive washback and the internal construction of the exam in terms of validity and reliability. After looking at the various aspects of putting together an examination I think we can make an assumption as to the ingredients that would make an excellent literature examination paper in an international context. It seems that a clear idea of the goal of the examination is needed, if this goal is to develop students with higher-order transferable skills, then an examination within the parameters of performance assessment is most desirable. After the skills that need to be assessed have been decided, then a well constructed exam, with questions that target and adequately assess the students ability to use the skills they have learned in class is required. It should therefore be able to demonstrate a positive effect on teaching and learning and not a negative effect. And finally, it should be possible to mark the results with a high level of reliability and ensure the trust of the institutions and organizations which will be using the results, and the students who will be relying on the results to shape their future.
The IBO is an example of such an organization that would give a literature examination in an international context. It is an international organization catering to schools around the world. They aim to promote international and intercultural understanding among the students taking the IB examinations. The DP examinations are high-stakes examinations, recognized by universities internationally, and therefore will have a high degree of washback. It is interesting to see how they have chosen certain assessment concepts in order to create a strong literature examination.
This is the outline of the IB’s Language A Literature examination. We can see right away that this is an example of performance assessment. Students are asked to use a range of skills and knowledge in order to analyze texts, some unseen, others studied in class. If we look at Paper 1 we can see that the question necessitates that students use skills acquired in class on unseen texts. They are not being asked to reiterate what they learned about a text they have already read, but to apply skills of literary analysis to interpret the texts and to discuss aspects such as content, technique, style, structure, theme and language. This is a good example of promoting the use of higher-order skills and performance assessment to assess the students. Paper 2 looks at yet another aspect of the skills the students should have learned during the literature course. In this case they are asked to look at texts all from the same literary genre but to focus on recognisable techniques or literary conventions applicable to that genre, therefore assessing another area of knowledge learned in the classroom.
The written assignment again tests the students in yet a different area. They are asked to write a reflective statement and an essay of up to 1500 words. What is interesting about the written statement is that preparation for this begins explicitly in the classroom and is carried out in stages. Teachers are asked to lead the students in discussion and address certain cultural and contextual aspects of the texts studied. The texts studied for this part of the examination are also works in translation. They will be texts from countries with different languages and cultures, thereby giving another dimension to, and support, intercultural understanding and contemplation.
Stage 2 consists of asking students to look back at the oral discussion and write a short reflective statement. This will be turned in with the completed statement.
Stage 3 requires students to write, using prompts from the teacher, for about 50 minutes on each of the texts studied. They will choose one of these texts to develop into an essay which will be marked externally. Stage 4 consists of writing the essay. What is really interesting about this assignment is that it sets out the skills the students are to be able to use in the examination, and therefore reflects back positively on the teaching and learning. The teacher is required to teach a wide range of skills so that the students will be knowledgeable and comfortable with using these skills for the examination. This is true for the entire examination. This literature exam has positive washback on teaching and learning. As we can see there is also a high level of construct validity due to the carefully chosen questions, and the fact that they were careful to design the exam to test the high level skills they want to assess. The IB also brings a high level of reliability to the assessment with detailed mark schemes, careful selection of examiners and a detailed moderation process.
The development of an examination paper is a mutli-faceted production. There are many aspects and much theoretical knowledge which goes into this, and this highlights the need to keep up to date of current theories. However, it is not just knowledge of the varying theories that is required, but a clear idea of the goal of the examination. We have seen how assessment can influence the teaching and learning in schools, and therefore shape the learning experience of students and the way they learn. We have also looked at the importance of concepts such as reliability and validity and how the importance of these changes with respect to the type of examination desired, whether it be psychometric assessment or performance assessment. In an international context we decided which of these aspects were the most important, and looked at the International Baccalaureate Organization literature examination as an example. We explored how they used the various aspects of putting together an examination paper in an international context, and created an excellent literature examination paper.
Literature examination paper international context (2)
The essential ingredients to an excellent literature examination paper in an international context The optimal balance
Ingredients? <ul><li>Subject </li></ul><ul><li>Skills to be assessed </li></ul><ul><li>Object of the examination </li></ul><ul><li>Who the examination is for? </li></ul><ul><li>What will the results be used for? </li></ul><ul><li>… </li></ul>
Psychometric testing <ul><li>Standardized testing </li></ul><ul><li>Accuracy of measurement </li></ul><ul><li>Student ranking </li></ul><ul><li>Testing of one narrow construct </li></ul><ul><li>Cost and time effective </li></ul><ul><li>Equity </li></ul>
Performance assessment <ul><li>Students demonstrate using a wide variety of skills and knowledge </li></ul><ul><li>Assessment exercises are varied </li></ul><ul><ul><li>Problem solving </li></ul></ul><ul><ul><li>Essays </li></ul></ul><ul><ul><li>Project work </li></ul></ul><ul><ul><li>Examinations </li></ul></ul>
“ A knowledge society” <ul><li>“ It is often stated that we now live in a knowledge society. This should not be taken to mean that the acquisition and retention of factual information is of prime importance. The explosion of information in recent times makes it impossible for individuals to achieve mastery of knowledge in many areas. The more valued academic skills of today are in accessing, ordering, sifting, synthesizing and evaluating information, and creatively constructing knowledge (16).” </li></ul>
Higher-order cognitive skills - learning to learn - http://www.odu.edu/educ/roverbau/Bloom/blooms_taxonomy.htm
Washback <ul><li>Positive effects on teaching and learning </li></ul><ul><li>Negative effects on teaching and learning </li></ul>
<ul><li>“ Good assessment comes from achieving a satisfactory compromise, and the nature of the balance between reliability and validity will depend on the context and purpose of a particular system (9).” </li></ul>Reliability and Validity International Baccalaureate Organization (2004), Diploma Programme Assessment Principles and Practice, Cardiff: IBO.
Review <ul><li>School/organization philosophy </li></ul><ul><li>Psychometrics v performance assessment </li></ul><ul><li>Washback: positive v negative </li></ul><ul><li>Reliability v Validity </li></ul><ul><li>Detailed mark scheme/reliable markers/moderation system </li></ul>
The IBO mission statement <ul><li>The International Baccalaureate Organization aims to develop inquiring, knowledgeable and caring young people who help to create a better and more peaceful world through intercultural understanding and respect. </li></ul><ul><li>To this end the IBO works with schools, governments and international organizations to develop challenging programmes of international education and rigorous assessment </li></ul><ul><li>These programmes encourage students across the world to become active, compassionate and lifelong learners who understand that other people, with their differences, can also be right. </li></ul>
Language A Literature External Assessment: Performance assessment Diploma Programme, Language A Literature, curriculum review report – update on progress November 2009 Standard Level –assessment component and weighting Higher Level –assessment component and weighting Paper 1 : Literary analysis (1 ½ hours) Literary analysis of one unseen text in response to guided questions 20% Paper 1: Literary analysis: unseen commentary (2 hours) Literary analysis of one unseen text 20% Paper 2 : Essay (1 ½ hours) An essay based on at least two works studied in part 3 25% Paper 2: Essay (2 hours) An essay based on at least two works studied in part 3 25% Written assignment A reflective statement and literary essay on one part 1 work Reflective statement 300-400 words Essay 1200-1500 words 25% Written assignment A reflective statement and literary essay on one part 1 work Reflective statement 300-400 words Essay 1200-1500 words 25%
Written assignment based on a work in translation <ul><li>Stage 1: The interactive oral </li></ul><ul><li>The interactive oral is a focused class discussion, led by students and in which all students and the teacher participate. The discussions should address the following cultural and contextual considerations: </li></ul><ul><ul><li>In what ways do time and place matter to this work? </li></ul></ul><ul><ul><li>What was easy to understand and what was difficult in relation to social and cultural context and issues? </li></ul></ul><ul><ul><li>What connections did you find between issues in the work and your own culture and experience? </li></ul></ul><ul><ul><li>What aspects of technique are interesting in the work? </li></ul></ul><ul><li>Each student should be responsible for initiating an aspect of the discussion in at least one of the interactive orals and at least one oral must be completed in relation to each work studied in part 1. </li></ul><ul><li>A required minimum time for discussion of each work is 30 minutes. Class size may require that more than one oral be done on each work in order to give every student an active role. </li></ul>Diploma Programme, Language A Literature, curriculum review report – update on progress November 2009
<ul><li>Stage 2: The reflective statement </li></ul><ul><li>Each student will be asked to reflect on each of the interactive orals in a short writing exercise that must be kept on file in the school. This piece of writing is termed the reflective statement. The relevant reflective statement will be submitted with their completed assignment. </li></ul><ul><li>The statement should be based on the questions listed below. Question 1 is required and questions 2 and 3 are optional: </li></ul><ul><li>1. What elements of the role played by context were illuminated or developed for you? </li></ul><ul><li>2. What aspects of the discussion most interested you? </li></ul><ul><li>3. What new angles on the work did the discussion provoke for you? </li></ul><ul><li>The required length of the reflective statement is 300-400 words and it will be assessed by criterion </li></ul><ul><li>A of the marking criteria for the written assignment. The reflective statement that is sent with the assignment must be on the work used but does not necessarily have to be explicitly connected to the topic. </li></ul>Diploma Programme, Language A Literature, curriculum review report – update on progress November 2009
<ul><li>Stage 3: Developing the topic-Supervised writing </li></ul><ul><li>The final goal of this stage of the process is to assist students in producing good essays with appropriate topics. To this end students will be required to respond to each of the works studied in a preliminary written exercise undertaken during class time. </li></ul><ul><li>Teachers need to provide three or four prompts for each work studied at the beginning of the designated lesson. It is essential that students are not given the prompts beforehand. These prompts are designed to encourage independent critical writing and stimulate thinking about the assignment topic. The prompts may be selected from the list below, from those given in the teacher support material, or teachers may devise their own… . </li></ul><ul><li>Stage 4: Production of the written assignment </li></ul><ul><li>Each student will produce a 1200-1500 word essay on a literary aspect of one work, developed from one of the pieces of supervised writing. </li></ul><ul><li>Working from the selected piece of supervised writing teachers should provide some guidance to the students about the development of the assignment topic. It is the teachers responsibility to ensure that this topic is suitable to the length and the focus of the task. … </li></ul><ul><li>From there, the student will complete the written assignment on his or her own and submit it for external assessment. … </li></ul>Diploma Programme, Language A Literature, curriculum review report – update on progress November 2009